Compliant, deviating or impossible to conclude: reading a Pépin result

A Pépin result does not read like an alert, it reads like a sentence with a subject, a verdict and a reason. Four statuses exist, and two of them do not mean what people assume: not-applicable and not-evaluated are refusals to conclude, not quiet successes. This page explains what each status lets you assert, why the severity of a deviation says nothing about how sure you are of it, and how a scan's provenance lets you re-read it six months later.

What you will learn

Tell apart the four statuses, and what each one lets you conclude.
Separate severity, which says how bad, from confidence, which says how sure.
Read the provenance block: tool, ruleset, scope, timestamp.
Spot a result that resists interpretation, and what to check when you meet one.

The four statuses

Two statuses conclude, two refuse to. That is the first distinction to make, and it matters more than severity.

Status	What Pépin asserts	What you can do with it
`pass`	the required data was collected, and it is compliant	rely on it
`fail`	the deviation is established, on a named resource	fix it, or accept it through an exception
`not-applicable`	the control does not concern this scope, and here is why	check that the reason suits you
`not-evaluated`	the required data was missing, and here is which	nothing, until you have collected it

A fifth status, exempted, appears when you cover a deviation with an exception. It never blends into pass: the deviation stays visible, it is simply owned.

not-evaluated is not a success. On the example inventory shipped with the repository, 27 controls are processed and 14 come back not evaluated. A tool counting them as compliant would announce a score above 90 % where half the referential was never looked at.

That shortcut is the most common one in compliance dashboards, and it produces the exact opposite of what an audit is for: assurance built on what was not seen.

So always read the full breakdown before the verdict:

pepin scan scaleway examples/scaleway/inventory.json -f assessment \
  | jq -r '.results[].status' | sort | uniq -c

Why a control is not evaluated

Every non-evaluation carries its reason, and the reasons are not equal. Two families show up, with opposite consequences.

The first is benign: the resource does not exist in the scope.

compute_instance_has_security_group
  no resource of type "compute_instance" in the evaluated inventory

If your inventory holds no machines, a control about machines has nothing to examine. Nothing to fix, beyond checking that the absence is expected.

The second deserves your attention: the data could not be collected.

governance_resource_required_tags
  collection of the required data not confirmed for this provider

That one signals a real limit, on the tool's side, the provider's side or your permissions. On a live collection it is often a missing permission: the scanner could not read, so it does not conclude, and that is the intended behaviour. An under-authorised account must never produce a pass.

What a `not-applicable` teaches you

Since version 0.4.0, an inapplicable control states why it does not apply, and the justification is often instructive.

blockstorage_volume_encryption
  At-rest encryption of block volumes on the guest side (LUKS/Cryptsetup),
  customer responsibility

This is not an escape hatch: it is a boundary of the shared responsibility model. The control does not apply to the provider's scope because it belongs to yours, inside the machine. Reading those justifications is the best way to discover what you are responsible for without knowing it.

Severity and confidence: two different questions

Severity says what the deviation would cost. Confidence says how sure we are that it exists. Confusing the two leads to treating a guess as an emergency, or to shrugging off a certainty.

	What it measures	Values observed
`severity`	how bad, if the deviation is real	`critical`, `high`, `medium`, `low`
`labels.confidence`	how sure the deviation is real	`confirmed` on the example deviations

On the example inventory, the four deviations come out as confirmed: they rest on directly observed data, a public ACL or an explicit firewall rule. No inference is involved.

Confidence is carried by deviations only. A pass has none, and that is consistent: the question "how sure am I that there is nothing" is already answered by the status itself, which is only granted when the data was collected.

Provenance: what is this scan a trace of?

Every result comes with a run block that lets you re-read the scan months later. That is what separates a report from a screenshot.

{
  "tool":    { "name": "pepin", "version": "0.4.0", "digest": "vcs:61e5de9…" },
  "ruleset": { "name": "pepin-config", "digest": "sha256:6eca0627…" },
  "target":  { "id": "scaleway", "provider": "scaleway" },
  "timestamp": "2026-09-09T05:25:56Z",
  "source": "export",
  "scope": {
    "included": ["access_key", "governance_provider",
                 "object_storage_bucket", "security_group_rule"]
  }
}

Four fields are worth reading every time:

ruleset.digest identifies the exact ruleset. Two scans diverging with the same digest point to a change in your infrastructure; with different digests, the comparison is meaningless.
source says where the data came from: an export, a Terraform plan or a live collection. The three do not prove the same thing.
scope.included lists the resource types actually examined. It is the honest counterpart to the verdict: whatever is not there was not looked at.
timestamp dates the observation. A compliant verdict only speaks about the moment it was issued.

The verdict, and what it does not say

The summary of a compliant scan is deliberately careful:

 Verdict : conforme sur le périmètre évalué
 (aucune non-conformité détectée, 12 contrôles conformes)

Three precautions fit inside that sentence. "Within the evaluated scope" refers back to scope.included. "No non-compliance detected" is not "no non-compliance". And the count of twelve compliant controls tells you what the statement rests on, which a global percentage would hide.

The report also closes with a reminder that doubles as a warning:

This report evaluates a tenant's configuration (customer scope). Normative
mappings (SecNumCloud, ISO, CIS) are indicative: they do not constitute proof
of qualification or certification, which applies to the cloud service provider.

Normative mappings

Every result carries its references in structured form rather than free text:

"references": [
  { "framework": "scsl",             "id": "CLD-IAM-2" },
  { "framework": "iso-27001",        "id": "A.5.17" },
  { "framework": "secnumcloud-3.2",  "id": "9.5" }
]

The scsl code belongs to the common referential and points at a published requirement. Since version 0.4.0, the link printed under each deviation resolves correctly to the relevant family page and its anchor:

↳ docs: …/socle/referentiel/cloud/iam-acces-cloud/#socle-cld-iam-2

Choosing an output format

Five formats, three uses. The human-readable output only shows deviations, which is convenient day to day and misleading for an audit, since it stays silent about non-evaluations.

Format	What it is for
`table`	day-to-day human reading, deviations only
`assessment`	the full truth: four statuses, provenance, references
`json`	processing by script
`oscal`	exchange with a compliance tool, in OSCAL 1.1.2
`sarif`	surfacing inside a code review interface

To judge a posture, take assessment. It is the only one showing you what the scan could not look at.

Common pitfalls

Symptom	Cause	What to do
A flattering score while nothing changed	`not-evaluated` counted as compliant	read the status breakdown, not just the verdict
Many `not-evaluated` on a live collection	insufficient read permissions on the API	extend the audit account's policy, then rerun
Two scans that cannot be compared	`ruleset.digest` differs between runs	pin the ruleset before comparing
A suspicious "not applicable"	the justification reveals a responsibility boundary	read the reason, it often points at work of your own
A deviation judged false	severity read as certainty	check `labels.confidence` before deciding

Key points

Two statuses conclude, two refuse to. pass and fail assert; not-applicable and not-evaluated explain why they abstain.
not-evaluated is never a success. On the repository example, 14 controls out of 27 fall in that bucket.
A non-evaluation has two very different causes: the resource is absent from the scope, or the data could not be collected. Only the second calls for action.
Since 0.4.0, a not-applicable justifies itself, and the justification often reveals a boundary of the shared responsibility model.
Severity says how bad, confidence says how sure. The example deviations come out as confirmed, with no inference.
The run block makes a scan re-readable: tool version, ruleset digest, source, examined scope, timestamp.
scope.included is the counterpart to the verdict: whatever is not there was not looked at.
To judge a posture, use assessment, the only format showing all four statuses.

External resources

OSCAL 1.1.2 specification: the exchange model behind the oscal format.
SARIF specification: the format code review interfaces expect.
Pépin on GitHub: the control referential and the full command-line surface.