Specifications
LIVEWritten as specifications rather than as product documentation. A standard is adoptable by academia and referenceable by a regulator only if it can be read, criticised and re-implemented by someone with no access to the implementers. Every rule below is one an independent party could check.
What the standard consists of
Five documents: 2 prose specifications and 6 machine-readable schemas. Each is read from the repository at build time and each is listed with the path it came from, so this page cannot claim a specification the repository does not hold.
| Specification | What it governs | Source | Size |
|---|---|---|---|
| MedEval-1 | The 9 dimensions, how each is computed, what may be claimed from a score, and what a task must provide to be admissible. | plan_d/docs/MEDEVAL-1-SPEC.md | 20 KB |
| Trust Runtime | The policy grammar — twelve operators, five actions — and how a gate reads a signed passport against an inference envelope. | plan_d/docs/RUNTIME-SPEC.md | 42 KB |
| Evidence Passport schema | The wire format of a signed passport, including the computed / declared / observed split that keeps a vendor's assertion from rendering as a measurement. | plan_d/spec/passport-v0.1.schema.json | 11 KB |
| Evidence Passport schema v0.2 | v0.1 byte-for-byte plus two optional top-level members, so every passport signed under v0.1 still validates here and nothing already issued was revalidated. envelope carries the fitted range of acquisition conditions the evidence covers; graded_artefact records which weights were graded. Both sit inside the signed region. The document version stays 0.1 on purpose — a deployed verifier dispatches on that, and an additive block must not break a verifier that has never heard of it. | plan_d/spec/passport-v0.2.schema.json | 34 KB |
| Policy schema v0.3 | Twelve operators. v0.3 adds within_envelope, which refuses or warns when a model is deployed outside the acquisition conditions its evidence covers. It has three states rather than two, because a site marginally outside a band is the common case and a binary gate would either block everyone or block no one — and it distinguishes a site that differs from a sample too small to tell, which are different claims. | plan_d/src/runtime/policy-v0.3.schema.json | 9 KB |
| Policy schema v0.2 | Eleven operators. v0.2 adds exists, which asks whether a path resolves at all, and older_than / newer_than, so a policy can say refuse evidence older than 90 days without arithmetic. Years and months are rejected — no fixed length means a calendar, and a calendar is computation. Superseded by v0.3 and kept byte-unchanged: v0.2 policies evaluate identically under the v0.3 runtime and a conformance check asserts it. | plan_d/src/runtime/policy-v0.2.schema.json | 7 KB |
| Policy schema v0.1 | The previous grammar, eight operators, kept byte-unchanged and still evaluable. A runtime that breaks its own published policies is not infrastructure, so v0.1 policies evaluate identically under v0.2 and a conformance check asserts it. | plan_d/src/runtime/policy-v0.1.schema.json | 5 KB |
| Inference envelope schema | What a caller must present about a single inference for a policy to be evaluable against it. | plan_d/src/runtime/envelope-v0.1.schema.json | 2 KB |
Evidence Passport schema
passport_versionspec_versionissued_utcexpiry_utcexpiry_basiscanonicalisationsubjectcomputeddeclaredobservedissuersignatureThe schema itself
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://nakedsignal.example/spec/passport-v0.1.schema.json",
"title": "MedEval-1 Evidence Passport v0.1",
"description": "A signed, machine-checkable statement of what a model scored, where it breaks, what its vendor claims, and when the evidence expires. Doc 02 of the skeleton plan. The three evidence classes are TOP-LEVEL OBJECTS, not tags on fields: it must be structurally impossible to render a declared field as though it were measured.",
"type": "object",
"additionalProperties": false,
"required": [
"passport_version",
"spec_version",
"issued_utc",
"expiry_utc",
"expiry_basis",
"canonicalisation",
"subject",
"computed",
"declared",
"observed",
"issuer",
"signature"
],
"properties": {
"passport_version": { "const": "0.1" },
"spec_version": {
"type": "string",
"description": "The MedEval-1 spec version the computed evidence was produced under. Copied from the result records, never asserted here."
},
"spec_versions": {
"type": "array",
"items": { "type": "string" },
"description": "Every distinct spec_version across this subject's evaluations. Present whenever the subject spans more than one."
},
"issued_utc": { "type": "string", "format": "date-time" },
"expiry_utc": { "type": "string", "format": "date-time" },
"expiry_basis": {
"type": "string",
"description": "In words, why the passport expires when it does. Doc 02 s8: a passport expires when its held-out generation retires or its spec version is superseded."
},
"canonicalisation": {
"type": "object",
"additionalProperties": false,
"required": ["form", "float_rounding", "signed_over"],
"properties": {
"form": { "type": "string" },
"float_rounding": { "type": "string" },
"signed_over": { "type": "string" }
},
"description": "The signature is over bytes, so the byte-producing rule is part of the document. Doc 02 s4."
},
"subject": {
"type": "object",
"additionalProperties": false,
"required": ["model_id", "model_name", "family", "params", "code_fingerprint", "code_fingerprints"],
"properties": {
"model_id": { "type": "string" },
"model_name": { "type": "string" },
"family": { "type": "string" },
"params": { "type": "string" },
"code_fingerprint": {
"type": ["string", "null"],
"description": "null when the subject's evaluations were produced by more than one code fingerprint; see code_fingerprints."
},
"code_fingerprints": {
"type": "array",
"items": { "type": "string" },
"minItems": 1
}
}
},
"computed": {
"type": "object",
"description": "MEASURED BY NAKEDSIGNAL. Every number here came out of a JSON file the harness wrote.",
"additionalProperties": false,
"required": ["evaluations", "cross_site", "heldout", "audit", "sources"],
"properties": {
"evaluations": {
"type": "array",
"minItems": 1,
"items": {
"type": "object",
"additionalProperties": false,
"required": [
"task", "task_name", "modality", "kind", "domain",
"spec_version", "code_fingerprint", "seed", "device",
"n_test", "n_classes", "split_origin", "timestamp",
"dimensions", "composite", "limitations"
],
"properties": {
"task": { "type": "string" },
"task_name": { "type": "string" },
"modality": { "type": "string" },
"kind": { "type": "string" },
"domain": {
"type": ["string", "null"],
"description": "null when the task registry has not declared one. Never guessed here."
},
"spec_version": { "type": "string" },
"code_fingerprint": { "type": "string" },
"seed": { "type": "integer" },
"device": { "type": "string" },
"n_test": { "type": "integer" },
"n_classes": { "type": "integer" },
"split_origin": { "type": "string" },
"timestamp": { "type": "string" },
"runtime_s": { "type": ["number", "null"] },
"dimensions": {
"type": "object",
"minProperties": 1,
"description": "A whole dimension is null when it does not apply to this task kind (there is no re-acquisition to simulate on a tabular corpus). A dimension's score is null when the dimension ran but could not be scored. Both are disclosures and both are preserved.",
"additionalProperties": {
"type": ["object", "null"],
"if": { "type": "object" },
"then": {
"required": ["score"],
"properties": {
"score": {
"type": ["number", "null"],
"description": "null stays null. Never coerced to 0 -- a null score is a disclosure, not a zero."
}
}
}
}
},
"composite": {
"type": "object",
"additionalProperties": false,
"required": ["index", "coverage", "dimensions_scored"],
"properties": {
"index": { "type": ["number", "null"] },
"coverage": { "type": ["number", "null"] },
"dimensions_scored": { "type": "array", "items": { "type": "string" } }
}
},
"limitations": {
"type": "object",
"additionalProperties": false,
"required": ["train_capped", "n_train_available", "has_real_subgroup_metadata"],
"properties": {
"train_capped": { "type": "boolean" },
"n_train_available": { "type": "integer" },
"has_real_subgroup_metadata": { "type": ["boolean", "null"] },
"subgroup_basis": { "type": ["string", "null"] },
"notes": { "type": ["string", "null"] }
}
}
}
}
},
"cross_site": {
"type": ["object", "null"],
"description": "null when this subject has no cross-site family run. Present-and-null, never absent.",
"required": ["family", "sites", "mean_retention", "worst_retention"],
"properties": {
"family": { "type": "string" },
"sites": { "type": "array", "items": { "type": "string" } },
"site_labels": { "type": "object" },
"matrix": { "type": "object" },
"mean_retention": { "type": ["number", "null"] },
"worst_retention": { "type": ["number", "null"] },
"score": { "type": ["number", "null"] },
"note": { "type": "string" }
}
},
"heldout": {
"type": ["object", "null"],
"description": "This subject's private held-out track entries, if it was submitted. null otherwise.",
"required": ["labelling", "entries"],
"properties": {
"labelling": { "type": "string" },
"entries": { "type": "array" }
}
},
"audit": {
"type": ["object", "null"],
"additionalProperties": false,
"required": ["bundles", "rederived", "ledger_entries", "ledger_head", "ledger_intact", "audit_command"],
"properties": {
"bundles": { "type": "integer" },
"rederived": { "type": "integer" },
"ledger_entries": { "type": "integer" },
"ledger_head": { "type": "string" },
"ledger_intact": { "type": "boolean" },
"audit_command": { "type": "string" }
}
},
"sources": {
"type": "array",
"minItems": 1,
"items": {
"type": "object",
"additionalProperties": false,
"required": ["path", "sha256"],
"properties": {
"path": { "type": "string" },
"sha256": { "type": "string", "pattern": "^[0-9a-f]{64}$" }
}
},
"description": "Every harness file this passport was built from, with its hash. Makes 'no number was hand-typed' checkable rather than asserted."
}
}
},
"declared": {
"type": ["object", "null"],
"description": "DECLARED BY VENDOR. null when there is no vendor -- see declared_note. Read from plan_d/declarations/<model>.yaml; NakedSignal never fills these in.",
"additionalProperties": false,
"required": ["vendor", "training_cutoff", "approved_uses", "not_approved_uses", "regulatory", "attestation"],
"properties": {
"vendor": { "type": "string", "minLength": 1 },
"training_cutoff": { "type": "string", "minLength": 1 },
"approved_uses": { "type": "array", "items": { "type": "string" }, "minItems": 1 },
"not_approved_uses": { "type": "array", "items": { "type": "string" }, "minItems": 1 },
"regulatory": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"required": ["authority", "type", "id", "url"],
"properties": {
"authority": { "type": "string" },
"type": { "type": "string" },
"id": { "type": "string" },
"url": { "type": ["string", "null"] }
}
}
},
"attestation": {
"type": "object",
"additionalProperties": false,
"required": ["signed_by", "role", "date"],
"properties": {
"signed_by": { "type": "string" },
"role": { "type": "string" },
"date": { "type": "string" }
}
}
}
},
"declared_note": { "type": ["string", "null"] },
"observed": {
"type": "null",
"description": "OBSERVED IN DEPLOYMENT. Always null at v0.1: production history and drift require MedEval Live (doc 06). Present-and-null on purpose -- the shape is the roadmap."
},
"observed_note": { "type": "string" },
"issuer": {
"type": "object",
"additionalProperties": false,
"required": ["name", "key_id", "algo", "public_key_b64"],
"properties": {
"name": { "type": "string" },
"key_id": { "type": "string" },
"algo": { "const": "Ed25519" },
"public_key_b64": {
"type": "string",
"description": "Raw 32-byte Ed25519 public key, base64. Convenience only: a verifier MUST check it against the key published in plan_d/docs/GOVERNANCE.md and reject a mismatch, otherwise the document certifies itself."
}
}
},
"signature": {
"type": "string",
"description": "base64 Ed25519 signature over the canonical bytes of this document with the `signature` member removed."
}
}
}
Read from plan_d/spec/passport-v0.1.schema.json at build time, and summarised from the parsed document — the counts above are not typed in.
Evidence Passport schema v0.2
passport_versionspec_versionissued_utcexpiry_utcexpiry_basiscanonicalisationsubjectcomputeddeclaredobservedissuersignatureThe schema itself
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://nakedsignal.example/spec/passport-v0.2.schema.json",
"title": "MedEval-1 Evidence Passport v0.2 (v0.1 plus the acquisition envelope and the graded artefact)",
"description": "v0.1, byte-for-byte, plus TWO optional top-level members: `envelope` and `graded_artefact`. Nothing else differs and nothing was removed, so every passport that validates under v0.1 validates under this. That is the whole design: v0.1 stays on disk and stays the thing already-signed passports were signed under, and this file is a superset validator for the ones that also carry a fitted acquisition envelope, a pinned graded artefact, or both. Each member was added the same way and under the same rule -- optional at the top level, closed and required-and-nullable inside -- so `graded_artefact` is not a second convention but the first one applied twice. Note `passport_version` is still const 0.1 and that is deliberate -- the document version is what a deployed verifier dispatches on, and the shipped verifier (packages/site-core/src/lib/passport/verify.ts) accepts 0.1 only. An additive block must not break a verifier that has never heard of it; signature verification is over bytes and does not parse the block, so an old verifier keeps returning exit 0 on a passport carrying a member it cannot read. Bumping the document version would have broken every one of them to announce fields they would ignore anyway. The schema version and the document version are separate things and this is the case that shows why: \"v0.2\" is the name of a schema, not of a document.",
"type": "object",
"additionalProperties": false,
"required": [
"passport_version",
"spec_version",
"issued_utc",
"expiry_utc",
"expiry_basis",
"canonicalisation",
"subject",
"computed",
"declared",
"observed",
"issuer",
"signature"
],
"properties": {
"passport_version": { "const": "0.1" },
"spec_version": {
"type": "string",
"description": "The MedEval-1 spec version the computed evidence was produced under. Copied from the result records, never asserted here."
},
"spec_versions": {
"type": "array",
"items": { "type": "string" },
"description": "Every distinct spec_version across this subject's evaluations. Present whenever the subject spans more than one."
},
"issued_utc": { "type": "string", "format": "date-time" },
"expiry_utc": { "type": "string", "format": "date-time" },
"expiry_basis": {
"type": "string",
"description": "In words, why the passport expires when it does. Doc 02 s8: a passport expires when its held-out generation retires or its spec version is superseded."
},
"canonicalisation": {
"type": "object",
"additionalProperties": false,
"required": ["form", "float_rounding", "signed_over"],
"properties": {
"form": { "type": "string" },
"float_rounding": { "type": "string" },
"signed_over": { "type": "string" }
},
"description": "The signature is over bytes, so the byte-producing rule is part of the document. Doc 02 s4."
},
"subject": {
"type": "object",
"additionalProperties": false,
"required": ["model_id", "model_name", "family", "params", "code_fingerprint", "code_fingerprints"],
"properties": {
"model_id": { "type": "string" },
"model_name": { "type": "string" },
"family": { "type": "string" },
"params": { "type": "string" },
"code_fingerprint": {
"type": ["string", "null"],
"description": "null when the subject's evaluations were produced by more than one code fingerprint; see code_fingerprints."
},
"code_fingerprints": {
"type": "array",
"items": { "type": "string" },
"minItems": 1
},
"behaviour_version": {
"type": ["string", "null"],
"description": "WHAT THE RECORDS' IDENTITY KEYS ON, as declared by plan_d/src/harness.py:behaviour_version(). The third additive member, and the only one that lives inside an existing block rather than at the top level -- it belongs to the subject because it is a property of the code that produced the subject's evidence, and putting it anywhere else would have made the passport carry two answers to `which code was this`. OPTIONAL, for the reason the other two are: every passport issued before the field existed must keep validating, and absent therefore means `these records predate the split` rather than `nobody checked`. Why it exists at all: `code_fingerprint` above is a sha256 over the scoring code's BYTES, so it moves on a docstring edit and on a newly registered model that cannot change any existing score. Once a subject's records legitimately span two byte hashes under one behaviour -- the healthy steady state after that change -- `code_fingerprint` goes null by the rule above, and a consumer that identified the code through it silently falls back to a bare model id. A passport that stops saying which code produced it is the same failure class as two evaluations a reader cannot tell apart. So the byte hashes stay, in full, in `code_fingerprints`, and this field carries the identity. null means the records DISAGREE about their behaviour version, or some declare one and others do not; that is refused by plan_d/src/passport.py:validate() with a sentence rather than by this schema with a type error, because a split behaviour is a real inconsistency and the reader needs to know which versions were found. A split byte hash is not: it is expected, and `code_fingerprints` states it."
}
}
},
"computed": {
"type": "object",
"description": "MEASURED BY NAKEDSIGNAL. Every number here came out of a JSON file the harness wrote.",
"additionalProperties": false,
"required": ["evaluations", "cross_site", "heldout", "audit", "sources"],
"properties": {
"evaluations": {
"type": "array",
"minItems": 1,
"items": {
"type": "object",
"additionalProperties": false,
"required": [
"task", "task_name", "modality", "kind", "domain",
"spec_version", "code_fingerprint", "seed", "device",
"n_test", "n_classes", "split_origin", "timestamp",
"dimensions", "composite", "limitations"
],
"properties": {
"task": { "type": "string" },
"task_name": { "type": "string" },
"modality": { "type": "string" },
"kind": { "type": "string" },
"domain": {
"type": ["string", "null"],
"description": "null when the task registry has not declared one. Never guessed here."
},
"spec_version": { "type": "string" },
"code_fingerprint": { "type": "string" },
"seed": { "type": "integer" },
"device": { "type": "string" },
"n_test": { "type": "integer" },
"n_classes": { "type": "integer" },
"split_origin": { "type": "string" },
"timestamp": { "type": "string" },
"runtime_s": { "type": ["number", "null"] },
"dimensions": {
"type": "object",
"minProperties": 1,
"description": "A whole dimension is null when it does not apply to this task kind (there is no re-acquisition to simulate on a tabular corpus). A dimension's score is null when the dimension ran but could not be scored. Both are disclosures and both are preserved.",
"additionalProperties": {
"type": ["object", "null"],
"if": { "type": "object" },
"then": {
"required": ["score"],
"properties": {
"score": {
"type": ["number", "null"],
"description": "null stays null. Never coerced to 0 -- a null score is a disclosure, not a zero."
}
}
}
}
},
"composite": {
"type": "object",
"additionalProperties": false,
"required": ["index", "coverage", "dimensions_scored"],
"properties": {
"index": { "type": ["number", "null"] },
"coverage": { "type": ["number", "null"] },
"dimensions_scored": { "type": "array", "items": { "type": "string" } }
}
},
"limitations": {
"type": "object",
"additionalProperties": false,
"required": ["train_capped", "n_train_available", "has_real_subgroup_metadata"],
"properties": {
"train_capped": { "type": "boolean" },
"n_train_available": { "type": "integer" },
"has_real_subgroup_metadata": { "type": ["boolean", "null"] },
"subgroup_basis": { "type": ["string", "null"] },
"notes": { "type": ["string", "null"] }
}
}
}
}
},
"cross_site": {
"type": ["object", "null"],
"description": "null when this subject has no cross-site family run. Present-and-null, never absent.",
"required": ["family", "sites", "mean_retention", "worst_retention"],
"properties": {
"family": { "type": "string" },
"sites": { "type": "array", "items": { "type": "string" } },
"site_labels": { "type": "object" },
"matrix": { "type": "object" },
"mean_retention": { "type": ["number", "null"] },
"worst_retention": { "type": ["number", "null"] },
"score": { "type": ["number", "null"] },
"note": { "type": "string" }
}
},
"heldout": {
"type": ["object", "null"],
"description": "This subject's private held-out track entries, if it was submitted. null otherwise.",
"required": ["labelling", "entries"],
"properties": {
"labelling": { "type": "string" },
"entries": { "type": "array" }
}
},
"audit": {
"type": ["object", "null"],
"additionalProperties": false,
"required": ["bundles", "rederived", "ledger_entries", "ledger_head", "ledger_intact", "audit_command"],
"properties": {
"bundles": { "type": "integer" },
"rederived": { "type": "integer" },
"ledger_entries": { "type": "integer" },
"ledger_head": { "type": "string" },
"ledger_intact": { "type": "boolean" },
"audit_command": { "type": "string" }
}
},
"sources": {
"type": "array",
"minItems": 1,
"items": {
"type": "object",
"additionalProperties": false,
"required": ["path", "sha256"],
"properties": {
"path": { "type": "string" },
"sha256": { "type": "string", "pattern": "^[0-9a-f]{64}$" }
}
},
"description": "Every harness file this passport was built from, with its hash. Makes 'no number was hand-typed' checkable rather than asserted."
}
}
},
"declared": {
"type": ["object", "null"],
"description": "DECLARED BY VENDOR. null when there is no vendor -- see declared_note. Read from plan_d/declarations/<model>.yaml; NakedSignal never fills these in.",
"additionalProperties": false,
"required": ["vendor", "training_cutoff", "approved_uses", "not_approved_uses", "regulatory", "attestation"],
"properties": {
"vendor": { "type": "string", "minLength": 1 },
"training_cutoff": { "type": "string", "minLength": 1 },
"approved_uses": { "type": "array", "items": { "type": "string" }, "minItems": 1 },
"not_approved_uses": { "type": "array", "items": { "type": "string" }, "minItems": 1 },
"regulatory": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"required": ["authority", "type", "id", "url"],
"properties": {
"authority": { "type": "string" },
"type": { "type": "string" },
"id": { "type": "string" },
"url": { "type": ["string", "null"] }
}
}
},
"attestation": {
"type": "object",
"additionalProperties": false,
"required": ["signed_by", "role", "date"],
"properties": {
"signed_by": { "type": "string" },
"role": { "type": "string" },
"date": { "type": "string" }
}
}
}
},
"declared_note": { "type": ["string", "null"] },
"observed": {
"type": "null",
"description": "OBSERVED IN DEPLOYMENT. Always null at v0.1: production history and drift require MedEval Live (doc 06). Present-and-null on purpose -- the shape is the roadmap."
},
"observed_note": { "type": "string" },
"issuer": {
"type": "object",
"additionalProperties": false,
"required": ["name", "key_id", "algo", "public_key_b64"],
"properties": {
"name": { "type": "string" },
"key_id": { "type": "string" },
"algo": { "const": "Ed25519" },
"public_key_b64": {
"type": "string",
"description": "Raw 32-byte Ed25519 public key, base64. Convenience only: a verifier MUST check it against the key published in plan_d/docs/GOVERNANCE.md and reject a mismatch, otherwise the document certifies itself."
}
}
},
"signature": {
"type": "string",
"description": "base64 Ed25519 signature over the canonical bytes of this document with the `signature` member removed."
},
"envelope": {
"type": ["object", "null"],
"description": "THE ACQUISITION ENVELOPE (S6.1). The fitted range of acquisition conditions this model's evidence covers, and the provenance of that fit. OPTIONAL at the top level, and that is the one place in this schema where an absent member is legitimate: a passport issued before the Variation Engine existed does not have one, and the alternative -- requiring the member and permitting null -- would have invalidated every already-signed passport, which is precisely what an additive bump must not do. Inside the block the opposite rule holds: `fidelity` and `off_generator` are REQUIRED and NULLABLE, because absent reads as 'not applicable to this kind of evidence' and null reads as 'applicable, not done'. The block is inside the signed region -- canonical_bytes() drops only `signature` -- so editing any field here invalidates the signature. Every field of the envelope's own hashed body is carried here, so `envelope_id` is RECOMPUTABLE from this block alone rather than being a number the reader is asked to trust; `variation/passport_envelope.py:rebuild_envelope()` does exactly that and the acceptance test asserts it. The two members that are NOT in the envelope's hash are `seed` and `off_generator`: the seed belongs to the sampling run this passport describes, and the off-generator check is evidence gathered after the fit. Both are inside the passport's signature regardless.",
"additionalProperties": false,
"required": [
"envelope_id",
"envelope_version",
"generator",
"generator_version",
"seed",
"pairing_grade",
"calibration_corpus",
"reference_condition",
"target_condition",
"parameters",
"fidelity",
"off_generator"
],
"properties": {
"envelope_id": {
"type": "string",
"pattern": "^[0-9a-f]{12}$",
"description": "sha256 of the envelope's canonical bytes, truncated to 12 hex -- the same truncation harness.code_fingerprint() uses, so the two read alike in one document. Derived, never asserted: a reader recomputes it from the rest of this block."
},
"envelope_version": { "type": "string", "minLength": 1 },
"generator": {
"type": "string",
"minLength": 1,
"description": "The forward model these parameters are parameters OF, e.g. `variation.forward`. A band is meaningless without the generator whose knob it names."
},
"generator_version": { "type": "string", "minLength": 1 },
"seed": {
"type": ["integer", "null"],
"description": "The seed of the sampling run this passport describes. null when nothing was sampled. Not part of envelope_id -- the envelope is a fitted range, not a draw from one."
},
"pairing_grade": {
"enum": ["PAIRED_PIXEL", "PAIRED_SUBJECT", "PAIRED_CLASS", "UNPAIRED_SITE"],
"description": "The pairing ladder, verbatim, from vision/synthetic_programme/BUILD_STATE.md s2. A PROPERTY OF THE DATA, taken off the corpus manifest and not overridable by whoever fills this in. Constrained to four values here because the single easiest way to destroy this programme's credibility is to let a corpus be quoted one rung higher than it sits, and a free-text grade is an invitation to do exactly that."
},
"calibration_corpus": {
"type": "object",
"description": "What the envelope was fitted on, hash-linked. `sha256` is the digest of the corpus manifest file, so 'which data was this fitted on' is checkable rather than remembered. Open to further members (the manifest may carry a pairing_note, per-condition detail and so on) -- they travel inside the signature and inside envelope_id either way.",
"required": ["name", "sha256", "conditions", "n"],
"properties": {
"name": { "type": "string", "minLength": 1 },
"sha256": {
"type": "string",
"pattern": "^[0-9a-f]{64}$",
"description": "sha256 of the corpus manifest's file bytes."
},
"conditions": {
"type": "array",
"items": { "type": "string" },
"minItems": 1,
"description": "The acquisition conditions the corpus holds, e.g. [axial, coronal, sagittal]. reference_condition and target_condition must both be among them."
},
"n": {
"type": ["integer", "null"],
"description": "Total images across the conditions. null when the manifest does not state one -- never 0, which would read as an empty corpus."
},
"pairing_grade": {
"enum": ["PAIRED_PIXEL", "PAIRED_SUBJECT", "PAIRED_CLASS", "UNPAIRED_SITE"],
"description": "The corpus's own grade. Where present it must equal the envelope's; the corpus wins."
},
"pairing_note": { "type": "string" },
"manifest": { "type": "string" }
}
},
"reference_condition": {
"type": "string",
"minLength": 1,
"description": "The condition the fit maps FROM. Must be one of calibration_corpus.conditions."
},
"target_condition": {
"type": "string",
"minLength": 1,
"description": "The condition the fit maps TO. Must be one of calibration_corpus.conditions."
},
"edge_tolerance": {
"type": "number",
"minimum": 0,
"maximum": 1,
"description": "How far outside the fitted interval still counts as `edge` rather than `outside`, as a fraction of the interval width. Default 0.25. It lives on the envelope, inside envelope_id and inside the signature ON PURPOSE: if the party being gated could pick it, the gate would be negotiable after the fact. Optional here only because Envelope.from_json() supplies the same default, so envelope_id stays recomputable when it is absent."
},
"parameters": {
"type": "array",
"minItems": 1,
"description": "The fitted bands, one per generator parameter. Sorted by name so that two envelopes meaning the same thing hash the same; physical stage order lives in each entry's `stage`.",
"items": {
"type": "object",
"additionalProperties": false,
"required": ["name", "stage", "unit", "point", "ci_low", "ci_high"],
"properties": {
"name": { "type": "string", "minLength": 1 },
"stage": {
"type": "string",
"minLength": 1,
"description": "Which stage of the forward model this knob sits in, e.g. optics / geometry / response / grain / quantisation / compression."
},
"unit": {
"type": "string",
"description": "May be empty for a dimensionless parameter, but the member is present so that 'dimensionless' is stated rather than left to the reader."
},
"point": { "type": "number" },
"ci_low": { "type": "number" },
"ci_high": { "type": "number" },
"floor": {
"type": ["number", "null"],
"description": "The most conservative low end a sceptical reader may take, stated once by whoever did the fit. At or below ci_low, never inside the interval. Where stated it REPLACES the tolerance factor on the low side -- a considered judgement outranks a default. null when unstated."
},
"note": {
"type": "string",
"description": "Where 'this number is doing something the field names do not say' goes. Inside the hash."
}
}
}
},
"fidelity": {
"type": ["object", "null"],
"description": "The S2 realism audit: does a perturbation drawn from this envelope resemble the real acquisition change? REQUIRED AND NULLABLE. null means the audit has not run, which is a different statement from 'ran and came out weak' -- an absent member would have read as 'not applicable', and there is no model for which this is not applicable. Each headline number is separately nullable for the same reason: the pixel half and the behaviour half can land at different times. Open to further members so the audit's supporting detail can travel without a schema bump; the five headlines are pinned.",
"required": ["pixel_auc", "pixel_auc_ci", "behaviour_rho", "behaviour_rho_ci", "residual"],
"properties": {
"pixel_auc": {
"type": ["number", "null"],
"minimum": 0,
"maximum": 1,
"description": "Held-out discriminator AUC on the physics feature vector, synthetic-vs-real. 0.5 is indistinguishable; higher is separable, i.e. WORSE. Report it whichever way it comes out."
},
"pixel_auc_ci": {
"type": ["array", "null"],
"items": { "type": "number" },
"minItems": 2,
"maxItems": 2,
"description": "[low, high]. null when the estimate carries no interval."
},
"behaviour_rho": {
"type": ["number", "null"],
"minimum": -1,
"maximum": 1,
"description": "Rank correlation between per-model retention under synthetic shift and under the real acquisition change. The sign is load-bearing: negative is a stronger negative result than zero, and the range here permits it because the schema must be able to express the finding that would embarrass its author."
},
"behaviour_rho_ci": {
"type": ["array", "null"],
"items": { "type": "number" },
"minItems": 2,
"maxItems": 2
},
"residual": {
"type": ["object", "null"],
"description": "The learned residual (S2.4 / S3.2), where one was fitted. null where none was. A residual trains only on calibration pairs and its hash-level disjointness from every sealed set is asserted in CI, not promised in prose."
},
"note": { "type": "string" },
"source": {
"type": "string",
"description": "The audit artifact these numbers were copied out of. No number in a passport is typed by hand."
}
}
},
"off_generator": {
"type": ["object", "null"],
"description": "The S5 check: does the envelope help on a condition it was NOT fitted on? REQUIRED AND NULLABLE, same rule as fidelity -- null is 'the loop has not been closed', absent would be 'the question does not arise', and it always arises. This is the block that separates a calibrated generator from a self-graded one, so it is stated explicitly even when the answer is 'not yet'. Open to further members (n, note, per-condition detail).",
"required": ["condition", "axis", "subject_disjoint", "gain", "gap"],
"properties": {
"condition": {
"type": "string",
"minLength": 1,
"description": "The held-out condition tested against, e.g. `sagittal`. It must NOT be the target_condition the envelope was fitted on -- that is what off-generator means."
},
"axis": {
"type": "string",
"minLength": 1,
"description": "The axis it is held out along: acquisition plane, institution, scanner, device generation."
},
"subject_disjoint": {
"type": ["boolean", "null"],
"description": "Whether the held-out condition shares no subject with the calibration corpus. null where the corpus carries no subject identifiers and disjointness therefore cannot be ASSERTED -- which is the case for organ{A,C,S}mnist. false and null are different claims and a gate may treat them differently; neither may be rendered as true."
},
"gain": {
"type": ["number", "null"],
"description": "How much of the real gap the envelope closes on the held-out condition. Negative is permitted and is a real outcome: a generator can move a model further from the truth."
},
"gap": {
"type": ["number", "null"],
"description": "What is left over after the gain. Reported alongside rather than instead, so a large gain on a larger gap cannot be quoted alone."
},
"n": { "type": ["integer", "null"] },
"note": { "type": "string" }
}
},
"provenance": {
"type": "array",
"items": { "type": "string" },
"description": "How this envelope came to exist, in the order it happened. Inside envelope_id."
},
"known_stale": {
"type": "object",
"additionalProperties": { "type": "string" },
"description": "{key: reason} for values deliberately NOT corrected. An interface with no field for 'known stale, deliberately, because --' will look like it fixed something someone chose not to fix."
},
"caveats": {
"type": "array",
"items": { "type": "string" },
"description": "What this envelope does not cover, in the fitter's own words, inside the signature so it cannot be dropped downstream by whoever finds it inconvenient."
}
}
},
"graded_artefact": {
"type": ["object", "null"],
"description": "WHAT WAS GRADED. The bytes and the code this passport's numbers came out of: the weights hash, the adapter shape, the task list, the code fingerprint, the moment. Written by plan_d/src/grade_model.py when it grades a model it has never seen. OPTIONAL at the top level, for the same reason `envelope` is and no other: a passport built by the in-house sweep pins its subject through plan_d/src/models.py, which is inside the code fingerprint already, and every passport issued before this member existed must keep validating. Absent therefore means `no external artefact was handed over`; it does NOT mean `nobody checked`. Inside the block the opposite rule holds and every member is REQUIRED -- nullable where the adapter shape genuinely cannot supply one, never omitted. This block exists because it previously did not: grade_model.py wrote the same fields to an unsigned `.graded-artifact.json` sidecar beside the passport, so `this passport is about these weights` was an unsigned claim sitting next to a signed one, and a sidecar is exactly the artefact a reader quotes as though it were attested. Here the block is inside the signed region -- canonical_bytes() drops only `signature` -- so editing any field invalidates the signature and the sidecar is no longer written. ⚠️ What this block attests is WHAT WAS GRADED, not that the weights are authentic: the grader runs what it is handed and cannot check that a file is the model its vendor says it is. Two passports claiming to be about the same model can be checked against each other here; neither can be checked against reality here.",
"additionalProperties": false,
"required": [
"model_id",
"adapter_shape",
"arch",
"weights_sha256",
"weights_filename",
"predictions_sha256",
"code_fingerprint",
"tasks",
"graded_utc",
"note"
],
"properties": {
"model_id": {
"type": "string",
"minLength": 1,
"description": "The subject this artefact was graded as. It MUST equal subject.model_id -- a block naming a different model would let one passport's evidence be quoted as another's, which is the single worst thing this member could be used for. JSON Schema cannot express the cross-reference, so plan_d/src/passport.py:validate() asserts it beyond the schema, in the same place the duplicate-task-id check lives."
},
"adapter_shape": {
"type": "string",
"minLength": 1,
"description": "How the model was reached: `timm`, `open_clip`, `predictions`, or `python:<module>:<Class>` for a caller's own Adapter. Free text rather than an enum because the python: form carries the caller's module path, and an enum that cannot express the fourth shape would push it into a fifth value meaning `other`. It caps what could be measured -- a prediction table cannot be perturbed or refitted -- so it belongs next to the coverage it explains."
},
"arch": {
"type": ["string", "null"],
"description": "The architecture the weights were loaded into, e.g. `vit_base_patch16_224`. REQUIRED AND NULLABLE: null where the adapter shape does not take one (a prediction table has no architecture; a caller's own Adapter class chooses its own). null is `the question does not arise for this shape`, and it is stated rather than left to be inferred from the shape by a reader who may not know the four shapes."
},
"weights_sha256": {
"type": ["string", "null"],
"pattern": "^[0-9a-f]{64}$",
"description": "sha256 of the weights file as handed over. REQUIRED AND NULLABLE: null means no weights file was supplied, which is the honest state of the `predictions` and `python:` shapes. Both this and predictions_sha256 may be null at once -- a caller's own Adapter can pin nothing hashable -- and the block says so rather than being omitted, because a reader must be able to tell `nothing was hashable` from `nobody recorded it`."
},
"weights_filename": {
"type": ["string", "null"],
"description": "The basename of the weights file, for a human reading the passport. Null exactly when weights_sha256 is null. Carried alongside the hash and never instead of it: a filename is a label a vendor chose, not evidence."
},
"predictions_sha256": {
"type": ["string", "null"],
"pattern": "^[0-9a-f]{64}$",
"description": "sha256 of the submitted probability file, for the `predictions` shape. REQUIRED AND NULLABLE, same rule as weights_sha256. This is the only artefact that exists on the path a counterparty who will not send weights actually uses, so it is the hash that will most often be the one there is."
},
"code_fingerprint": {
"type": "string",
"pattern": "^[0-9a-f]{12}$",
"description": "harness.code_fingerprint() at the moment of grading -- sha256 over the scoring code, truncated to 12 hex, the same truncation subject.code_fingerprint uses so the two read alike in one document. ⛔ It covers the CALLING CODE. It does not cover the graded weights, and it does not cover the data: a training-set cap set in an adapter changes the score from outside this hash. Do not describe it as covering either."
},
"tasks": {
"type": "array",
"minItems": 1,
"items": { "type": "string" },
"description": "The task ids this grading run was asked to score, as requested rather than as survived: a task that failed still appears here. That is deliberate -- computed.evaluations lists what scored, and the difference between the two lists is how a reader sees that something was attempted and did not come back. A block that quietly narrowed to the successes would make a partial run look like a complete one."
},
"graded_utc": {
"type": "string",
"format": "date-time",
"description": "When the grading run happened, UTC, to the second. Distinct from issued_utc: a passport can be re-issued from records without re-running the model, and the two dates drifting apart is information rather than an inconsistency."
},
"note": {
"type": "string",
"minLength": 1,
"description": "The standing caveat, carried in the document rather than in a README. REQUIRED and non-empty on purpose: it is the sentence that keeps this block from being read as an authenticity claim, and the one thing a downstream quoter has an incentive to drop. Inside the signature, so dropping it is detectable."
}
}
}
}
}
Read from plan_d/spec/passport-v0.2.schema.json at build time, and summarised from the parsed document — the counts above are not typed in.
Policy schema v0.3
schema_versionpolicypolicy_versionauthorexampledefault_actionrulesThe schema itself
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://nakedsignal.dev/spec/policy-v0.3.schema.json",
"title": "NakedSignal Trust Runtime policy, v0.3",
"description": "A policy is data, not code, so it can be reviewed, diffed, version-controlled, signed and shown to a regulator. v0.3 adds ONE operator to v0.2 — within_envelope, the acquisition-envelope gate — and changes nothing else. v0.1 and v0.2 policies remain valid against their own schemas and evaluate identically under this runtime; only a policy's declared schema_version selects which schema it is checked against. See plan_d/docs/RUNTIME-SPEC.md 3.4.10.",
"type": "object",
"additionalProperties": false,
"required": [
"schema_version",
"policy",
"policy_version",
"author",
"example",
"default_action",
"rules"
],
"properties": {
"schema_version": { "const": "nakedsignal-policy-0.3" },
"policy": {
"type": "string",
"minLength": 1,
"description": "Stable identifier for this policy."
},
"policy_version": { "type": "string", "minLength": 1 },
"author": {
"type": "string",
"minLength": 1,
"description": "Who authored these thresholds. Required. Policies shipped by NakedSignal must contain the word EXAMPLE here; validate_policy() enforces that when example is true."
},
"example": {
"type": "boolean",
"description": "REQUIRED, and required for a regulatory reason (RUNTIME-SPEC 2.2). true means this policy is a worked illustration, not a NakedSignal recommendation and not a default. There is no third state and no way to omit the question."
},
"description": { "type": "string" },
"binding_time": {
"enum": ["inference", "procurement"],
"default": "inference",
"description": "inference: evaluated per request, reads inference.* and passport.*. procurement: evaluated once at purchase, reads passport.* only. Same engine (RUNTIME-SPEC 8)."
},
"mode": {
"enum": ["log_only", "enforce"],
"default": "log_only",
"description": "An absent mode is log_only, in both implementations. Under log_only the decision is computed and recorded in full but effective_action is always log_only and nothing is blocked."
},
"default_action": {
"$ref": "#/$defs/action",
"description": "Applied when no rule matches."
},
"rules": {
"type": "array",
"items": { "$ref": "#/$defs/rule" },
"description": "First matching rule wins. Evaluation order is file order."
}
},
"$defs": {
"action": {
"enum": ["allow", "defer_to_human", "route_to_fallback", "reject", "log_only"]
},
"path": {
"type": "string",
"pattern": "^(passport|inference|runtime)(\\.[A-Za-z0-9_-]+)+$",
"description": "Dotted path into exactly one of the three namespaces. Any other root is a validation error, so a policy cannot reach into the host program."
},
"duration": {
"type": "string",
"pattern": "^P(?=\\d|T\\d)(?:\\d+W)?(?:\\d+D)?(?:T(?=\\d)(?:\\d+H)?(?:\\d+M)?(?:\\d+S)?)?(?![\\s\\S])",
"description": "An ISO-8601 duration restricted to the components that have a fixed length: weeks, days, hours, minutes, seconds, integer valued, in canonical order, at least one of them. Years and months are REJECTED on purpose — neither has a fixed length, so P3M would need a calendar and a calendar is computation (RUNTIME-SPEC 3.1). Write P90D. Examples: P90D, P2W, PT12H, P1DT6H30M."
},
"envelope_verdict": {
"enum": [
"within",
"warn",
"outside",
"warn:marginal",
"warn:unresolved_at_n",
"warn:incomplete"
],
"description": "Which answer this rule is asking about. `outside` is the only verdict that is a statement about the site strong enough to refuse on. `warn` matches any warn; the three qualified forms match one reason each, and the reasons are NOT interchangeable — `marginal` means the feature was placed and sits just outside the band (look at the scanner), `unresolved_at_n` means the sample could not place it at all (upload more images), `incomplete` means the fingerprint answered no reading for a band the envelope declares (fix the client). A policy that routed all three the same way would tell a hospital its scanner is unusual when the truth is that it uploaded fifty images. An omitted value means `within`, and `within` is the absence of every other state rather than a count of `inside` — see RUNTIME-SPEC 3.4.10."
},
"rule": {
"type": "object",
"additionalProperties": false,
"required": ["id", "when", "then"],
"properties": {
"id": { "type": "string", "minLength": 1 },
"when": { "$ref": "#/$defs/condition" },
"then": { "$ref": "#/$defs/action" },
"because": {
"type": "string",
"description": "Logged verbatim on every decision this rule produces. This is how the customer audits their own policy."
}
}
},
"condition": {
"oneOf": [
{ "$ref": "#/$defs/all" },
{ "$ref": "#/$defs/any" },
{ "$ref": "#/$defs/not" },
{ "$ref": "#/$defs/leaf" }
]
},
"all": {
"type": "object",
"additionalProperties": false,
"required": ["all"],
"properties": {
"all": { "type": "array", "items": { "$ref": "#/$defs/condition" } }
}
},
"any": {
"type": "object",
"additionalProperties": false,
"required": ["any"],
"properties": {
"any": { "type": "array", "items": { "$ref": "#/$defs/condition" } }
}
},
"not": {
"type": "object",
"additionalProperties": false,
"required": ["not"],
"properties": {
"not": { "$ref": "#/$defs/condition" }
}
},
"leaf": {
"type": "object",
"additionalProperties": false,
"required": ["path", "op"],
"properties": {
"path": { "$ref": "#/$defs/path" },
"op": {
"enum": [
"<",
"<=",
">",
">=",
"==",
"!=",
"in",
"not_in",
"exists",
"older_than",
"newer_than",
"within_envelope"
],
"description": "The whole operator set. No arithmetic, no loops, no user functions IN A POLICY. The last three ordering-free ones arrived in v0.2; within_envelope is new in v0.3 and is the only operator that reads two documents rather than two values."
},
"value": {
"description": "A JSON literal. For the eight v0.1 operators, exactly one of value or ref. For exists it is an optional boolean naming the polarity. For older_than / newer_than it is a required ISO-8601 duration. For within_envelope it is an optional verdict selector; omitted, it means `within`, so the bare operator reads as its own name and a `warn` is not a pass."
},
"ref": {
"$ref": "#/$defs/path",
"description": "A dotted path resolved at evaluation time. Exactly one of value or ref for the eight v0.1 operators. Not permitted on exists, older_than or newer_than. REQUIRED on within_envelope, where it names the passport envelope the fingerprint at `path` is gated against — that operator is the one place a leaf legitimately carries both ref and value, because its two arguments are two documents and its selector is a third thing."
},
"as": {
"const": "timestamp",
"description": "Compare both sides as ISO-8601 instants rather than as strings. Date rules using an ordering operator should always set this. Not permitted on exists, older_than or newer_than — those already parse instants by definition."
}
},
"allOf": [
{
"title": "the eight v0.1 operators take exactly one of value or ref",
"if": {
"required": ["op"],
"properties": {
"op": {
"enum": ["<", "<=", ">", ">=", "==", "!=", "in", "not_in"]
}
}
},
"then": {
"oneOf": [{ "required": ["value"] }, { "required": ["ref"] }]
}
},
{
"title": "exists takes a path and, optionally, a boolean polarity",
"if": {
"required": ["op"],
"properties": { "op": { "const": "exists" } }
},
"then": {
"properties": { "value": { "type": "boolean" } },
"not": {
"anyOf": [{ "required": ["ref"] }, { "required": ["as"] }]
}
}
},
{
"title": "the temporal operators take a path and a duration literal",
"if": {
"required": ["op"],
"properties": {
"op": { "enum": ["older_than", "newer_than"] }
}
},
"then": {
"required": ["value"],
"properties": { "value": { "$ref": "#/$defs/duration" } },
"not": {
"anyOf": [{ "required": ["ref"] }, { "required": ["as"] }]
}
}
},
{
"title": "within_envelope takes a fingerprint path, an envelope ref and an optional verdict selector",
"if": {
"required": ["op"],
"properties": { "op": { "const": "within_envelope" } }
},
"then": {
"required": ["ref"],
"properties": { "value": { "$ref": "#/$defs/envelope_verdict" } },
"not": { "required": ["as"] }
}
}
]
}
}
}
Read from plan_d/src/runtime/policy-v0.3.schema.json at build time, and summarised from the parsed document — the counts above are not typed in.
Policy schema v0.2
schema_versionpolicypolicy_versionauthorexampledefault_actionrulesThe schema itself
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://nakedsignal.dev/spec/policy-v0.2.schema.json",
"title": "NakedSignal Trust Runtime policy, v0.2",
"description": "A policy is data, not code, so it can be reviewed, diffed, version-controlled, signed and shown to a regulator. v0.2 adds three operators to v0.1 — exists, older_than, newer_than — and changes nothing else. A v0.1 policy is still valid v0.1 and evaluates identically under this runtime; only its declared schema_version selects which schema it is checked against. See plan_d/docs/RUNTIME-SPEC.md.",
"type": "object",
"additionalProperties": false,
"required": [
"schema_version",
"policy",
"policy_version",
"author",
"example",
"default_action",
"rules"
],
"properties": {
"schema_version": { "const": "nakedsignal-policy-0.2" },
"policy": {
"type": "string",
"minLength": 1,
"description": "Stable identifier for this policy."
},
"policy_version": { "type": "string", "minLength": 1 },
"author": {
"type": "string",
"minLength": 1,
"description": "Who authored these thresholds. Required. Policies shipped by NakedSignal must contain the word EXAMPLE here; validate_policy() enforces that when example is true."
},
"example": {
"type": "boolean",
"description": "REQUIRED, and required for a regulatory reason (RUNTIME-SPEC 2.2). true means this policy is a worked illustration, not a NakedSignal recommendation and not a default. There is no third state and no way to omit the question."
},
"description": { "type": "string" },
"binding_time": {
"enum": ["inference", "procurement"],
"default": "inference",
"description": "inference: evaluated per request, reads inference.* and passport.*. procurement: evaluated once at purchase, reads passport.* only. Same engine (RUNTIME-SPEC 8)."
},
"mode": {
"enum": ["log_only", "enforce"],
"default": "log_only",
"description": "An absent mode is log_only, in both implementations. Under log_only the decision is computed and recorded in full but effective_action is always log_only and nothing is blocked."
},
"default_action": {
"$ref": "#/$defs/action",
"description": "Applied when no rule matches."
},
"rules": {
"type": "array",
"items": { "$ref": "#/$defs/rule" },
"description": "First matching rule wins. Evaluation order is file order."
}
},
"$defs": {
"action": {
"enum": ["allow", "defer_to_human", "route_to_fallback", "reject", "log_only"]
},
"path": {
"type": "string",
"pattern": "^(passport|inference|runtime)(\\.[A-Za-z0-9_-]+)+$",
"description": "Dotted path into exactly one of the three namespaces. Any other root is a validation error, so a policy cannot reach into the host program."
},
"duration": {
"type": "string",
"pattern": "^P(?=\\d|T\\d)(?:\\d+W)?(?:\\d+D)?(?:T(?=\\d)(?:\\d+H)?(?:\\d+M)?(?:\\d+S)?)?(?![\\s\\S])",
"description": "An ISO-8601 duration restricted to the components that have a fixed length: weeks, days, hours, minutes, seconds, integer valued, in canonical order, at least one of them. Years and months are REJECTED on purpose — neither has a fixed length, so P3M would need a calendar and a calendar is computation (RUNTIME-SPEC 3.1). Write P90D. Examples: P90D, P2W, PT12H, P1DT6H30M."
},
"rule": {
"type": "object",
"additionalProperties": false,
"required": ["id", "when", "then"],
"properties": {
"id": { "type": "string", "minLength": 1 },
"when": { "$ref": "#/$defs/condition" },
"then": { "$ref": "#/$defs/action" },
"because": {
"type": "string",
"description": "Logged verbatim on every decision this rule produces. This is how the customer audits their own policy."
}
}
},
"condition": {
"oneOf": [
{ "$ref": "#/$defs/all" },
{ "$ref": "#/$defs/any" },
{ "$ref": "#/$defs/not" },
{ "$ref": "#/$defs/leaf" }
]
},
"all": {
"type": "object",
"additionalProperties": false,
"required": ["all"],
"properties": {
"all": { "type": "array", "items": { "$ref": "#/$defs/condition" } }
}
},
"any": {
"type": "object",
"additionalProperties": false,
"required": ["any"],
"properties": {
"any": { "type": "array", "items": { "$ref": "#/$defs/condition" } }
}
},
"not": {
"type": "object",
"additionalProperties": false,
"required": ["not"],
"properties": {
"not": { "$ref": "#/$defs/condition" }
}
},
"leaf": {
"type": "object",
"additionalProperties": false,
"required": ["path", "op"],
"properties": {
"path": { "$ref": "#/$defs/path" },
"op": {
"enum": [
"<",
"<=",
">",
">=",
"==",
"!=",
"in",
"not_in",
"exists",
"older_than",
"newer_than"
],
"description": "The whole operator set. No arithmetic, no loops, no user functions. The last three are new in v0.2."
},
"value": {
"description": "A JSON literal. For the eight v0.1 operators, exactly one of value or ref. For exists it is an optional boolean naming the polarity. For older_than / newer_than it is a required ISO-8601 duration."
},
"ref": {
"$ref": "#/$defs/path",
"description": "A dotted path resolved at evaluation time. Exactly one of value or ref. Not permitted on exists, older_than or newer_than."
},
"as": {
"const": "timestamp",
"description": "Compare both sides as ISO-8601 instants rather than as strings. Date rules using an ordering operator should always set this. Not permitted on exists, older_than or newer_than — those already parse instants by definition."
}
},
"allOf": [
{
"title": "the eight v0.1 operators take exactly one of value or ref",
"if": {
"required": ["op"],
"properties": {
"op": {
"enum": ["<", "<=", ">", ">=", "==", "!=", "in", "not_in"]
}
}
},
"then": {
"oneOf": [{ "required": ["value"] }, { "required": ["ref"] }]
}
},
{
"title": "exists takes a path and, optionally, a boolean polarity",
"if": {
"required": ["op"],
"properties": { "op": { "const": "exists" } }
},
"then": {
"properties": { "value": { "type": "boolean" } },
"not": {
"anyOf": [{ "required": ["ref"] }, { "required": ["as"] }]
}
}
},
{
"title": "the temporal operators take a path and a duration literal",
"if": {
"required": ["op"],
"properties": {
"op": { "enum": ["older_than", "newer_than"] }
}
},
"then": {
"required": ["value"],
"properties": { "value": { "$ref": "#/$defs/duration" } },
"not": {
"anyOf": [{ "required": ["ref"] }, { "required": ["as"] }]
}
}
}
]
}
}
}
Read from plan_d/src/runtime/policy-v0.2.schema.json at build time, and summarised from the parsed document — the counts above are not typed in.
Policy schema v0.1
schema_versionpolicypolicy_versionauthorexampledefault_actionrulesThe schema itself
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://nakedsignal.dev/spec/policy-v0.1.schema.json",
"title": "NakedSignal Trust Runtime policy, v0.1",
"description": "A policy is data, not code, so it can be reviewed, diffed, version-controlled, signed and shown to a regulator. See plan_d/docs/RUNTIME-SPEC.md.",
"type": "object",
"additionalProperties": false,
"required": [
"schema_version",
"policy",
"policy_version",
"author",
"example",
"default_action",
"rules"
],
"properties": {
"schema_version": { "const": "nakedsignal-policy-0.1" },
"policy": {
"type": "string",
"minLength": 1,
"description": "Stable identifier for this policy."
},
"policy_version": { "type": "string", "minLength": 1 },
"author": {
"type": "string",
"minLength": 1,
"description": "Who authored these thresholds. Required. Policies shipped by NakedSignal must contain the word EXAMPLE here; validate_policy() enforces that when example is true."
},
"example": {
"type": "boolean",
"description": "REQUIRED, and required for a regulatory reason (RUNTIME-SPEC 2.2). true means this policy is a worked illustration, not a NakedSignal recommendation and not a default. There is no third state and no way to omit the question."
},
"description": { "type": "string" },
"binding_time": {
"enum": ["inference", "procurement"],
"default": "inference",
"description": "inference: evaluated per request, reads inference.* and passport.*. procurement: evaluated once at purchase, reads passport.* only. Same engine (RUNTIME-SPEC 8)."
},
"mode": {
"enum": ["log_only", "enforce"],
"default": "log_only",
"description": "An absent mode is log_only, in both implementations. Under log_only the decision is computed and recorded in full but effective_action is always log_only and nothing is blocked."
},
"default_action": {
"$ref": "#/$defs/action",
"description": "Applied when no rule matches."
},
"rules": {
"type": "array",
"items": { "$ref": "#/$defs/rule" },
"description": "First matching rule wins. Evaluation order is file order."
}
},
"$defs": {
"action": {
"enum": ["allow", "defer_to_human", "route_to_fallback", "reject", "log_only"]
},
"path": {
"type": "string",
"pattern": "^(passport|inference|runtime)(\\.[A-Za-z0-9_-]+)+$",
"description": "Dotted path into exactly one of the three namespaces. Any other root is a validation error, so a policy cannot reach into the host program."
},
"rule": {
"type": "object",
"additionalProperties": false,
"required": ["id", "when", "then"],
"properties": {
"id": { "type": "string", "minLength": 1 },
"when": { "$ref": "#/$defs/condition" },
"then": { "$ref": "#/$defs/action" },
"because": {
"type": "string",
"description": "Logged verbatim on every decision this rule produces. This is how the customer audits their own policy."
}
}
},
"condition": {
"oneOf": [
{ "$ref": "#/$defs/all" },
{ "$ref": "#/$defs/any" },
{ "$ref": "#/$defs/not" },
{ "$ref": "#/$defs/leaf" }
]
},
"all": {
"type": "object",
"additionalProperties": false,
"required": ["all"],
"properties": {
"all": { "type": "array", "items": { "$ref": "#/$defs/condition" } }
}
},
"any": {
"type": "object",
"additionalProperties": false,
"required": ["any"],
"properties": {
"any": { "type": "array", "items": { "$ref": "#/$defs/condition" } }
}
},
"not": {
"type": "object",
"additionalProperties": false,
"required": ["not"],
"properties": {
"not": { "$ref": "#/$defs/condition" }
}
},
"leaf": {
"type": "object",
"additionalProperties": false,
"required": ["path", "op"],
"properties": {
"path": { "$ref": "#/$defs/path" },
"op": {
"enum": ["<", "<=", ">", ">=", "==", "!=", "in", "not_in"],
"description": "The whole operator set. No arithmetic, no loops, no user functions."
},
"value": {
"description": "A JSON literal. Exactly one of value or ref."
},
"ref": {
"$ref": "#/$defs/path",
"description": "A dotted path resolved at evaluation time. Exactly one of value or ref."
},
"as": {
"const": "timestamp",
"description": "Compare both sides as ISO-8601 instants rather than as strings. Date rules should always set this."
}
},
"oneOf": [
{ "required": ["value"] },
{ "required": ["ref"] }
]
}
}
}
Read from plan_d/src/runtime/policy-v0.1.schema.json at build time, and summarised from the parsed document — the counts above are not typed in.
Inference envelope schema
model_idThe schema itself
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://nakedsignal.dev/spec/envelope-v0.1.schema.json",
"title": "NakedSignal inference envelope, v0.1",
"description": "What the caller supplies per inference. Kept minimal, because every field is an integration cost for the customer. Only model_id is required; an absent field behaves per RUNTIME-SPEC 3.4, which means a caller that does not report an OOD score does not get to satisfy an OOD rule.",
"type": "object",
"additionalProperties": true,
"required": ["model_id"],
"properties": {
"model_id": { "type": "string", "minLength": 1 },
"confidence": {
"type": "number",
"minimum": 0,
"maximum": 1,
"description": "The model's own confidence in this output, as it reports it. The runtime does not recalibrate it."
},
"ood_score": {
"type": "number",
"minimum": 0,
"maximum": 1,
"description": "Caller-supplied out-of-distribution score. Higher means further from the training regime."
},
"site_profile": {
"type": "string",
"description": "Acquisition setting, e.g. a scanner model. Compared against passport.computed.validated_settings."
},
"subgroup": {
"type": "string",
"description": "Patient subgroup this inference concerns. Compared against passport.computed.validated_subgroups."
},
"indication": { "type": "string" },
"ts": {
"type": "string",
"description": "ISO-8601 timestamp of the inference."
}
}
}
Read from plan_d/src/runtime/envelope-v0.1.schema.json at build time, and summarised from the parsed document — the counts above are not typed in.
MedEval-1 — Specification
Version 0.1 · 28 July 2026 · status: draft, open for comment
A standard for measuring medical AI systems: what is measured, how it is measured, how it is reported, and what makes a result admissible.
This document is deliberately written as a specification rather than as product documentation. A standard is adoptable by academia and referenceable by a regulator only if it can be read, criticised and re-implemented by someone with no access to the implementers. Every rule below is one an independent party could check.
1. Scope and principles
1.1 Scope
MedEval-1 evaluates clinical prediction systems: models that take a clinical input (image, waveform, tabular record, text) and produce a diagnostic, prognostic or classificatory output with an associated confidence.
Out of scope in v0.1, specified for later versions: generative clinical text, segmentation and detection with spatial ground truth, longitudinal prediction, agentic or multi-step clinical workflows.
1.2 The five principles
- Held-out means held out. The test split is read by exactly one operation — producing predictions from a finalised model. No fitting, no hyperparameter choice, no early-stopping decision, no threshold selection may consult it.
- A score must decompose. Every headline number resolves to the raw measurements underneath it. A rating that cannot be taken apart is an opinion.
- Reproducible from the manifest. Seed, code fingerprint, split sizes, device and dataset version are recorded with every result. A third party with the manifest and the code reproduces the number.
- Failure is a result. A model that cannot be evaluated on a task is published as a failure on that task, by name. It is never a zero, and never a silent omission.
- Disclose the compute. Where a budget constrains the evaluation — a training-set cap, a test subsample for perturbation scoring — the constraint is recorded in the result row. Undisclosed budget limits are how benchmarks quietly become unfair.
1.3 Versioning
MedEval-1 v<major>.<minor>. Dimension definitions, normalisation transforms and composite weights are frozen within a major version. Scores from different major versions are not comparable and must not be plotted on the same axis. The spec version appears on every result record and on the leaderboard.
2. The seven dimensions
🔴 This document describes seven dimensions. The instrument currently scores nine, and the version has not yet been bumped to say so. Read this section as the `v0.1` definition, not as a description of the live scorer. What is actually running. Every run since the 1 Aug sweep scores nine dimensions and weights the composite over all nine: the seven below, unrenamed, plusspec_sensitivity("does the score survive a faithful rewording of the task?") andcost("what does one prediction cost?"). That the seven original keys survived unrenamed is load-bearing — it is what let MedEval-W claim modality invariance against a live table rather than a copied constant. And five of the seven weights below moved when 8 and 9 landed — accuracy .25→.22, acquisition .20→.17, calibration .15→.13, limited_data .10→.08, corruption .15→.14 — whilespec_versionstayedMedEval-1 v0.1. §1 of this document says composite weights are frozen within a major version, so two indices computed under different weightings are both published as v0.1, which is exactly what a version exists to prevent.tests/regressions/dimension-count.spec.tsfails on purpose until the call is made. ⭐ The consequence is now measured rather than argued, so this is a ten-minute decision. Recomputing every published index under the original seven weights: all 196 indices move (mean |Δ| 0.60, max 4.92), two boards move one model by one place, and no board winner changes on any board. Records that measure neither new dimension still move, because the five surviving weights were rescaled non-proportionally. Full table:experiments/benchmark_ext/results/e4_spec_version.json, harness verified by reproducing all 196 published indices to 4.3 × 10⁻¹⁴ before any counterfactual. So the two live options are: bump to v0.2, state the nine weights here, and record that v0.1 and v0.2 indices are not comparable while rankings survive — or re-freeze the original seven and give 8 and 9 their own additive block. This note does not choose. It exists so that the document stops asserting something untrue while the choice is open. ⚠️ One thing to settle in the same decision:spec_sensitivityisnot_measurableon 184 of 196 records — scored on 12, all of them one model — andcoston 151 of 196, with 91% of those sitting exactly on the 100.0 cap and a within-task spread of 0.08 points against 13.2 for accuracy. Both still carry weight, and renormalisation hands any instrumented model +1.77 index points, enough to move a text-board rank.DIMENSIONS-DECISION.mdalready calls a dimension that is null nearly everywhere "a footnote wearing a weight"; the size of the footnote is now known.
Each dimension produces raw measurements plus one normalised score in [0, 100].
Chance correction. All accuracy-like quantities are mapped by cc(x) = max(0, (x − 1/k) / (1 − 1/k)) where k is the number of classes. Without it, a binary task and an eleven-class task cannot appear on the same axis and a composite index degenerates into a measure of how many classes a task has.
`k` is declared, not observed. It comes from the task's — or the sealed set's — shape declaration, never from the labels that happen to appear in a split. Deriving it from the data would score a split that drew three of eight classes as an easier task than the one it belongs to, and would hand anyone minting a split a lever on the number.
On the held-out track the published score is an integer. score_micro — millionths of a point, computed from counts by integer arithmetic under one stated rounding rule (half up, at every step, working in units of 10⁻¹²) — is the canonical value, and the float is a rendering of it. Metric version MedEval-1 v0.2 D1 normalisation, integer-exact. A number independent parties are expected to reproduce should not depend on the last bits of a float64 reduction, or on which BLAS the auditor happens to have installed. The re-expression was executed against every historical run before it landed: worst disagreement 4.9 × 10⁻⁷ points, which is the final rounding to micro-points and nothing else, and no published number moved at displayed precision. The rest of the harness still reports float64 — the discipline is applied where numbers are attested rather than everywhere at once, and that is a deliberate scope, not an oversight.
| # | Key | Dimension | Normalised score |
|---|---|---|---|
| 1 | accuracy | Task accuracy | cc(balanced accuracy) |
| 2 | acquisition | Cross-site robustness | mean retention under acquisition shift |
| 3 | calibration | Calibration | 1 − ECE / 0.20, floored at 0 |
| 4 | limited_data | Limited data | mean retention at 5 / 20 / 100 labels per class |
| 5 | subgroup | Subgroup safety | cc(worst-group performance) |
| 6 | corruption | Corruption / OOD | mean retention under degradation |
| 7 | uncertainty | Uncertainty & deferral | cc(area under the accuracy–coverage curve) |
2.1 Task accuracy
Balanced accuracy on the full official held-out split; accuracy, macro one-vs-rest AUC and a 200-resample percentile bootstrap 95% confidence interval are reported alongside.
Balanced accuracy rather than accuracy is the headline because clinical class balance is usually severe, and accuracy on an imbalanced task rewards a model for ignoring the rare class — which is generally the class that matters.
2.2 Cross-site robustness
The question: does the model survive the same patient being imaged somewhere else?
Two protocols, in descending order of evidential strength:
- Protocol A — paired multi-site (definitive). The same subjects, imaged on different scanners, sites or acquisition protocols. Score = retention of chance-corrected balanced accuracy on the shifted site relative to the source site, plus prediction agreement (Cohen's κ) case-by-case. This requires paired data and is what the private track exists to supply.
- Protocol B — cross-acquisition transfer (available in public data). Datasets sharing a source population and label space but differing in acquisition. Train on one, test on the others; score = mean off-diagonal retention over the transfer matrix.
- Protocol C — simulated re-acquisition (v0 fallback, weakest). Deterministic transforms that emulate a different imaging chain rather than a degraded one: gamma response, window/level preset, reconstruction resolution, field of view, detector noise floor. Applied at three severities to a seeded subsample. Labels are unchanged by construction, so every prediction flip is site sensitivity.
A result must state which protocol produced it. They are not interchangeable, and a Protocol C number must never be presented as if it were Protocol A. The v0 leaderboard runs Protocol B on the organ CT family and Protocol C everywhere else, and labels both.
2.2.1 Retention is not admissible on its own
Retention is a ratio against a model's own clean performance. A model that barely works has little to lose and therefore retains a large fraction of it — so a retention column, read alone, ranks weak models highly. This is not a defect in the measure: retention is the right way to isolate robustness as a property distinct from accuracy. It is a defect in reading it in isolation.
Two requirements follow, both binding:
- Retention is always published beside absolute retained performance — chance-corrected performance under shift, reported as robust skill — and beside task accuracy.
- No ranking may be produced on retention alone. The composite index weights accuracy (0.25) above robustness (0.20) precisely so that a model cannot climb by being uniformly poor.
The same caveat applies verbatim to dimension 6.
2.3 Calibration
Expected calibration error over 15 equal-width bins of top-1 confidence, plus the multiclass Brier score and the overconfidence gap (mean confidence − accuracy).
A miscalibrated model is not a cosmetic problem in clinical use: it is the difference between a system that can be deferred to a clinician on a threshold and one that cannot.
2.4 Limited data
The model is refitted from scratch at 5, 20 and 100 labels per class (seeded, stratified) and rescored on the full held-out split. Score = mean retention of chance-corrected performance relative to the full-data fit.
This is the dimension that matters most for rare disease and for any site that cannot produce ten thousand labelled cases.
2.5 Subgroup safety
Where the source supplies real patient attributes, balanced accuracy is computed per group, and the score uses the worst group; the max−min gap is reported. Groups with fewer than 20 held-out cases are dropped — a disparity computed on eight patients is noise wearing the costume of a finding.
Where the source supplies no attributes, the score falls back to worst diagnostic class recall, and the result is labelled with the fallback basis.
A finding, not a limitation. Almost no public medical imaging corpus ships demographic metadata:has_real_metadataisfalseon every imaging task in this suite. So the public imaging data the field benchmarks on cannot support the subgroup audit that regulators are increasingly going to require. ⚠️ Scope it to imaging, not to medicine. Sixteen records in the suite do carry real attributes, including the tabular clinical setsdiabetes_pimaandheart_cleveland. Richly-annotated private data is what closes the imaging gap — seeHELDOUT-PROTOCOL.md§2.
2.6 Corruption / OOD
Seven deterministic degradations — Gaussian noise, shot noise, defocus blur, contrast, brightness, pixelation, bit-depth reduction — at three severities. Score = mean retention of chance-corrected balanced accuracy.
Distinct from dimension 2 by intent: these make the image worse, whereas dimension 2 makes it different. A model can be robust to one and not the other, and the distinction is clinically meaningful — a noisy scan gets rejected and repeated, whereas a scan from a different vendor's machine looks perfectly fine and is silently mis-read.
2.7 Uncertainty and deferral
Area under the accuracy–coverage curve (sweeping coverage from 5% to 100% by descending confidence), plus selective accuracy at 80% and 50% coverage.
This is the discriminative analogue of hallucination control: a system that knows when it does not know can defer the remainder to a clinician, which is the only safe deployment mode for most clinical AI. Generative hallucination and safety metrics are specified for v1.0 and not scored in v0.1 — no generative model is evaluated in this release.
3. Composite index
index = Σ w_d · score_d / Σ w_d over dimensions actually measured
| Dimension | Weight |
|---|---|
| Task accuracy | 0.25 |
| Cross-site robustness | 0.20 |
| Calibration | 0.15 |
| Corruption / OOD | 0.15 |
| Limited data | 0.10 |
| Subgroup safety | 0.10 |
| Uncertainty & deferral | 0.05 |
Weights are fixed by spec version and published before results. A standard that tunes its weights after seeing the leaderboard is not a standard.
Coverage is published with the index. When a dimension is not measurable for a task, the weights are renormalised over what was measured and the surviving weight fraction is reported. A model scored on five axes is never silently compared with one scored on seven.
Models are ranked within a modality class. An imaging model and a text model are never averaged into a single number.
4. Tasks and data
4.1 Admissibility
A task is admissible if it has: a citable public source or a documented private provenance; a defined label space; a test split that is either official, or minted by the standard's operator and disclosed as minted; and a licence permitting evaluation use.
4.2 Splits
Official splits are used unmodified where they exist. Where a source ships no split, the operator mints one (stratified, seeded, disclosed as minted). Validation is used for model selection and calibration only.
Training-set caps are permitted for compute reasons, must be stratified and seeded, and must be recorded in every affected result row. The test split is never capped.
4.3 Contamination status
Every task carries a contamination status: clean (private, never published), at-risk (public, plausibly in pretraining corpora), or known-contaminated.
Every public task in v0.1 is `at-risk` by default. This is not a hedge — for any model pretrained on web-scale data after the dataset's publication, the honest position is that contamination cannot be excluded. It is the central argument for the private track.
4.4 Restricted-use data
Data under a use agreement that restricts processing is not admitted to the harness. Specifically, corpora whose DUA prohibits processing by third-party AI tooling — PPMI's DUA v5.0 is the working example — may never be loaded, cached or transmitted by any component of this system. Where such data is evaluated at all, it is evaluated offline by the data holder and enters the standard as a score with a provenance attestation, never as records.
This is a governance requirement, not a technical one, and it generalises: as a standard operating on other people's clinical data, the ability to state precisely what was processed, where, and under what authority is a precondition of being trusted with any of it.
5. Models
5.1 The adapter contract
A model enters the standard through an adapter exposing exactly:
fit(x_train, y_train, x_val, y_val, n_classes) -> fitted predict_proba(x) -> (n, k) row-stochastic describe() -> identity and configuration
The scorer knows nothing else about the model. This is what allows a competitor's model, a partner's model and the operator's own model to be scored on identical terms, with no adapter holding privileged access to anything.
5.2 Submission integrity
For private evaluation, the model is executed against sealed data by the operator, or in a sealed environment with logged I/O. The submitter never receives the test cases. Model weights need not be disclosed; the interface must be.
5.3 Declared configuration
Adapters declare their training recipe (augmentation, epochs, initialisation). Augmentation in particular is a robustness defence: a standard whose own baselines all use it can no longer measure whether a submitted model has one. Every baseline in v0.1 is trained without augmentation, and says so.
6. Reporting
6.1 The result record
One record per (task, model), carrying: task and model identity, spec version, code fingerprint, seed, device, split sizes, class count, training cap status, split origin, runtime, per-dimension raw measurements and scores, the composite index with its coverage, and any error.
6.2 The report card
The public unit of reporting is the report card, not a single number: seven dimension scores, per-task detail with confidence intervals, and the failure profile across perturbations. A leaderboard position is a summary of a report card, never a substitute.
6.3 Claims discipline
Results are reported with the protocol that produced them (§2.2), the contamination status of the task (§4.3), and any compute constraint (§4.2). A result that omits these is not a MedEval-1 result.
7. Governance
Summarised here; specified in docs/GOVERNANCE.md.
The standard's credibility rests on the separation between the party that sets the exam and the parties that sit it. The requirements are: an oversight board with a majority independent of the operator; published methodology with a comment period before each major version; external audit rights over the evaluation pipeline; a published conflict-of- interest register; and a rule that any change to a scoring definition takes effect only in a subsequent version.
The operator may submit its own models. The operator may not score them differently, and the independence of the pipeline must be externally verifiable — which is the point of §1.3, §4.2 and §6.1 in combination.
8. Reference implementation
The v0.1 reference implementation accompanies this document.
src/tasks.py task registry src/adapters/ dataset adapters (MedMNIST, tabular, text) src/models.py model adapters src/metrics.py scoring primitives src/corruptions.py deterministic perturbations src/dimensions.py the seven dimension evaluators src/harness.py orchestration, caching, provenance src/run_all.py sweep runner src/build_site.py leaderboard renderer src/validate_published.py external agreement check
Reproduce: python src/run_all.py && python src/build_site.py.
Conformance for v0.1 requires: official splits used unmodified; test data untouched by fitting; every record carrying a manifest per §6.1; dimensions computed per §2; the composite computed per §3 with coverage published; and every result labelled with its protocol, contamination status and compute constraints.
9. Open questions for v1.0
- Should the composite index exist at all, or should the standard publish only report cards? A single number drives adoption and invites gaming; this is the LIBOR trade-off in miniature.
- Weight setting — expert elicitation, or clinical-outcome-derived weights?
- Segmentation and detection: how do spatial metrics enter a cross-modality composite?
- Generative safety: hallucination, sycophancy and unsafe-advice metrics for clinical text.
- Continuous surveillance: how often must a rating be re-run before it goes stale, and what is the drift trigger?
- Multi-site: the minimum paired-data design that makes Protocol A statistically sound.
Comments to the operator; each will be answered in public before v1.0 is frozen.
Rendered from plan_d/docs/MEDEVAL-1-SPEC.mdat build time. Checked against the hosting spec’s scrub grep before rendering: zero hits.
Trust Runtime — specification v0.3
Status: PROTOTYPE. The evaluator, the policy schema, the decision log and the CLI exist and run. Nothing in this repo gates a real inference, and nothing here has been reviewed by counsel.
What changed in v0.3. One operator — within_envelope — and a third schema file. Nothing else. It is the acquisition-envelope gate: it refuses, or warns, when a model is deployed outside the conditions its evidence covers, and it is the first operator whose answer has three states. §3.4.10 states its semantics and §3.4.11 states what it measured on real data, which is not what the product story assumed.
What changed in v0.2. Three operators — exists, older_than, newer_than — and a second schema file. Nothing else. The eight v0.1 operators have identical semantics, v0.1 policies validate against the v0.1 schema forever, and every v0.1 fixture passes unchanged. §3.6 states the compatibility rule and how it is checked.
Implementation: plan_d/src/runtime/ (Python, reference) and packages/site-core/src/lib/runtime/evaluate.ts (TypeScript, browser port). Source spec: vision/skeleton_plan/03_RUNTIME.md.
1. What the runtime is
A function, not a service:
(passport, inference_envelope, policy) -> Decision
It sits in front of an inference and returns one of five actions. Version 0.1 is a library that a hospital's own engineers can read in an afternoon. It does not proxy traffic, it is not hosted, and it does not execute the fallback it names.
2. Regulatory posture — the constraint this design is built around
Software that intercepts a clinical inference and decides whether a clinician sees it is influencing clinical decision-making. Under FDA's device-software framing and the EU AI Act's high-risk classification for AI in medical devices, that is plausibly in scope, which would mean this "neutral infrastructure" needs clearance of its own.
Four design commitments follow, and they are enforced by the code, not by convention:
2.1 No built-in clinical defaults. Ever.
The runtime holds no thresholds of its own. There is no fallback policy, no "recommended" confidence floor, no baseline OOD limit compiled into the evaluator. evaluate() requires a policy argument and fails without one:
evaluate(policy=None, ...) # ValueError: no policy supplied
Every number that influences a decision came out of a file the customer wrote. The runtime's opinion is the empty set. If a NakedSignal engineer ever wants to add a default threshold, that is a change to this section first.
2.2 Shipped policies are examples, in a separate directory, marked as examples
Example policies live in site-data/policies/ and nowhere else. They are never loaded implicitly — the CLI takes --policy <path> with no default, and the /runtime page presents them as a picker of examples, not as configuration.
The schema makes the labelling structural rather than cosmetic:
exampleis a required boolean on every policy. A policy that does not say whether it is an example does not validate.authoris required. Every example we ship sets it to"EXAMPLE — not a NakedSignal recommendation".validate_policy()refuses any policy withexample: truewhoseauthorstring does not contain the wordEXAMPLE.
In deployment the policy is authored by the operating institution, with its thresholds, reviewed under its clinical governance.
2.3 log_only is the default mode
mode has two values, log_only and enforce. A policy that omits `mode` is evaluated as `log_only`, in both implementations. Under log_only the decision is computed and recorded in full — the action, the rule, the reason — and the effective_action returned to the caller is log_only. Nothing is blocked, deferred or rerouted.
A gate that observes and records is unambiguously not a device. It is also what a hospital actually wants first: months of shadow logs are the evidence that the policy is sane before it is allowed to stop anything.
Turning on enforcement is a deliberate, visible, single-field edit to the customer's own policy file. It is never a runtime flag, an environment variable or a default.
2.4 Every decision logs the rule that fired
A Decision always carries rule_id and because. When no rule matched, rule_id is null and because states that the policy's default_action was applied. The customer can therefore audit their own policy from the log alone, without re-running anything.
2.5 Still open
Get counsel on the classification question before anything gates a real inference (doc 06 §6). Until then: log_only, examples only, no real traffic.
3. The policy language
Declarative. A policy is data, so it can be reviewed, diffed, version-controlled, signed and shown to a regulator. Canonical serialisation is JSON (plan_d/src/runtime/policy-v0.1.schema.json, policy-v0.2.schema.json and policy-v0.3.schema.json), because the same bytes are parsed by the Python evaluator and fetched by the browser evaluator, and a second surface syntax is a second thing that can drift. YAML authoring is a post-Friday convenience, not a format.
{
"schema_version": "nakedsignal-policy-0.2",
"policy": "example-conservative-radiology",
"policy_version": "0.1",
"author": "EXAMPLE — not a NakedSignal recommendation",
"example": true,
"binding_time": "inference",
"mode": "log_only",
"default_action": "allow",
"rules": [
{
"id": "low-confidence-ood",
"when": { "all": [
{ "path": "inference.confidence", "op": "<", "value": 0.82 },
{ "path": "inference.ood_score", "op": ">", "value": 0.25 }
]},
"then": "defer_to_human",
"because": "model has not demonstrated calibrated confidence in this regime"
}
]
}3.1 Grammar
Deliberately small. No arithmetic, no loops, no user functions. A policy language that can compute is a policy language that can surprise you.
| Form | Shape | |
|---|---|---|
| conjunction | { "all": [ condition, … ] } — empty array is true | |
| disjunction | { "any": [ condition, … ] } — empty array is false | |
| negation | { "not": condition } | |
| leaf | `{ "path": …, "op": …, "value" \ | "ref": …, "as"?: "timestamp" }` |
Operators — twelve, and nothing else.
| Operator | Since | Right-hand side | Shape |
|---|---|---|---|
< <= > >= | v0.1 | value or ref | ordering, numbers or strings |
== != | v0.1 | value or ref | strict equality |
in not_in | v0.1 | value or ref | set membership, fails closed |
exists | v0.2 | optional boolean value | is the path resolvable at all |
older_than newer_than | v0.2 | required duration value | age of an instant against runtime.now |
within_envelope | v0.3 | required ref, optional verdict value | is this site inside the conditions the evidence covers |
For the eight v0.1 operators a leaf carries exactly one of value (a JSON literal) or ref (a dotted path resolved at evaluation time). ref is what makes inference.site_profile not_in passport.computed.validated_settings expressible without adding syntax.
The three v0.2 operators take a path and a literal, never a ref, and never as — the schema forbids both on them, and the evaluator settles them before it looks at the right-hand side at all.
{ "path": "passport.declared.vendor", "op": "exists" } // present?
{ "path": "passport.declared.vendor", "op": "exists", "value": false } // absent?
{ "path": "passport.issued_utc", "op": "older_than", "value": "P90D" } // stale?
{ "path": "passport.issued_utc", "op": "newer_than", "value": "P90D" } // fresh?within_envelope is the one place a leaf legitimately carries both ref and value, because its two arguments are two documents and its selector is a third thing. path is the site's fingerprint, ref is the passport's envelope, and value names which of the three verdicts this rule is asking about. An omitted value means within, so the bare operator reads as its own name and a warn is not a pass.
{ "path": "inference.site_fingerprint", "op": "within_envelope",
"ref": "passport.envelope" } // inside it?
{ "path": "inference.site_fingerprint", "op": "within_envelope",
"ref": "passport.envelope", "value": "outside" } // outside it?
{ "path": "inference.site_fingerprint", "op": "within_envelope",
"ref": "passport.envelope", "value": "warn:unresolved_at_n" } // too few images?Durations. An ISO-8601 duration restricted to the components that have a fixed length: W D T H M S, integer valued, in canonical order, at least one of them. P90D, P2W, PT12H, P1DT6H30M.
Years and months are rejected on purpose. Neither has a fixed length, so P3M would need a calendar, and a calendar is computation — the thing §3.1 exists to keep out. The schema rejects P3M at validation time with a message saying so; write P90D. Bare P and PT, a dangling T (P1DT), fractional components and lower case are all rejected as well, by the schema and by both evaluators, in agreement.
Still, deliberately: no arithmetic, no loops, no user functions. older_than reads one instant and subtracts it from another instant the runtime already holds; it does not let a policy compute a value and then test it.
`within_envelope` is the operator that most looks like an exception, and is not one. Under the hood it walks 38 bands and does three multiplications each. But the prohibition here is that a policy cannot compute: a policy names two paths and one word, and it cannot choose the features, supply a weight, tune a tolerance or vary the aggregation. Every number in that arithmetic comes off the publisher's signed envelope, and the roll-up has no threshold in it at all (§3.4.10). An operator is allowed to have a definition; what a policy is not allowed to have is a program. The line is the same one older_than sits on — it hides a subtraction and a duration parse — only further along it.
3.2 Namespaces
Paths are dotted and must begin with one of exactly three roots. Any other root is a validation error, so a policy cannot reach into the host program.
| Root | Supplied by | Contents | |
|---|---|---|---|
passport. | the signed Evidence Passport (doc 02 §3) | the whole passport document | |
inference. | the caller, per inference | the inference envelope, §4 | |
runtime. | the runtime | now (ISO-8601 Z), signature_valid (bool\ | null), passport_present (bool) |
An all-digit path segment indexes an array: a passport's computed.evaluations is a list, so passport.computed.evaluations.0.dimensions.accuracy.auc resolves. That is the whole of the array support — there is no way to iterate, filter or aggregate, because that would be computation.
runtime.signature_valid is an input, not a claim the evaluator makes. The runtime does not verify signatures; verify_passport.py (doc 02) does, and the caller passes the result in. Two jobs, two modules.
3.3 Actions
allow · defer_to_human · route_to_fallback · reject · log_only.
First matching rule wins. Evaluation order is file order. Nothing matched falls through to default_action. The runtime does not perform any of these — it returns the word. Executing a fallback route is the caller's job and is explicitly out of scope this week.
3.4 Evaluation semantics, stated exactly
Both implementations must agree bit for bit, so the awkward cases are pinned here rather than left to whichever language you read first.
- Missing path. A path that does not resolve yields
MISSING.MISSINGis distinct from JSONnull; JSONnullis also treated asMISSINGfor comparison purposes, because a passport field that is present-and-null is a disclosure of absent evidence, not a value. - Comparisons with `MISSING` are `false`, and the path is added to the decision's
unresolvedlist. A rule cannot fire on evidence that is not there. - `in` / `not_in` fail closed. The right-hand collection, if
MISSINGornull, is treated as the empty list. So an absentpassport.computed.validated_settingsmakesnot_intrue and the conservative rule fires. Absence of evidence of validation is not evidence of validation. A non-list right-hand side is treated as a one-element list. AMISSINGleft-hand side givesin→false,not_in→true, and is recorded as unresolved. - Types. Number↔number compares numerically. String↔string compares lexicographically by code point (policies are expected to be ASCII; see §7.2). Booleans support only
==and!=. Mixed types:==isfalse,!=istrue, ordering operators arefalse+ unresolved. - Timestamps.
"as": "timestamp"parses both sides as ISO-8601 and compares as instants. Unparseable either side →false+ unresolved. Withoutas, two ISO strings compare as strings, which is correct for identical formats and wrong for mixed offsets — so date rules should always setas. - Determinism.
nowis an argument, never read from the system clock insideevaluate(). The CLI supplies it; fixtures pin it. - Neither `all` nor `any` short-circuits. Every branch is evaluated, so the
unresolvedlist reflects everything the policy reached for and the two implementations cannot drift on evaluation order. Rule selection does stop at the first match, per §3.3. This holds for the v0.2 operators too: in{"any": [exists false, older_than P90D]}the temporal branch runs even when theexistsbranch has already decided the rule, and records the date it could not read.
- `exists` (v0.2) is true when the path resolves to anything other than `MISSING`, and false otherwise — including for a present-and-null field, which is
MISSINGby (1). An optional booleanvaluenames the polarity:value: falseasks is this absent? and inverts the answer. Omittingvaluemeanstrue.refandasare not permitted.
`exists` is the one operator that never adds to `unresolved`. Presence is the question, so a path that does not resolve is the answer rather than a failure. Every other operator treats an unreadable path as something it could not do; exists treats it as something it just did.
- `older_than` / `newer_than` (v0.2) parse the path as an ISO-8601 instant and compare its age against
runtime.now, whereage = now − path.valueis a required duration literal (§3.1).
older_thanisage > span.newer_thanisage < span. Both are strict, so at exactly the span neither is true, and the two are not complements at the boundary. Fixturesolder-than-exact-boundary-is-falseandnewer-than-exact-boundary-is-falsepin it.- A future instant has a negative age:
older_thanfalse,newer_thantrue. - The reference instant is always `runtime.now`, never the system clock, per (6). To compare against some other instant, use an ordering operator with
"as": "timestamp"and aref— that is what it is for. - `MISSING` path → `false`, and the path is added to `unresolved`. These are comparisons, so they join the ordering operators on the fail-open side of the asymmetry below.
- Unparseable path value → `false` + `unresolved`. Identical treatment to
MISSING: a date that says"last Tuesday"is not a date. - Unparseable duration literal → `false` + `unresolved` on the path. The schema rejects a malformed duration before evaluation, so this is a defensive path rather than a route an author can reach through
validate_policy. It is specified anyway, and pinned by a fixture, because an implementation that guessedP3M ≈ 30 dayswould be silently wrong rather than loudly wrong. - Unparseable `runtime.now` → `false`, with
runtime.nowitself added tounresolved.
- `within_envelope` (v0.3) asks whether a site sits inside the acquisition conditions a passport's evidence covers.
pathresolves to the site's fingerprint — the 38-column physics feature vectorvariation/fingerprint.pycomputes client-side from the site's own images — andrefresolves to the passport'senvelopeblock. It is the only operator that reads two documents rather than two values, and the only one whose answer has three states.
Why three. A site marginally outside a band is the common case, so a two-state gate either blocks everyone or blocks no one. warn is not decoration and it is not a softened outside.
Per band, the fingerprint's own 95% median interval [f_lo, f_hi] is compared against the band [ci_low, ci_high] and its grace region — interval against interval, not point against interval, which matters because a fifty-image point puts a site that is inside by construction outside on 9 of 38 features. The grace region is edge_high = ci_high + edge_tolerance × (ci_high − ci_low), and edge_low is the parameter's stated floor if it has one, else the mirror of edge_high. `edge_tolerance` is read off the envelope, defaulting to 0.25 only when the block omits the member: it lives inside the envelope's hash and inside the passport's signature precisely so that the party being gated cannot renegotiate the gate after the fact. That is not a runtime threshold in the §2.1 sense — the runtime supplies no number of its own.
| Per-band state | Condition |
|---|---|
inside | ci_low <= f_lo and f_hi <= ci_high |
outside | f_hi < edge_low, or f_lo > edge_high |
edge/marginal | f_lo >= ci_low and f_hi <= edge_high, or f_hi <= ci_high and f_lo >= edge_low |
edge/unresolved_at_n | everything else — the interval straddles a boundary and this many images cannot place the feature |
incomplete | the fingerprint carries no usable interval for a band the envelope declares |
The roll-up, and the order is the design. Outside → outside. Otherwise incomplete → warn/incomplete. Otherwise unresolved → warn/unresolved_at_n. Otherwise marginal → warn/marginal. Otherwise within. There is no count, no ratio and no threshold anywhere in that chain.
The two warn reasons are not interchangeable and must never be merged. marginal means the feature was placed and sits just outside the band — a statement about the site, and the action is to look at the scanner. unresolved_at_n means the sample could not place it at all — a statement about the sample, and the action is to upload more images. The evidence reasons outrank the site reason on purpose: while a cheaper explanation for the disagreement is still on the table, "your scanner is unusual" is a claim the sample does not support. A gate that reported them the same way would tell a hospital its scanner is unusual when the truth is that it uploaded fifty images.
`within` is never inferred from a count of `inside`. It is the absence of every other state. The measurement that forces this: against the envelope fitted on axial and sagittal, held-out sagittal — a condition that envelope covers — reads inside on 22 of 38 features, while coronal, which it does not cover, reads 29. A condition that defines a band endpoint lands on both sides of it by construction, so a rule that scored coverage by counting inside would have ranked the uncovered condition above the covered one. Read within as "nothing here places this site outside the conditions the evidence covers", never as "this site is certified covered". By the same token a warn/marginal from a boundary condition is the expected reading and is not evidence of drift.
Not comparable → `false` + `unresolved` on both paths. The operator returns no verdict when the fingerprint's feature_vector_version differs from the envelope's generator_version, when a band is malformed, when edge_tolerance is outside [0, 1], or when either operand is not an object. The version guard is what stops a generator envelope — bands over gamma and psf_sigma_px — from being gated against a physics feature vector; that comparison would look completely normal and mean nothing. Which document is wrong is not something the operator can know, so both go on the record.
It fails open, like every other comparison and unlike in/not_in. An absent envelope is the legitimate case, not the exceptional one: envelope is optional at the top of the passport v0.2 schema because every passport issued before the Variation Engine existed has none. *So the gate cannot be made to fire by removing the evidence* — which is worth saying out loud, because the party that supplies the passport is the party the gate constrains. The way out is the one §3.4 already gives for a missing date:
``jsonc { "any": [ { "path": "passport.envelope", "op": "exists", "value": false }, { "path": "inference.site_fingerprint", "op": "within_envelope", "ref": "passport.envelope", "value": "outside" } ]} ``
Fixtures a-passport-with-no-envelope-cannot-be-gated and exists-makes-the-envelope-gate-fail-closed pin both halves, and the-fail-closed-pairing-still-passes-a-covered-site pins that fail-closed does not mean fail-always.
The reason is in the `rule_id`, not in the Decision. The Decision's shape is unchanged by v0.3, deliberately — a new member would have moved the canonical bytes of every decision ever logged. A policy that cares about the difference between the three warn reasons writes one rule per reason, and then rule_id and because say which one fired, which is what §2.4 was for. fixture-acquisition-envelope is that policy.
- What `within_envelope` measured, because a gate that has never been pointed at real data is a claim rather than a component.
variation/out/within_envelope_organ.jsonruns this operator over every fingerprint S6.2 measured, against both fitted envelopes.
- At the full corpus it separates cleanly. Every condition the envelope covers reads
warn; every condition it does not cover readsoutside, on exactly one feature of 38 —contrast_local_sdfor sagittal,geom_radius_gyrationfor coronal. One feature is enough, becauseoutsideoutranks everything. - No real condition reaches `within` at any sample size. The covered conditions define the band's own endpoints, so a held-out draw of one straddles it.
withinis reachable — a constructed fingerprint sitting on the envelope's own point estimates reaches it — and on this corpus the gate's working discrimination is betweenwarnandoutside. - 🔴 At fifty images it does not work, and that is the number the product story rests on. Over 200 seeded fifty-image draws per condition: 135–171 draws return
warn/unresolved_at_n, and the rest returnoutside. It never returnswithinand never returnswarn/marginal. A condition the envelope's own evidence covers is refused in 29–46 draws of 200 — between 14% and 23%. An uncovered one is refused in 51–65. The rates do separate, but they are the same order of magnitude and the product asks for one upload, not two hundred. - How many images it actually needs, measured on a ladder: the covered conditions stop being falsely refused between 500 and 1,000 images; one uncovered condition reaches unanimous refusal only at 5,000, and the other does not reach it anywhere on the ladder.
This is a property of the envelope, not of the operator. S6.2 already measured why: fifty images resolve 0–1 of 38 features because the band spans two reconstruction planes of the same 201 CT volumes — near neighbours in feature space rather than two institutions. A genuine UNPAIRED_SITE envelope over two hospitals would be far wider and the same fifty images would resolve far more of it. The honest reading is that the operator is ready and the calibration corpus is not, and the artifact says so in finding_at_n_50.the_product_consequence rather than leaving it to be discovered.
Note the asymmetry in (2), (3) and (9), because it is a real limitation rather than an oversight: set membership fails closed, comparison fails open. A missing collection makes not_in fire; a missing number makes < not fire; a missing date makes older_than not fire. So a passport too thin to judge slips past a threshold rule and falls through to default_action. Fixture thin-passport-falls-through exists to make that visible rather than surprising, and still behaves exactly as it did in v0.1.
v0.2 does not change that asymmetry — it gives the author a way out of it. The fix is not to redefine what < does to a missing number; policies already in production depend on that. The fix is exists, which lets a rule say and I checked:
{ "any": [
{ "path": "passport.issued_utc", "op": "exists", "value": false },
{ "path": "passport.issued_utc", "op": "older_than", "value": "P90D" } ]}That rule fires on an undated passport where the temporal test alone would have let it through. Fixture exists-makes-a-date-rule-fail-closed. Choosing per rule is the point: a policy author decides which of their own thresholds are safety-critical enough to fail closed, and the runtime holds no opinion (§2.1).
3.5 What exists replaced
v0.1 had no exists, and the documented workaround inside its grammar was
{ "not": { "path": "passport.declared.vendor", "op": ">=", "value": "" } }It still works, unchanged, and it is still wrong in a way worth naming. It is true when the path is missing — but also when the path is present and holds anything that is not a string, because a mixed-type ordering comparison is false by (4) and the not inverts it. So a passport declaring "vendor": 4471 is read as having declared nothing.
Three fixtures pin all of this: v01-workaround-still-detects-absence (the trick still detects absence), v01-workaround-misreads-a-non-string (and still misreads a number as absence), and exists-reads-a-non-string-correctly (v0.2 does not). A fourth difference is visible in the decision itself: the workaround leaves the probed path in unresolved, because it really did fail to compare; exists leaves it empty, because it did not fail at anything. That is the ergonomic argument. The type bug is the correctness one, and it is why this became a ninth operator rather than staying a documented trick.
3.6 Schema versions, and what a version is allowed to break
schema_version is a const in each schema file, so the version a policy declares and the schema it is checked against cannot disagree. validate_policy() reads the declared version, looks that schema up, and rejects an unknown version by name.
| Declared | Schema file | Operators |
|---|---|---|
nakedsignal-policy-0.1 | policy-v0.1.schema.json | the eight |
nakedsignal-policy-0.2 | policy-v0.2.schema.json | the eight, plus exists, older_than, newer_than |
nakedsignal-policy-0.3 | policy-v0.3.schema.json | those eleven, plus within_envelope |
The compatibility rule: an older policy must validate, and decide identically, forever. A runtime that breaks its own published policies is not infrastructure. Both halves are checked mechanically rather than asserted — check_agreement.py step 2 validates every shipped example against the version it declares, then bumps only its schema_version to the newest and requires every decision it makes to come out byte-identical. The newest version is read out of policy.py, not written into that check, so a schema bump cannot leave it quietly testing the wrong thing.
Each schema is a new file, and the older ones stay byte-unchanged. That is the same discipline passport-v0.1.schema.json is held to, and for the same reason: every policy already signed and in circulation still validates against the thing it was signed under.
The evaluator is version-agnostic. It dispatches on the operator string alone and has no idea which schema a policy declared. What a schema version decides is which operators a policy may declare, not what an evaluator does when it meets one. A v0.1 policy containing exists will be rejected by validate_policy() and evaluated correctly by evaluate() — and that is the right split, because validation is where a version is a promise and evaluation is where it would be a trap. Fixture a-v02-policy-may-not-declare-within-envelope-but-still-evaluates pins the same split one version on.
4. The inference envelope
What the caller supplies per inference. Kept minimal, because every field is an integration cost for the customer. Schema: plan_d/src/runtime/envelope-v0.1.schema.json.
{ "model_id": "cnn_scratch", "confidence": 0.79, "ood_score": 0.31,
"site_profile": "siemens-somatom-force", "subgroup": "paediatric",
"indication": "…", "ts": "2026-07-31T09:00:00Z" }Only model_id is required. Everything else is optional and, when absent, behaves per §3.4 — which is the honest behaviour: a caller that does not report an OOD score does not get to satisfy an OOD rule.
`site_fingerprint` (v0.3). within_envelope reads the site's physics fingerprint from inference.site_fingerprint, in the shape variation/fingerprint.py emits: feature_vector_version, and a features map of {ci_low, ci_high, median, …} per column. The envelope schema is open (additionalProperties: true), so this needed no schema bump — and it is genuinely per-inference data supplied by the caller, computed from the site's own images in the site's own browser, which is the only reason a hospital would ever produce one. A caller that does not report a fingerprint does not get to satisfy an envelope rule.
envelope_hash on every decision is sha256 of the canonical envelope bytes: sorted keys, (",", ":") separators, ASCII-escaped, and integral floats narrowed to integers so that Python and JavaScript emit the same digits. That last rule exists because json.dumps(3.0) is 3.0 and JSON.stringify(3.0) is 3, and a hash that disagrees across languages is a hash nobody can use.
5. The decision
{ "action": "defer_to_human",
"effective_action": "log_only",
"mode": "log_only",
"rule_id": "low-confidence-ood",
"because": "model has not demonstrated calibrated confidence in this regime",
"policy": "example-conservative-radiology",
"policy_version": "0.1",
"passport_id": "cnn_scratch@bv:1.0.0",
"envelope_hash": "sha256:…",
"unresolved": [],
"ts": "2026-07-31T09:00:00Z" }action is what the policy says. effective_action is what the caller must actually do, and under log_only it is always log_only. Both are recorded, so a shadow deployment produces exactly the evidence needed to decide whether enforcement would have been safe.
passport_id, and why the behaviour version outranks the byte hash
passportId() derives the id from the first of these the passport can supply: an explicit passport_id; model_id@bv:<subject.behaviour_version>; model_id@<subject.code_fingerprint>; a bare model_id.
code_fingerprint is a sha256 over the scoring code's bytes, so it moves on a docstring edit and on a newly registered model — neither of which can change a published number. behaviour_version is hand-bumped only when the numbers can move, and harness.py already made it the identity a cached record keys on. Two passports for one model under one behaviour but different bytes describe the same evidence and must get the same id; preferring the byte hash would give them two, which is the churn the behaviour version exists to stop.
⚠️ It also closes a silent failure. passport.py sets subject.code_fingerprint to null when a subject's records span more than one byte hash — which, after identity moved to the behaviour version, is the expected steady state rather than a fault. A runtime that identified the code through the fingerprint alone then fell back to a bare model_id, and a decision that no longer says which code produced its evidence is the same failure class as two evaluations a reader cannot tell apart. Every byte hash is still carried, in full, in subject.code_fingerprints.
The bv: prefix keeps the two namespaces apart: without it, model@1.0.0 could not be told from a truncated hash that happened to look like a version. A passport that declares no behaviour version — every one issued before the field existed — keeps model_id@fingerprint unchanged, which is what makes this additive rather than a rename of every id already in circulation. Both halves are pinned as conformance vectors: split-byte-hashes-under-one-behaviour-still-identify-the-code and a-passport-with-no-behaviour-version-keeps-the-id-it-always-had.
6. The decision log
plan_d/src/runtime/log.py. Append-only JSONL, hash-chained: each entry carries prev_hash, and entry_hash = sha256(canonical(body)). Editing any historical field breaks every subsequent hash.
This is not a second ledger. log.py imports Ledger from plan_d/src/heldout/ledger.py — the same class, the same chain format, the same verify() — which has already been exercised across the live held-out run. The runtime adds a chain tag and an event name and nothing else. Default path: plan_d/logs/runtime/decisions.jsonl.
python src/runtime/cli.py verify-log --log <path> recomputes the chain and exits non-zero if it is broken.
Same caveat as the held-out ledger: the chain is anchored only to itself. An external timestamping anchor (RFC 3161) is a deployment step, not a code change.
7. Two implementations, one behaviour
7.1 Why there are two
The /runtime page runs client-side with no server, because the shell is a static export. So the evaluator is ported to TypeScript. Two evaluators that disagree is a bug you find in front of an audience.
7.2 How they are kept honest
plan_d/src/runtime/fixtures/cases.json holds (policy, passport, envelope, now, signature_valid) -> expected decision triples. Both implementations run every case and must produce byte-identical canonical decisions, envelope_hash and unresolved list included.
python plan_d/src/runtime/check_agreement.py [--live]
runs the Python evaluator, shells out to node plan_d/src/runtime/ts_runner.mjs for the TypeScript one, compares canonical JSON per case, and exits non-zero on any divergence. It is the §7 definition-of-done check for doc 03. Four assertions, in order of how much they buy you:
- every policy validates — the examples in
site-data/policies/against the version each declares, and the fixture policies too. A policy a fixture markspolicy_invalidmust be rejected; that is the assertion, not an exemption. - v0.1 policies survive v0.2 — §3.6's compatibility rule, checked by bumping and diffing rather than by asserting.
- the Python decision matches the hand-written `expected` fields. This is the real correctness test. Agreement between two implementations of the same mistake is not correctness.
- Python and TypeScript are byte-identical after canonicalisation.
--live adds a fifth: every real passport in site-data/passports/ crossed with every example policy, a grid of envelopes and all three signature states, asserting (4) only. Those files are emitted by doc 02 and change, so pinning expectations against them would be brittle. The combination count is a product of four things that all live on disk, so it is derived and printed with its factors — adding one passport moves it, and nothing downstream is allowed to hardcode the total.
A case names a policy file in site-data/policies/, so the files the demo loads are the files the agreement test covers — or a key into the spec's own policies map, which is where the v0.2 operators are exercised. Fixture policies live in the fixture file rather than in site-data/policies/, because that directory is the set of examples the demo offers a customer, not a test bed.
One divergence risk `within_envelope` deliberately removes rather than manages. variation/fingerprint.py:DRIFT_RISKS names nine operations whose result can differ between a Python implementation and a JavaScript one; two of them reach this operator, and both are handled by not doing the thing. Nothing is sorted, so JavaScript's lexicographic default sort cannot bite. Nothing is re-rounded, because rounding is the single operation the two languages genuinely define differently — Python's round() is ties-to-even, Number(x.toFixed(6)) is ties-away-from-zero, and Math.round(x * 1e6) / 1e6 is inexact in a third way. Every number on both sides already arrived rounded to 6 dp through the one canonicaliser, and raw IEEE-754 multiply, add and subtract are bit-identical in both languages. The cost is that the grace bounds this operator computes can differ in the last bit from the ones variation/envelope.py publishes after rounding them; that would only matter for a fingerprint landing within an ULP of a grace bound, and it is checked rather than argued — the two implementations are compared on all 456 real per-feature rows and agree on every one.
Known and accepted divergence: string ordering above the Basic Multilingual Plane differs (JavaScript compares UTF-16 code units, Python compares code points). Policies are ASCII; the fixtures do not exercise it; it is written down rather than silently assumed.
8. One engine, two binding times
The same evaluator, run without an inference envelope, answers a different question: may we buy this at all? A procurement policy reads only passport.* — "only allow AUROC > 0.92, corruption retention > 0.85, FDA cleared, evidence unexpired" — and returns allow or reject at contract time rather than at request time.
That is why binding_time ("inference" | "procurement") is a field on the policy and not a separate product. site-data/policies/example-procurement-filter.json is that second binding time, running on the same 200 lines of evaluator. The procurement filter the original vision put in a separate product is a policy file.
9. The /runtime panel
documentation/src/app/runtime/. A client component, because a static export has no server. It loads the example policies and the real signed passports at build time, evaluates in the browser on every control change, and appends to an in-memory hash-chained log on demand.
Three honesty notes that belong in the spec rather than only in the code:
- It loads a projection of each passport, not the whole document: identity, expiry, issuer, the cross-site block without its full matrix, and the scalar metrics of each evaluation. Twenty-five complete passports is close to a megabyte of inlined JSON. Fields are copied or dropped, never re-derived, re-rounded or re-keyed, and the page says so. The document of record is the full signed one on
/passport. - The log in the browser is not the log on disk. Same chain construction, same failure mode if you edit history, but it lives in the tab and is not persisted. The CLI writes the real one.
- Defaults are set so the page loads on `DEFER TO HUMAN`, not
ALLOW. The interesting state is the one where the system stops something.
Badge: PROTOTYPE.
10. What v0.1 deliberately does not do
- Proxy or intercept real traffic. It is a function you call.
- Execute a fallback route. It returns the word
route_to_fallback. - Verify signatures. It consumes
runtime.signature_validfrom the caller. - Arithmetic, or any rule that derives a value rather than reading one.
- Ship a default policy, a default threshold, or a clinical opinion of any kind.
Closed in v0.2 — `exists`. v0.1 could not say "reject if the vendor supplied no declaration" and shipped a workaround for it here. §3.5 records what that workaround was, that it still works, and the type bug that made it worth replacing rather than documenting forever.
Closed in v0.2 — evidence age. A policy can now say "refuse evidence older than 90 days" in one leaf (§3.4.9) instead of hard-coding a cut-off date that goes stale the day after it is written.
Closed in v0.3 — deployment outside the conditions the evidence covers. A policy can now say "refuse if this site's acquisition physics is outside the envelope this model was signed with" (§3.4.10). What that took is worth stating, because it is the reason the gate is hard to copy rather than hard to write: the envelope has to have been fitted (S2), signed into the passport (S6.1) and checked off-generator (S5), and the site has to be able to compute its own fingerprint without sending anybody its images (S6.2). The operator itself is 90 lines. Anyone can build a slider.
Still not done, and named rather than implied:
- Envelope gating at procurement time.
within_envelopereads the fingerprint frominference.*, so a policy withbinding_time: procurementcannot use it —validate_policy()rejects a procurement policy that reads the inference namespace, and it is right to (§8). But "is this model's evidence valid for our scanners" is a procurement question at least as much as an inference one. Answering it properly means a fourth namespace for the deploying site rather than smuggling site data through the per-request envelope, and inventing one to make this lane's operator look more general is exactly the move that should be refused.
- Calendar durations.
P3MandP1Yare rejected, not approximated (§3.1). A policy that genuinely means "one calendar year" has to writeP365Dand accept what that means on a leap year. Supporting the real thing would put a calendar inside the evaluator, and the two implementations would then have to agree about one — which is a much larger promise than this file currently makes. - `exists` on an array element or a wildcard.
existsresolves the same paths every other operator does, so it can ask aboutcomputed.evaluations.0.dimensionsbut not "any evaluation with a calibration block". That is iteration, and iteration is still out (§3.2). - A third schema version's deprecation story. v0.2 is purely additive, so nothing needed removing yet. The rule for when something can be removed is not written, and should be written before it is needed rather than during.
Rendered from plan_d/docs/RUNTIME-SPEC.mdat build time. Checked against the hosting spec’s scrub grep before rendering: zero hits.