NakedSignal OS · Documentation
Back to the consoleDocumentation

Live evaluation

PROTOTYPE

A private held-out track that rotates: sealed sets minted from disjoint accrual windows, an append-only ledger, a structural separation between the party that scores and the party that submits, and an audit that re-derives every published number from manifests alone.

What this is notThis is a public-data demonstration of a private protocol, executed end to end on BloodMNIST as a clearly-labelled stand-in. It has not been run on clinical data.

What the run caught

PUBLIC-DATA DEMONSTRATION OF THE PRIVATE PROTOCOL
100.0
score on the sealed private track, by a model trained on it
76.8
the same model on the public control arm
-15.11
z against a threshold written into the ledger before the first case was drawn
32.8%
of the advantage removed by one rotation
100
ledger entries, chain intact
corpus BloodMNIST (peripheral blood cells, 8 classes)code fingerprint d50ab00f9ad4advantage, generation 1 24.95 → generation 2 1.17re-derive it: python src/heldout/audit.py 1f66c7719ab3943c6fcc

Verifying without seeing

PROTOTYPE

A held-out track has to keep its test set secret, or it stops measuring anything. But secrecy is the enemy of verifiability: a number computed on data nobody can inspect is a number you are being asked to take on trust. Those two requirements pull in opposite directions and no amount of care resolves it.

What resolves it today
  • The set hash is written into the ledger at sealing, before submissions open, so the set cannot be edited after seeing a result.
  • The ledger is hash-chained, so an entry cannot be reordered or rewritten without breaking every entry after it.
  • An external auditor re-derives every published score from manifests carrying salted case identifiers and no image, feature or patient attribute.
  • A Merkle root over the same cases goes into the same ledger entry as the set hash. Hand someone one case and eleven sibling hashes and they can prove it was sealed into generation 1 — while learning nothing about the other 1,399.

The score reproduction is still a social resolution wearing cryptography: the auditor runs our pipeline, so that part of the independence is partial and TODO.md §1 says so. Inclusion is the exception — it is checked by src/verify_inclusion.py, which shares no code with the sealing module, and the audit fails if the two ever disagree.

The direction

What the Merkle root does not buy is a clock. It is committed to our own ledger, so the ordering is cryptographic — the root provably precedes every score against that set — while the absolute time of the commitment is still attested by us. Countersigning the ledger head to an RFC 3161 timestamping authority or a public append-only log closes that, and it is a deployment step rather than a code change.

The end state is that nobody has to be trusted, including us. Two things still stand between here and there: that external anchor, and an auditor who re-implements the scoring rather than re-running ours. The Ed25519 signature on every Evidence Passport, the hash chain, and now the inclusion proofs are the spine those extend, not plumbing underneath them.

The full artifact runs below, step by step, with its own ledger feed. It also opens standalone.