An independent, versioned standard for measuring medical AI — scored on seven dimensions, across every modality, on held-out data. This is the working v0 of the harness, not a published benchmark.
Every point is one model, averaged over the imaging tasks it completed. If the two axes
agreed, the points would fall along a rising line. They do not — the cloud slopes the
wrong way, which is the entire argument for scoring a model on more than one number.
Read the vertical axis carefully. It is a retention ratio: the share of a model's
own clean performance that survives acquisition shift. A model with little to lose
retains a lot of it, which is why the weakest classifiers sit high here and why retention is
never admissible on its own. The absolute counterpart — chance-corrected skill that actually
survives the shift — is the Robust skill column in the leaderboard above. The model
worth deploying is the one that is high on both, and that is what the composite index is for.
Retention of chance-corrected balanced accuracy under each perturbation, averaged over three severities, on a fixed seeded subsample of the held-out split. The five acquisition shifts change how the scan was made; the seven corruptions degrade it. κ is test-retest agreement against the model's own clean predictions — it answers "would the clinician get the same call twice?", which is a different question from whether accuracy held up.
Every task carries its source, its citation, its split provenance, and its class list. Adding a modality is a registry row plus an adapter — the scorer, the models and this page do not change.
Public benchmarks leak into training data and die. The fix is a test set the model builder can never see and can never train on: minted fresh, sealed, rotated on a published expiry, scored by the standard's operator rather than by the submitter. Below is the protocol — and it now executes end to end, on a public corpus used as a clearly-labelled stand-in for the clinical stream.
Full protocol, governance and audit trail: docs/HELDOUT-PROTOCOL.md and docs/MEDEVAL-1-SPEC.md. The live run, its ledger and its audit bundles: results/heldout/.
Weights are fixed by the spec version and published before results. An accuracy-only index is the thing this layer exists to replace.