Back to the consoleDocumentation
Evidence Passport
LIVEWhat a vendor's PDF becomes when a computer has to read it: a small signed file that says how the model scored, where it breaks, what it may be used for, and when the evidence expires.
When a hospital buys an AI system today, the evidence that it works arrives as a PDF written by the vendor. A human reads it, forms an impression, and the system gets deployed. Nothing checks the PDF, nothing stops the model being used for something the PDF never claimed, and nothing notices when it goes stale.
An Evidence Passport is that PDF turned into a small signed file a computer can read — the same idea as a passport at a border. It says who made the model, how it scored on data it had never seen, which patient group it is worst on, what the vendor says it may and may not be used for, when the evidence expires, and it carries a signature proving none of that has been edited since we issued it.
Three things follow. A machine can check it — procurement stops being a reading exercise and becomes medeval verify passport.json, exit zero or non-zero. The vendor's claims and our measurements stay visibly separate — they are different objects in the file, not differently-labelled rows in one table, so a marketing claim cannot be laundered into a measurement. And something can refuse to run without one — that is the Trust Runtime. A passport nobody reads is a PDF with extra steps; a passport a gate reads is infrastructure.
Issued passport
One document, signed with the NakedSignal issuing key. Press Verify. Then tick Tamper, which changes a single score, and press Verify again.
SubjectSmall CNN, trained from scratch
cnn_scratchConfiguration3 conv blocks, 20 epochs, adam 1e-3, early stop on val
Code fingerprintb444c02112cc
Spec versionMedEval-1 v0.1
Issued2026-08-03T16:03:42Z by NakedSignal
Expires2026-09-20T00:00:00Z — the sealed held-out generation this evidence cycle is anchored to (gen2, set hash ad30d214daa285c5...) is published to expire 2026-09-20; per spec, a passport lapses when its generation retires or its spec version is superseded, whichever comes first
key ns-passport-2026-07 · Ed25519
The check runs against the public key published on the governance page, using your browser's own Ed25519 implementation. The exit code shown is the exit code verify_passport.py returns for the same document.
Measured by NakedSignal
Computed in this repo by the MedEval-1 harness on data the model had never seen. Every number below was copied out of a file the harness wrote — 17 of them, each listed with its SHA-256 at the foot of this panel — and is printed exactly as it was signed, not re-rounded for display.
| Task | Domain | n test | accuracy | calibration | uncertainty | subgroup | corruption | acquisition | limited data | cost | spec sensitivity | Index |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bloodmnist | medicine | 3421 | 95.670132 | 92.839737 | 99.539304 | 91.549296 | 48.281147 | 62.262611 | 36.159496 | n/a | n/a | 75.826128 |
| breastmnist | medicine | 156 | 59.649123 | 57.62214 | 89.000119 | 33.333333 | 49.329732 | 73.445378 | 36.904762 | n/a | n/a | 57.012677 |
| dermamnist | medicine | 2005 | 44.349021 | 88.548352 | 91.054933 | 0 | 59.039044 | 76.717358 | 41.153195 | n/a | n/a | 56.652235 |
| octmnist | medicine | 1000 | 57.066667 | 0 | 78.10626 | 1.866667 | 31.964842 | 52.305296 | 32.476636 | n/a | n/a | 36.642422 |
| organamnist | medicine | 17778 | 91.496937 | 84.485694 | 99.13683 | 77.776163 | 72.660343 | 83.556087 | 43.447091 | n/a | n/a | 80.561432 |
| organcmnist | medicine | 8216 | 90.014245 | 80.353307 | 98.791999 | 75.639615 | 75.19459 | 83.541043 | 41.967047 | n/a | n/a | 79.594617 |
| organsmnist | medicine | 8827 | 72.475324 | 53.915875 | 93.222958 | 37.6082 | 65.178139 | 78.199663 | 40.691244 | n/a | n/a | 64.100882 |
| pathmnist | medicine | 7180 | 78.7404 | 61.2935 | 93.718695 | 38.539192 | 38.292373 | 46.412525 | 42.077896 | n/a | n/a | 56.68336 |
| pneumoniamnist | medicine | 624 | 74.871795 | 64.538792 | 95.083525 | 53.846154 | 55.300065 | 72.709285 | 0 | n/a | n/a | 61.913727 |
| retinamnist | medicine | 400 | 17.238807 | 80.109489 | 64.736128 | 0 | 94.321353 | 89.328189 | 36.139308 | n/a | n/a | 54.747757 |
| tissuemnist | medicine | 47280 | 41.647253 | 93.716198 | 72.569134 | 8.43836 | 48.226827 | 58.390874 | 30.629144 | n/a | n/a | 50.501494 |
Cross-site retention · organ_ct
retention is absolute retained skill (chance-corrected), per spec 2.2.1; the diagonal is same-site, every off-diagonal cell is train-on-row, test-on-column
Sitesorganamnist, organcmnist, organsmnist
Mean retention0.63221
Worst retention0.374032
Private held-out track
this subject was not submitted to a sealed generation.Known limitations
Subgroup metadataabsent on 11 of 11 corpora, so the subgroup score falls back to the worst diagnostic class. That absence is a disclosure, not a pass.
Training set cappedyes on at least one task — see the raw JSON for which
Observed in deploymentnothing — see the third panel
Audit chain
Bundles re-derived48 of 48
Ledger100 entries, head 6b7b371ff650a8ed…, chain intact
Re-run it yourself
python src/heldout/audit.py 1f66c7719ab3943c6fcc17 source files, with hashes
| results/cross_site/organ_ct__cnn_scratch.json | 2cb33bd19620e036d679332fd717024b5b74f1cae25c506645558f82ba8018ac |
| results/heldout/audit_report.json | df460b6d91a6512dfa1e7dc650637fce2b4dd3a1ec32e03446bf6e11f9c597d2 |
| results/heldout/cycle.json | e012b8978457b51b5c31d18d14d2cc92f100731897523306acae19934f943502 |
| results/heldout/manifests/gen1.seal.json | 91a7adfa29870a0719125cdd48f6dcb149f1f6485ac1cfa104c6e00057ad9b81 |
| results/heldout/manifests/gen2.seal.json | f61470918a09cf39e074ef14fb7ade4ac3a13d36d9385f18cf9b92e4b5329547 |
| results/heldout/manifests/public.seal.json | 1a4fd930d2e781ac7163483990f15d6694e28e507d1b3ca009ff09395e47497c |
| results/records/bloodmnist__cnn_scratch.json | 658e3e5f7cfa4ad5fd6c7b39ba0aeafd63cd24b954494a2ea543b80339ff90ee |
| results/records/breastmnist__cnn_scratch.json | 2212aec836e7f76f734e4219f2440104ddf80ee2db0595211786509d77fc9ad6 |
| results/records/dermamnist__cnn_scratch.json | ecafb8abee712c7ae14cefcbc3e079ecdbbec6ca0eaf1915cd70d66b033443c4 |
| results/records/octmnist__cnn_scratch.json | 051221f9e35353e03d86cf3fc99a668fc38a9dde7a433eeee6c08430ff7577d7 |
| results/records/organamnist__cnn_scratch.json | ac309b752caab9b09114ef594183239dc8b5fdb3732f0e694a9bf2aef0a9e46d |
| results/records/organcmnist__cnn_scratch.json | b00ac3f602f5687f42d5649af03dea831ae0f80f650542f3bbc20a3766d12225 |
| results/records/organsmnist__cnn_scratch.json | 83a9069311350c3760a04eb18c472285d1d26facd163ea735ef6448af4187a95 |
| results/records/pathmnist__cnn_scratch.json | 19764268891a48ac331ae275acc81c85b16828b02614ee8f32abda10afd4d909 |
| results/records/pneumoniamnist__cnn_scratch.json | e15163c863218cf6062262c8bfc0c3800e7a3b6995751ffb2802e5a204876209 |
| results/records/retinamnist__cnn_scratch.json | 702b5c2292a18b58c0d1deb42fee6ea2dc74f3c6afa7e10203958a5ec14b8b01 |
| results/records/tissuemnist__cnn_scratch.json | 792813023724858e90cb26cab9ff90c5a00925ee563ecf98bccb4e4c08ae4396 |
Declared by vendor
Five fields a vendor asserts about its own product — intended use, forbidden use, training cutoff, regulatory clearances, and who is personally attesting. NakedSignal never fills these in on a vendor's behalf.
No vendor declaration. This is an open reference baseline implemented by NakedSignal, not a commercial product: there is no vendor to assert an intended use or to sign an attestation, so the declared block is null rather than filled in on the vendor's behalf.
Observed in deployment
Production history and drift — how the model has actually behaved since it was deployed, on real traffic.
Not available: this requires continuous monitoring. Production history and drift are observed fields, and nothing is deployed, so there is nothing to observe. The object is present and null on purpose. The shape is the roadmap, not a claim.
Raw JSON — the whole signed document, 81,184 characters
{
"canonicalisation": {
"float_rounding": "every float is rounded once at emit to 6 decimal places (Python round(), banker's rounding); integers are left exact",
"form": "RFC 8785 (JCS)-compatible: UTF-8, object keys sorted by code point, no insignificant whitespace, ECMAScript number formatting",
"signed_over": "the whole document with the `signature` member removed, canonicalised as above and encoded UTF-8"
},
"computed": {
"audit": {
"audit_command": "python src/heldout/audit.py 1f66c7719ab3943c6fcc",
"bundles": 48,
"ledger_entries": 100,
"ledger_head": "6b7b371ff650a8edaf477490f2a5f9b043433f3ba9941989ee1962ed8f1b1d88",
"ledger_intact": true,
"rederived": 48
},
"cross_site": {
"code_fingerprint": "b444c02112cc",
"device": "mps",
"family": "organ_ct",
"matrix": {
"organamnist": {
"organamnist": {
"balanced_accuracy": 0.922699,
"chance_corrected": 0.914969,
"ece": 0.031029
},
"organcmnist": {
"balanced_accuracy": 0.739566,
"chance_corrected": 0.713523,
"ece": 0.144638,
"retention": 0.779832
},
"organsmnist": {
"balanced_accuracy": 0.402025,
"chance_corrected": 0.342228,
"ece": 0.399281,
"retention": 0.374032
}
},
"organcmnist": {
"organamnist": {
"balanced_accuracy": 0.670079,
"chance_corrected": 0.637087,
"ece": 0.180735,
"retention": 0.707762
},
"organcmnist": {
"balanced_accuracy": 0.90922,
"chance_corrected": 0.900142,
"ece": 0.039293
},
"organsmnist": {
"balanced_accuracy": 0.500357,
"chance_corrected": 0.450392,
"ece": 0.318423,
"retention": 0.500357
}
},
"organsmnist": {
"organamnist": {
"balanced_accuracy": 0.436419,
"chance_corrected": 0.38006,
"ece": 0.312101,
"retention": 0.5244
},
"organcmnist": {
"balanced_accuracy": 0.688418,
"chance_corrected": 0.65726,
"ece": 0.112176,
"retention": 0.906874
},
"organsmnist": {
"balanced_accuracy": 0.749776,
"chance_corrected": 0.724753,
"ece": 0.092168
}
}
},
"mean_retention": 0.63221,
"note": "retention is absolute retained skill (chance-corrected), per spec 2.2.1; the diagonal is same-site, every off-diagonal cell is train-on-row, test-on-column",
"score": 63.220953,
"site_labels": {
"organamnist": "axial",
"organcmnist": "coronal",
"organsmnist": "sagittal"
},
"sites": [
"organamnist",
"organcmnist",
"organsmnist"
],
"spec_version": "MedEval-1 v0.1",
"worst_retention": 0.374032
},
"evaluations": [
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 75.826128
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.964338,
"auc": 0.998087,
"balanced_accuracy": 0.962114,
"ci95": [
0.954715,
0.969113
],
"n_test": 3421,
"score": 95.670132
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.956948,
"mean_agreement": 0.675333,
"mean_kappa": 0.629668,
"mean_retention": 0.622626,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9783,
0.9475,
0.7467
],
"mean_kappa": 0.872276,
"mean_retention": 0.873256,
"retention_by_severity": [
0.9889,
0.9405,
0.6904
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8592,
0.8308,
0.6725
],
"mean_kappa": 0.749683,
"mean_retention": 0.729549,
"retention_by_severity": [
0.8312,
0.7962,
0.5613
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.7392,
0.1175,
0.0917
],
"mean_kappa": 0.244572,
"mean_retention": 0.250653,
"retention_by_severity": [
0.6868,
0.0538,
0.0114
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.8308,
0.7283,
0.6367
],
"mean_kappa": 0.689386,
"mean_retention": 0.696087,
"retention_by_severity": [
0.8,
0.6898,
0.5985
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.8775,
0.6892,
0.3842
],
"mean_kappa": 0.592421,
"mean_retention": 0.563584,
"retention_by_severity": [
0.8252,
0.5872,
0.2783
],
"scored": true
}
},
"score": 62.262611,
"worst_retention": 0.011446
},
"calibration": {
"accuracy": 0.964338,
"brier": 0.054458,
"ece": 0.014321,
"mean_confidence": 0.9765,
"overconfidence": 0.012162,
"score": 92.839737
},
"corruption": {
"clean_reference_cc": 0.956948,
"mean_agreement": 0.542698,
"mean_kappa": 0.477133,
"mean_retention": 0.482811,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.6908,
0.32,
0.2125
],
"mean_kappa": 0.285003,
"mean_retention": 0.227443,
"retention_by_severity": [
0.4935,
0.1427,
0.0461
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.3658,
0.2192,
0.2058
],
"mean_kappa": 0.160208,
"mean_retention": 0.183528,
"retention_by_severity": [
0.2769,
0.1476,
0.1261
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.8467,
0.6642,
0.4583
],
"mean_kappa": 0.605387,
"mean_retention": 0.621074,
"retention_by_severity": [
0.8174,
0.6348,
0.411
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9108,
0.6,
0.0875
],
"mean_kappa": 0.479048,
"mean_retention": 0.492202,
"retention_by_severity": [
0.9141,
0.552,
0.0105
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.845,
0.2517,
0.7017
],
"mean_kappa": 0.552308,
"mean_retention": 0.575394,
"retention_by_severity": [
0.817,
0.2444,
0.6648
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9892,
0.9742,
0.9175
],
"mean_kappa": 0.953413,
"mean_retention": 0.967025,
"retention_by_severity": [
0.9981,
0.9884,
0.9145
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.7067,
0.3483,
0.0808
],
"mean_kappa": 0.304562,
"mean_retention": 0.313014,
"retention_by_severity": [
0.6481,
0.291,
0
],
"scored": true
}
},
"score": 48.281147,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.956701,
"mean_retention": 0.361595,
"per_budget": {
"n100": {
"chance_corrected": 0.854905,
"n_labels": 800,
"retention": 0.893597
},
"n20": {
"chance_corrected": 0.18291,
"n_labels": 160,
"retention": 0.191188
},
"n5": {
"chance_corrected": 0,
"n_labels": 40,
"retention": 0
}
},
"score": 36.159496
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 91.549296,
"worst_class_recall": 0.926056
},
"uncertainty": {
"full_accuracy": 0.964338,
"risk_coverage_auc": 0.995969,
"score": 99.539304,
"selective_acc_at_50": 0.999415,
"selective_acc_at_80": 0.996346
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 11959,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Microscopy",
"n_classes": 8,
"n_test": 3421,
"runtime_s": 34.2,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "bloodmnist",
"task_name": "Peripheral blood cells (8-class)",
"timestamp": "2026-08-01T19:19:11+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 57.012677
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.858974,
"auc": 0.902673,
"balanced_accuracy": 0.798246,
"ci95": [
0.723641,
0.860586
],
"n_test": 156,
"score": 59.649123
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.596491,
"mean_agreement": 0.82265,
"mean_kappa": 0.549964,
"mean_retention": 0.734454,
"n_cases": 156,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9808,
0.9487,
0.7821
],
"mean_kappa": 0.633228,
"mean_retention": 0.636555,
"retention_by_severity": [
0.9349,
0.8697,
0.105
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8397,
0.7756,
0.7564
],
"mean_kappa": 0.535653,
"mean_retention": 0.892857,
"retention_by_severity": [
1,
0.8466,
0.8319
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9103,
0.859,
0.7821
],
"mean_kappa": 0.588286,
"mean_retention": 0.742297,
"retention_by_severity": [
0.9412,
0.8067,
0.479
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.7179,
0.7115,
0.5962
],
"mean_kappa": 0.38574,
"mean_retention": 0.820728,
"retention_by_severity": [
0.9538,
0.8887,
0.6197
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9679,
0.9167,
0.7949
],
"mean_kappa": 0.606914,
"mean_retention": 0.579832,
"retention_by_severity": [
0.9643,
0.645,
0.1303
],
"scored": true
}
},
"score": 73.445378,
"worst_retention": 0.105042
},
"calibration": {
"accuracy": 0.858974,
"brier": 0.209601,
"ece": 0.084756,
"mean_confidence": 0.924424,
"overconfidence": 0.06545,
"score": 57.62214
},
"corruption": {
"clean_reference_cc": 0.596491,
"mean_agreement": 0.815018,
"mean_kappa": 0.440312,
"mean_retention": 0.493297,
"n_cases": 156,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9359,
0.8462,
0.7692
],
"mean_kappa": 0.419008,
"mean_retention": 0.403361,
"retention_by_severity": [
0.7605,
0.4496,
0
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8974,
0.8205,
0.7308
],
"mean_kappa": 0.545439,
"mean_retention": 0.558824,
"retention_by_severity": [
0.8277,
0.5252,
0.3235
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.8077,
0.5769,
0.5064
],
"mean_kappa": 0.344081,
"mean_retention": 0.672969,
"retention_by_severity": [
0.9958,
0.6849,
0.3382
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.8782,
0.7692,
0.7692
],
"mean_kappa": 0.193072,
"mean_retention": 0.153361,
"retention_by_severity": [
0.4601,
0,
0
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.8654,
0.8974,
0.8333
],
"mean_kappa": 0.638861,
"mean_retention": 0.742997,
"retention_by_severity": [
0.9727,
0.8824,
0.3739
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9936,
0.9679,
0.8974
],
"mean_kappa": 0.851457,
"mean_retention": 0.864846,
"retention_by_severity": [
1,
0.9895,
0.605
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.8141,
0.7692,
0.7692
],
"mean_kappa": 0.090264,
"mean_retention": 0.056723,
"retention_by_severity": [
0.1702,
0,
0
],
"scored": true
}
},
"score": 49.329732,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.596491,
"mean_retention": 0.369048,
"per_budget": {
"n100": {
"chance_corrected": 0.359649,
"n_labels": 200,
"retention": 0.602941
},
"n20": {
"chance_corrected": 0.017544,
"n_labels": 40,
"retention": 0.029412
},
"n5": {
"chance_corrected": 0.283208,
"n_labels": 10,
"retention": 0.47479
}
},
"score": 36.904762
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 33.333333,
"worst_class_recall": 0.666667
},
"uncertainty": {
"full_accuracy": 0.858974,
"risk_coverage_auc": 0.945001,
"score": 89.000119,
"selective_acc_at_50": 0.935897,
"selective_acc_at_80": 0.92
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 546,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Ultrasound",
"n_classes": 2,
"n_test": 156,
"runtime_s": 2.5,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "breastmnist",
"task_name": "Breast ultrasound (malignancy)",
"timestamp": "2026-08-01T19:12:59+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 56.652235
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.771571,
"auc": 0.926125,
"balanced_accuracy": 0.522992,
"ci95": [
0.482709,
0.566408
],
"n_test": 2005,
"score": 44.349021
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.435485,
"mean_agreement": 0.778889,
"mean_kappa": 0.606071,
"mean_retention": 0.767174,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.89,
0.5975,
0.2533
],
"mean_kappa": 0.433673,
"mean_retention": 0.6537,
"retention_by_severity": [
0.9681,
0.7079,
0.2851
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.925,
0.8908,
0.8525
],
"mean_kappa": 0.754461,
"mean_retention": 0.840558,
"retention_by_severity": [
0.918,
0.8298,
0.7739
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8833,
0.7817,
0.6983
],
"mean_kappa": 0.559295,
"mean_retention": 0.609188,
"retention_by_severity": [
0.8306,
0.6287,
0.3683
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.8925,
0.86,
0.84
],
"mean_kappa": 0.690457,
"mean_retention": 0.847676,
"retention_by_severity": [
0.8655,
0.8422,
0.8354
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9058,
0.7892,
0.6233
],
"mean_kappa": 0.592466,
"mean_retention": 0.884745,
"retention_by_severity": [
1,
0.9052,
0.7491
],
"scored": true
}
},
"score": 76.717358,
"worst_retention": 0.285065
},
"calibration": {
"accuracy": 0.771571,
"brier": 0.320743,
"ece": 0.022903,
"mean_confidence": 0.794337,
"overconfidence": 0.022766,
"score": 88.548352
},
"corruption": {
"clean_reference_cc": 0.435485,
"mean_agreement": 0.666032,
"mean_kappa": 0.433985,
"mean_retention": 0.59039,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.82,
0.6758,
0.545
],
"mean_kappa": 0.382733,
"mean_retention": 0.614601,
"retention_by_severity": [
0.8398,
0.6934,
0.3106
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8367,
0.6925,
0.5975
],
"mean_kappa": 0.463335,
"mean_retention": 0.618717,
"retention_by_severity": [
0.9085,
0.6407,
0.3069
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9125,
0.84,
0.7867
],
"mean_kappa": 0.640404,
"mean_retention": 0.7437,
"retention_by_severity": [
0.8448,
0.7796,
0.6067
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.5717,
0.25,
0.34
],
"mean_kappa": 0.125043,
"mean_retention": 0.316395,
"retention_by_severity": [
0.7269,
0.2223,
0
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9375,
0.7183,
0.7742
],
"mean_kappa": 0.641592,
"mean_retention": 0.762315,
"retention_by_severity": [
1,
0.3425,
0.9444
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9775,
0.9117,
0.7133
],
"mean_kappa": 0.769279,
"mean_retention": 0.92525,
"retention_by_severity": [
0.9717,
0.9739,
0.8301
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.3242,
0.2983,
0.4633
],
"mean_kappa": 0.015512,
"mean_retention": 0.151756,
"retention_by_severity": [
0.3168,
0.1384,
0
],
"scored": true
}
},
"score": 59.039044,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.44349,
"mean_retention": 0.411532,
"per_budget": {
"n100": {
"chance_corrected": 0.510548,
"n_labels": 679,
"retention": 1
},
"n20": {
"chance_corrected": 0.089798,
"n_labels": 140,
"retention": 0.20248
},
"n5": {
"chance_corrected": 0.014243,
"n_labels": 35,
"retention": 0.032115
}
},
"score": 41.153195
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0.086957
},
"uncertainty": {
"full_accuracy": 0.771571,
"risk_coverage_auc": 0.923328,
"score": 91.054933,
"selective_acc_at_50": 0.95509,
"selective_acc_at_80": 0.850374
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 7007,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Dermatoscopy",
"n_classes": 7,
"n_test": 2005,
"runtime_s": 16.7,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "dermamnist",
"task_name": "Dermatoscopy (7-class skin lesion)",
"timestamp": "2026-08-01T19:17:31+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 36.642422
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.678,
"auc": 0.922577,
"balanced_accuracy": 0.678,
"ci95": [
0.657351,
0.70015
],
"n_test": 1000,
"score": 57.066667
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.570667,
"mean_agreement": 0.585067,
"mean_kappa": 0.357663,
"mean_retention": 0.523053,
"n_cases": 1000,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.622,
0.475,
0.334
],
"mean_kappa": 0.247795,
"mean_retention": 0.329439,
"retention_by_severity": [
0.5607,
0.3201,
0.1075
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.708,
0.592,
0.5
],
"mean_kappa": 0.33114,
"mean_retention": 0.301402,
"retention_by_severity": [
0.507,
0.2967,
0.1005
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.783,
0.651,
0.524
],
"mean_kappa": 0.522297,
"mean_retention": 0.913551,
"retention_by_severity": [
1,
1,
0.7407
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.533,
0.468,
0.457
],
"mean_kappa": 0.092338,
"mean_retention": 0.095016,
"retention_by_severity": [
0.1939,
0.0584,
0.0327
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.801,
0.694,
0.634
],
"mean_kappa": 0.594745,
"mean_retention": 0.975857,
"retention_by_severity": [
1,
1,
0.9276
],
"scored": true
}
},
"score": 52.305296,
"worst_retention": 0.03271
},
"calibration": {
"accuracy": 0.678,
"brier": 0.508364,
"ece": 0.206592,
"mean_confidence": 0.884592,
"overconfidence": 0.206592,
"score": 0
},
"corruption": {
"clean_reference_cc": 0.570667,
"mean_agreement": 0.53081,
"mean_kappa": 0.281624,
"mean_retention": 0.319648,
"n_cases": 1000,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.719,
0.595,
0.513
],
"mean_kappa": 0.340942,
"mean_retention": 0.313084,
"retention_by_severity": [
0.528,
0.2921,
0.1192
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.78,
0.635,
0.523
],
"mean_kappa": 0.435665,
"mean_retention": 0.46729,
"retention_by_severity": [
0.7079,
0.4509,
0.243
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.609,
0.453,
0.451
],
"mean_kappa": 0.129465,
"mean_retention": 0.130062,
"retention_by_severity": [
0.3364,
0.0257,
0.028
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.424,
0.307,
0.256
],
"mean_kappa": 0.08228,
"mean_retention": 0.140187,
"retention_by_severity": [
0.2967,
0.0818,
0.0421
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.678,
0.513,
0.491
],
"mean_kappa": 0.294225,
"mean_retention": 0.338785,
"retention_by_severity": [
0.486,
0.3715,
0.1589
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.894,
0.737,
0.57
],
"mean_kappa": 0.599565,
"mean_retention": 0.682243,
"retention_by_severity": [
0.9393,
0.6846,
0.4229
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.436,
0.3,
0.263
],
"mean_kappa": 0.089228,
"mean_retention": 0.165888,
"retention_by_severity": [
0.3598,
0.1075,
0.0304
],
"scored": true
}
},
"score": 31.964842,
"worst_retention": 0.025701
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.570667,
"mean_retention": 0.324766,
"per_budget": {
"n100": {
"chance_corrected": 0.556,
"n_labels": 400,
"retention": 0.974299
},
"n20": {
"chance_corrected": 0,
"n_labels": 80,
"retention": 0
},
"n5": {
"chance_corrected": 0,
"n_labels": 20,
"retention": 0
}
},
"score": 32.476636
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 1.866667,
"worst_class_recall": 0.264
},
"uncertainty": {
"full_accuracy": 0.678,
"risk_coverage_auc": 0.835797,
"score": 78.10626,
"selective_acc_at_50": 0.828,
"selective_acc_at_80": 0.73875
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 97477,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "OCT",
"n_classes": 4,
"n_test": 1000,
"runtime_s": 36.3,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "octmnist",
"task_name": "Retinal OCT (4-class)",
"timestamp": "2026-08-01T19:35:47+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 80.561432
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.931432,
"auc": 0.996829,
"balanced_accuracy": 0.922699,
"ci95": [
0.918021,
0.926978
],
"n_test": 17778,
"score": 91.496937
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.919705,
"mean_agreement": 0.787389,
"mean_kappa": 0.763876,
"mean_retention": 0.835561,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9875,
0.9167,
0.7175
],
"mean_kappa": 0.859811,
"mean_retention": 0.923762,
"retention_by_severity": [
0.9988,
0.978,
0.7945
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.86,
0.7383,
0.5892
],
"mean_kappa": 0.700363,
"mean_retention": 0.806403,
"retention_by_severity": [
0.9422,
0.8239,
0.6531
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9408,
0.7983,
0.6342
],
"mean_kappa": 0.766239,
"mean_retention": 0.775105,
"retention_by_severity": [
0.9542,
0.777,
0.5941
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.7733,
0.6675,
0.5458
],
"mean_kappa": 0.627291,
"mean_retention": 0.77018,
"retention_by_severity": [
0.8753,
0.7748,
0.6605
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9617,
0.8967,
0.7833
],
"mean_kappa": 0.865676,
"mean_retention": 0.902354,
"retention_by_severity": [
0.9872,
0.9251,
0.7947
],
"scored": true
}
},
"score": 83.556087,
"worst_retention": 0.594089
},
"calibration": {
"accuracy": 0.931432,
"brier": 0.104176,
"ece": 0.031029,
"mean_confidence": 0.962225,
"overconfidence": 0.030793,
"score": 84.485694
},
"corruption": {
"clean_reference_cc": 0.919705,
"mean_agreement": 0.710516,
"mean_kappa": 0.678368,
"mean_retention": 0.726603,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9192,
0.6717,
0.2392
],
"mean_kappa": 0.574978,
"mean_retention": 0.605434,
"retention_by_severity": [
0.9444,
0.6443,
0.2276
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.855,
0.3567,
0.1542
],
"mean_kappa": 0.40381,
"mean_retention": 0.456673,
"retention_by_severity": [
0.882,
0.3615,
0.1266
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.8317,
0.555,
0.3358
],
"mean_kappa": 0.531501,
"mean_retention": 0.658234,
"retention_by_severity": [
0.9247,
0.658,
0.392
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.98,
0.9233,
0.545
],
"mean_kappa": 0.787217,
"mean_retention": 0.794492,
"retention_by_severity": [
0.9903,
0.9159,
0.4773
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9108,
0.7583,
0.7033
],
"mean_kappa": 0.767267,
"mean_retention": 0.865359,
"retention_by_severity": [
0.9713,
0.832,
0.7928
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9942,
0.9892,
0.9783
],
"mean_kappa": 0.985669,
"mean_retention": 0.993537,
"retention_by_severity": [
0.9966,
0.9967,
0.9873
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9433,
0.8075,
0.4692
],
"mean_kappa": 0.698131,
"mean_retention": 0.712495,
"retention_by_severity": [
0.9556,
0.7984,
0.3835
],
"scored": true
}
},
"score": 72.660343,
"worst_retention": 0.126602
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.914969,
"mean_retention": 0.434471,
"per_budget": {
"n100": {
"chance_corrected": 0.831124,
"n_labels": 1100,
"retention": 0.908363
},
"n20": {
"chance_corrected": 0.296169,
"n_labels": 220,
"retention": 0.323693
},
"n5": {
"chance_corrected": 0.06529,
"n_labels": 55,
"retention": 0.071357
}
},
"score": 43.447091
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 77.776163,
"worst_class_recall": 0.797965
},
"uncertainty": {
"full_accuracy": 0.931432,
"risk_coverage_auc": 0.992153,
"score": 99.13683,
"selective_acc_at_50": 0.998425,
"selective_acc_at_80": 0.992828
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 34561,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 17778,
"runtime_s": 30.2,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organamnist",
"task_name": "Abdominal CT organs — axial (A)",
"timestamp": "2026-08-01T19:33:13+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 79.594617
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.919669,
"auc": 0.99405,
"balanced_accuracy": 0.90922,
"ci95": [
0.902614,
0.916165
],
"n_test": 8216,
"score": 90.014245
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.876982,
"mean_agreement": 0.784111,
"mean_kappa": 0.760243,
"mean_retention": 0.83541,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9875,
0.9292,
0.7083
],
"mean_kappa": 0.859823,
"mean_retention": 0.90103,
"retention_by_severity": [
0.9983,
0.9677,
0.7371
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8733,
0.7533,
0.5875
],
"mean_kappa": 0.710093,
"mean_retention": 0.829637,
"retention_by_severity": [
0.944,
0.8499,
0.6949
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9158,
0.8208,
0.7008
],
"mean_kappa": 0.788765,
"mean_retention": 0.834876,
"retention_by_severity": [
0.9438,
0.8436,
0.7173
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.6892,
0.5525,
0.4617
],
"mean_kappa": 0.524922,
"mean_retention": 0.650335,
"retention_by_severity": [
0.7579,
0.6474,
0.5457
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9758,
0.9342,
0.8717
],
"mean_kappa": 0.917614,
"mean_retention": 0.961174,
"retention_by_severity": [
1,
0.976,
0.9075
],
"scored": true
}
},
"score": 83.541043,
"worst_retention": 0.545704
},
"calibration": {
"accuracy": 0.919669,
"brier": 0.124899,
"ece": 0.039293,
"mean_confidence": 0.958782,
"overconfidence": 0.039113,
"score": 80.353307
},
"corruption": {
"clean_reference_cc": 0.876982,
"mean_agreement": 0.72869,
"mean_kappa": 0.699588,
"mean_retention": 0.751946,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9133,
0.7408,
0.2917
],
"mean_kappa": 0.614051,
"mean_retention": 0.677024,
"retention_by_severity": [
0.9742,
0.7672,
0.2897
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8792,
0.5408,
0.2008
],
"mean_kappa": 0.497414,
"mean_retention": 0.540865,
"retention_by_severity": [
0.9112,
0.5235,
0.1879
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.7825,
0.4667,
0.3267
],
"mean_kappa": 0.481739,
"mean_retention": 0.592335,
"retention_by_severity": [
0.8613,
0.5529,
0.3628
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9642,
0.8917,
0.7017
],
"mean_kappa": 0.832162,
"mean_retention": 0.862097,
"retention_by_severity": [
0.9855,
0.8977,
0.7031
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.92,
0.5933,
0.7433
],
"mean_kappa": 0.719758,
"mean_retention": 0.782324,
"retention_by_severity": [
0.9708,
0.5765,
0.7996
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9933,
0.9842,
0.965
],
"mean_kappa": 0.97834,
"mean_retention": 0.996824,
"retention_by_severity": [
0.9985,
1,
0.9919
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9233,
0.8192,
0.6608
],
"mean_kappa": 0.773653,
"mean_retention": 0.812152,
"retention_by_severity": [
0.9425,
0.8374,
0.6565
],
"scored": true
}
},
"score": 75.19459,
"worst_retention": 0.187869
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.900142,
"mean_retention": 0.41967,
"per_budget": {
"n100": {
"chance_corrected": 0.838214,
"n_labels": 1100,
"retention": 0.931202
},
"n20": {
"chance_corrected": 0.285778,
"n_labels": 220,
"retention": 0.317481
},
"n5": {
"chance_corrected": 0.009298,
"n_labels": 55,
"retention": 0.010329
}
},
"score": 41.967047
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 75.639615,
"worst_class_recall": 0.778542
},
"uncertainty": {
"full_accuracy": 0.919669,
"risk_coverage_auc": 0.989018,
"score": 98.791999,
"selective_acc_at_50": 0.997322,
"selective_acc_at_80": 0.986916
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 12975,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 8216,
"runtime_s": 38.1,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organcmnist",
"task_name": "Abdominal CT organs — coronal (C)",
"timestamp": "2026-08-01T19:28:20+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 64.100882
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.792228,
"auc": 0.978035,
"balanced_accuracy": 0.749776,
"ci95": [
0.7408,
0.75798
],
"n_test": 8827,
"score": 72.475324
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.722033,
"mean_agreement": 0.702833,
"mean_kappa": 0.664866,
"mean_retention": 0.781997,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.975,
0.835,
0.5808
],
"mean_kappa": 0.769526,
"mean_retention": 0.864886,
"retention_by_severity": [
1,
0.9617,
0.6329
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.785,
0.6467,
0.5025
],
"mean_kappa": 0.601249,
"mean_retention": 0.744786,
"retention_by_severity": [
0.9155,
0.7387,
0.5801
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8858,
0.7158,
0.56
],
"mean_kappa": 0.68117,
"mean_retention": 0.774015,
"retention_by_severity": [
0.948,
0.7799,
0.5942
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.6308,
0.5342,
0.4592
],
"mean_kappa": 0.490679,
"mean_retention": 0.629866,
"retention_by_severity": [
0.7242,
0.6355,
0.5299
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.93,
0.8383,
0.6633
],
"mean_kappa": 0.781708,
"mean_retention": 0.89643,
"retention_by_severity": [
0.9888,
0.9237,
0.7768
],
"scored": true
}
},
"score": 78.199663,
"worst_retention": 0.529926
},
"calibration": {
"accuracy": 0.792228,
"brier": 0.296927,
"ece": 0.092168,
"mean_confidence": 0.884397,
"overconfidence": 0.092168,
"score": 53.915875
},
"corruption": {
"clean_reference_cc": 0.722033,
"mean_agreement": 0.629008,
"mean_kappa": 0.574335,
"mean_retention": 0.651781,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.8208,
0.5192,
0.2575
],
"mean_kappa": 0.469462,
"mean_retention": 0.538917,
"retention_by_severity": [
0.8961,
0.5317,
0.189
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.6992,
0.3283,
0.1858
],
"mean_kappa": 0.34016,
"mean_retention": 0.419347,
"retention_by_severity": [
0.7862,
0.3269,
0.1449
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.7217,
0.4333,
0.3125
],
"mean_kappa": 0.439614,
"mean_retention": 0.559454,
"retention_by_severity": [
0.8261,
0.5094,
0.3428
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9417,
0.8183,
0.3975
],
"mean_kappa": 0.655566,
"mean_retention": 0.728886,
"retention_by_severity": [
0.9772,
0.887,
0.3225
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.85,
0.4333,
0.6625
],
"mean_kappa": 0.596312,
"mean_retention": 0.705502,
"retention_by_severity": [
0.9624,
0.4215,
0.7325
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9883,
0.98,
0.94
],
"mean_kappa": 0.96474,
"mean_retention": 0.989163,
"retention_by_severity": [
1,
0.9991,
0.9684
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.8683,
0.7117,
0.3392
],
"mean_kappa": 0.554494,
"mean_retention": 0.621202,
"retention_by_severity": [
0.9398,
0.7072,
0.2166
],
"scored": true
}
},
"score": 65.178139,
"worst_retention": 0.144947
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.724753,
"mean_retention": 0.406912,
"per_budget": {
"n100": {
"chance_corrected": 0.655458,
"n_labels": 1100,
"retention": 0.904388
},
"n20": {
"chance_corrected": 0.229275,
"n_labels": 220,
"retention": 0.316349
},
"n5": {
"chance_corrected": 0,
"n_labels": 55,
"retention": 0
}
},
"score": 40.691244
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 37.6082,
"worst_class_recall": 0.432802
},
"uncertainty": {
"full_accuracy": 0.792228,
"risk_coverage_auc": 0.938391,
"score": 93.222958,
"selective_acc_at_50": 0.973267,
"selective_acc_at_80": 0.876097
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 13932,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 8827,
"runtime_s": 25.4,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organsmnist",
"task_name": "Abdominal CT organs — sagittal (S)",
"timestamp": "2026-08-01T19:30:56+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 56.68336
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.848607,
"auc": 0.965322,
"balanced_accuracy": 0.811026,
"ci95": [
0.80097,
0.820898
],
"n_test": 7180,
"score": 78.7404
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.796619,
"mean_agreement": 0.484333,
"mean_kappa": 0.419403,
"mean_retention": 0.464125,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.8217,
0.2375,
0.125
],
"mean_kappa": 0.332934,
"mean_retention": 0.379424,
"retention_by_severity": [
0.8731,
0.2187,
0.0464
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.725,
0.675,
0.5867
],
"mean_kappa": 0.611504,
"mean_retention": 0.652849,
"retention_by_severity": [
0.7339,
0.664,
0.5607
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8717,
0.6458,
0.4692
],
"mean_kappa": 0.612889,
"mean_retention": 0.666107,
"retention_by_severity": [
0.9031,
0.6423,
0.4529
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.5217,
0.45,
0.3817
],
"mean_kappa": 0.370622,
"mean_retention": 0.407381,
"retention_by_severity": [
0.5001,
0.4109,
0.3112
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.3775,
0.2308,
0.1458
],
"mean_kappa": 0.169064,
"mean_retention": 0.214865,
"retention_by_severity": [
0.3973,
0.1836,
0.0638
],
"scored": true
}
},
"score": 46.412525,
"worst_retention": 0.04645
},
"calibration": {
"accuracy": 0.848607,
"brier": 0.24416,
"ece": 0.077413,
"mean_confidence": 0.92602,
"overconfidence": 0.077413,
"score": 61.2935
},
"corruption": {
"clean_reference_cc": 0.796619,
"mean_agreement": 0.413373,
"mean_kappa": 0.344685,
"mean_retention": 0.382924,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.4442,
0.2383,
0.2167
],
"mean_kappa": 0.219092,
"mean_retention": 0.258912,
"retention_by_severity": [
0.4543,
0.1678,
0.1547
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8142,
0.5933,
0.3567
],
"mean_kappa": 0.525403,
"mean_retention": 0.589015,
"retention_by_severity": [
0.867,
0.5949,
0.3052
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.63,
0.3717,
0.3275
],
"mean_kappa": 0.35723,
"mean_retention": 0.389592,
"retention_by_severity": [
0.6356,
0.2997,
0.2336
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.2575,
0.1092,
0.0825
],
"mean_kappa": 0.072155,
"mean_retention": 0.102473,
"retention_by_severity": [
0.2592,
0.0241,
0.0241
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.7758,
0.4992,
0.5567
],
"mean_kappa": 0.549219,
"mean_retention": 0.573188,
"retention_by_severity": [
0.785,
0.4159,
0.5186
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.955,
0.8067,
0.3692
],
"mean_kappa": 0.677348,
"mean_retention": 0.737653,
"retention_by_severity": [
0.9832,
0.8677,
0.3621
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.1158,
0.0858,
0.075
],
"mean_kappa": 0.012349,
"mean_retention": 0.029633,
"retention_by_severity": [
0.0414,
0.0146,
0.0329
],
"scored": true
}
},
"score": 38.292373,
"worst_retention": 0.014553
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.787404,
"mean_retention": 0.420779,
"per_budget": {
"n100": {
"chance_corrected": 0.604068,
"n_labels": 900,
"retention": 0.767164
},
"n20": {
"chance_corrected": 0.320459,
"n_labels": 180,
"retention": 0.406981
},
"n5": {
"chance_corrected": 0.069442,
"n_labels": 45,
"retention": 0.088191
}
},
"score": 42.077896
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 38.539192,
"worst_class_recall": 0.453682
},
"uncertainty": {
"full_accuracy": 0.848607,
"risk_coverage_auc": 0.944166,
"score": 93.718695,
"selective_acc_at_50": 0.953203,
"selective_acc_at_80": 0.913301
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 89996,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "Histopathology",
"n_classes": 9,
"n_test": 7180,
"runtime_s": 40.3,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "pathmnist",
"task_name": "Colorectal pathology (9-class tissue)",
"timestamp": "2026-08-01T19:42:24+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 61.913727
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.900641,
"auc": 0.968606,
"balanced_accuracy": 0.874359,
"ci95": [
0.849435,
0.902377
],
"n_test": 624,
"score": 74.871795
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.748718,
"mean_agreement": 0.87938,
"mean_kappa": 0.680321,
"mean_retention": 0.727093,
"n_cases": 624,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9696,
0.9279,
0.6699
],
"mean_kappa": 0.722389,
"mean_retention": 0.903349,
"retention_by_severity": [
1,
1,
0.71
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.859,
0.7468,
0.6987
],
"mean_kappa": 0.274432,
"mean_retention": 0.247336,
"retention_by_severity": [
0.5708,
0.1712,
0
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9744,
0.9439,
0.8974
],
"mean_kappa": 0.84745,
"mean_retention": 0.883942,
"retention_by_severity": [
0.9977,
0.9053,
0.7489
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9054,
0.8942,
0.8077
],
"mean_kappa": 0.638753,
"mean_retention": 0.600837,
"retention_by_severity": [
0.7272,
0.6872,
0.3881
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9824,
0.9647,
0.9487
],
"mean_kappa": 0.918579,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
}
},
"score": 72.709285,
"worst_retention": 0
},
"calibration": {
"accuracy": 0.900641,
"brier": 0.162961,
"ece": 0.070922,
"mean_confidence": 0.971563,
"overconfidence": 0.070922,
"score": 64.538792
},
"corruption": {
"clean_reference_cc": 0.748718,
"mean_agreement": 0.747634,
"mean_kappa": 0.487033,
"mean_retention": 0.553001,
"n_cases": 624,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9487,
0.8494,
0.7228
],
"mean_kappa": 0.528407,
"mean_retention": 0.531583,
"retention_by_severity": [
0.9155,
0.5685,
0.1107
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.9295,
0.8045,
0.6987
],
"mean_kappa": 0.417033,
"mean_retention": 0.396499,
"retention_by_severity": [
0.8128,
0.3767,
0
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9343,
0.8061,
0.7019
],
"mean_kappa": 0.428219,
"mean_retention": 0.407915,
"retention_by_severity": [
0.8299,
0.3824,
0.0114
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9119,
0.4615,
0.3013
],
"mean_kappa": 0.319598,
"mean_retention": 0.4414,
"retention_by_severity": [
1,
0.3242,
0
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9375,
0.8974,
0.8221
],
"mean_kappa": 0.752833,
"mean_retention": 0.950152,
"retention_by_severity": [
0.8505,
1,
1
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.984,
0.9744,
0.9006
],
"mean_kappa": 0.895122,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.5048,
0.3077,
0.3013
],
"mean_kappa": 0.068015,
"mean_retention": 0.143455,
"retention_by_severity": [
0.4167,
0.0137,
0
],
"scored": true
}
},
"score": 55.300065,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.748718,
"mean_retention": 0,
"per_budget": {
"n100": {
"chance_corrected": 0,
"n_labels": 200,
"retention": 0
},
"n20": {
"chance_corrected": 0,
"n_labels": 40,
"retention": 0
},
"n5": {
"chance_corrected": 0,
"n_labels": 10,
"retention": 0
}
},
"score": 0
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 53.846154,
"worst_class_recall": 0.769231
},
"uncertainty": {
"full_accuracy": 0.900641,
"risk_coverage_auc": 0.975418,
"score": 95.083525,
"selective_acc_at_50": 0.987179,
"selective_acc_at_80": 0.963928
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 4708,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Chest X-ray",
"n_classes": 2,
"n_test": 624,
"runtime_s": 9.1,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "pneumoniamnist",
"task_name": "Paediatric chest X-ray (pneumonia)",
"timestamp": "2026-08-01T19:16:26+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 54.747757
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.52,
"auc": 0.716715,
"balanced_accuracy": 0.33791,
"ci95": [
0.307025,
0.373363
],
"n_test": 400,
"score": 17.238807
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.172388,
"mean_agreement": 0.8015,
"mean_kappa": 0.706372,
"mean_retention": 0.893282,
"n_cases": 400,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.955,
0.9025,
0.8325
],
"mean_kappa": 0.84486,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.935,
0.8925,
0.805
],
"mean_kappa": 0.815123,
"mean_retention": 0.930762,
"retention_by_severity": [
0.9434,
0.9036,
0.9453
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.83,
0.6325,
0.5125
],
"mean_kappa": 0.485451,
"mean_retention": 0.856788,
"retention_by_severity": [
0.9731,
0.9127,
0.6846
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9625,
0.9475,
0.9325
],
"mean_kappa": 0.919558,
"mean_retention": 0.993513,
"retention_by_severity": [
0.9805,
1,
1
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.785,
0.6075,
0.49
],
"mean_kappa": 0.46687,
"mean_retention": 0.685347,
"retention_by_severity": [
0.8359,
0.847,
0.3732
],
"scored": true
}
},
"score": 89.328189,
"worst_retention": 0.373159
},
"calibration": {
"accuracy": 0.52,
"brier": 0.590948,
"ece": 0.039781,
"mean_confidence": 0.495429,
"overconfidence": -0.024571,
"score": 80.109489
},
"corruption": {
"clean_reference_cc": 0.172388,
"mean_agreement": 0.813333,
"mean_kappa": 0.72937,
"mean_retention": 0.943214,
"n_cases": 400,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.8025,
0.675,
0.3025
],
"mean_kappa": 0.443145,
"mean_retention": 0.71089,
"retention_by_severity": [
0.9899,
0.89,
0.2528
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8425,
0.6825,
0.62
],
"mean_kappa": 0.580462,
"mean_retention": 0.922481,
"retention_by_severity": [
0.949,
0.9314,
0.887
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9675,
0.9425,
0.9075
],
"mean_kappa": 0.906883,
"mean_retention": 0.969124,
"retention_by_severity": [
0.9648,
0.9954,
0.9472
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9125,
0.8,
0.67
],
"mean_kappa": 0.693887,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.96,
0.9525,
0.9225
],
"mean_kappa": 0.916338,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9725,
0.93,
0.865
],
"mean_kappa": 0.88401,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.8675,
0.7575,
0.7275
],
"mean_kappa": 0.680863,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
}
},
"score": 94.321353,
"worst_retention": 0.252787
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.172388,
"mean_retention": 0.361393,
"per_budget": {
"n100": {
"chance_corrected": 0.212155,
"n_labels": 466,
"retention": 1
},
"n20": {
"chance_corrected": 0.007184,
"n_labels": 100,
"retention": 0.041673
},
"n5": {
"chance_corrected": 0.007328,
"n_labels": 25,
"retention": 0.042506
}
},
"score": 36.139308
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0
},
"uncertainty": {
"full_accuracy": 0.52,
"risk_coverage_auc": 0.717889,
"score": 64.736128,
"selective_acc_at_50": 0.705,
"selective_acc_at_80": 0.5875
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 1080,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Fundus",
"n_classes": 5,
"n_test": 400,
"runtime_s": 4.3,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "retinamnist",
"task_name": "Fundus photography (DR grade)",
"timestamp": "2026-08-01T19:13:18+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 50.501494
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.592809,
"auc": 0.892541,
"balanced_accuracy": 0.489413,
"ci95": [
0.484492,
0.4951
],
"n_test": 47280,
"score": 41.647253
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.395455,
"mean_agreement": 0.4975,
"mean_kappa": 0.375259,
"mean_retention": 0.583909,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.5958,
0.3817,
0.2483
],
"mean_kappa": 0.306152,
"mean_retention": 0.51586,
"retention_by_severity": [
0.7317,
0.486,
0.3298
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.7767,
0.6917,
0.5567
],
"mean_kappa": 0.580545,
"mean_retention": 0.792985,
"retention_by_severity": [
0.9192,
0.824,
0.6358
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.5317,
0.38,
0.3442
],
"mean_kappa": 0.263092,
"mean_retention": 0.442412,
"retention_by_severity": [
0.6699,
0.4059,
0.2515
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.6908,
0.6067,
0.5575
],
"mean_kappa": 0.527708,
"mean_retention": 0.798538,
"retention_by_severity": [
0.8877,
0.7682,
0.7398
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.4092,
0.3642,
0.3275
],
"mean_kappa": 0.1988,
"mean_retention": 0.36975,
"retention_by_severity": [
0.5264,
0.341,
0.2419
],
"scored": true
}
},
"score": 58.390874,
"worst_retention": 0.241852
},
"calibration": {
"accuracy": 0.592809,
"brier": 0.540139,
"ece": 0.012568,
"mean_confidence": 0.605165,
"overconfidence": 0.012356,
"score": 93.716198
},
"corruption": {
"clean_reference_cc": 0.395455,
"mean_agreement": 0.493889,
"mean_kappa": 0.367043,
"mean_retention": 0.482268,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.5067,
0.3858,
0.3633
],
"mean_kappa": 0.235683,
"mean_retention": 0.361514,
"retention_by_severity": [
0.5582,
0.2758,
0.2506
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.6233,
0.4725,
0.4067
],
"mean_kappa": 0.329419,
"mean_retention": 0.328757,
"retention_by_severity": [
0.5515,
0.2867,
0.148
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.75,
0.5667,
0.4742
],
"mean_kappa": 0.501726,
"mean_retention": 0.744164,
"retention_by_severity": [
0.9487,
0.7357,
0.5481
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.385,
0.2558,
0.1733
],
"mean_kappa": 0.131399,
"mean_retention": 0.2462,
"retention_by_severity": [
0.4579,
0.274,
0.0067
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.8408,
0.6317,
0.6783
],
"mean_kappa": 0.634637,
"mean_retention": 0.682928,
"retention_by_severity": [
0.8728,
0.5901,
0.5859
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.8383,
0.7017,
0.3867
],
"mean_kappa": 0.545015,
"mean_retention": 0.656888,
"retention_by_severity": [
0.9174,
0.7146,
0.3387
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.4117,
0.3083,
0.2108
],
"mean_kappa": 0.191426,
"mean_retention": 0.355426,
"retention_by_severity": [
0.5618,
0.3994,
0.1051
],
"scored": true
}
},
"score": 48.226827,
"worst_retention": 0.006732
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.416473,
"mean_retention": 0.306291,
"per_budget": {
"n100": {
"chance_corrected": 0.325091,
"n_labels": 800,
"retention": 0.780583
},
"n20": {
"chance_corrected": 0.057595,
"n_labels": 160,
"retention": 0.138291
},
"n5": {
"chance_corrected": 0,
"n_labels": 40,
"retention": 0
}
},
"score": 30.629144
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 8.43836,
"worst_class_recall": 0.198836
},
"uncertainty": {
"full_accuracy": 0.592809,
"risk_coverage_auc": 0.75998,
"score": 72.569134,
"selective_acc_at_50": 0.75956,
"selective_acc_at_80": 0.656647
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 165466,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "Microscopy",
"n_classes": 8,
"n_test": 47280,
"runtime_s": 41.8,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "tissuemnist",
"task_name": "Kidney cortex cells (8-class)",
"timestamp": "2026-08-01T19:39:02+00:00"
}
],
"heldout": null,
"sources": [
{
"path": "results/cross_site/organ_ct__cnn_scratch.json",
"sha256": "2cb33bd19620e036d679332fd717024b5b74f1cae25c506645558f82ba8018ac"
},
{
"path": "results/heldout/audit_report.json",
"sha256": "df460b6d91a6512dfa1e7dc650637fce2b4dd3a1ec32e03446bf6e11f9c597d2"
},
{
"path": "results/heldout/cycle.json",
"sha256": "e012b8978457b51b5c31d18d14d2cc92f100731897523306acae19934f943502"
},
{
"path": "results/heldout/manifests/gen1.seal.json",
"sha256": "91a7adfa29870a0719125cdd48f6dcb149f1f6485ac1cfa104c6e00057ad9b81"
},
{
"path": "results/heldout/manifests/gen2.seal.json",
"sha256": "f61470918a09cf39e074ef14fb7ade4ac3a13d36d9385f18cf9b92e4b5329547"
},
{
"path": "results/heldout/manifests/public.seal.json",
"sha256": "1a4fd930d2e781ac7163483990f15d6694e28e507d1b3ca009ff09395e47497c"
},
{
"path": "results/records/bloodmnist__cnn_scratch.json",
"sha256": "658e3e5f7cfa4ad5fd6c7b39ba0aeafd63cd24b954494a2ea543b80339ff90ee"
},
{
"path": "results/records/breastmnist__cnn_scratch.json",
"sha256": "2212aec836e7f76f734e4219f2440104ddf80ee2db0595211786509d77fc9ad6"
},
{
"path": "results/records/dermamnist__cnn_scratch.json",
"sha256": "ecafb8abee712c7ae14cefcbc3e079ecdbbec6ca0eaf1915cd70d66b033443c4"
},
{
"path": "results/records/octmnist__cnn_scratch.json",
"sha256": "051221f9e35353e03d86cf3fc99a668fc38a9dde7a433eeee6c08430ff7577d7"
},
{
"path": "results/records/organamnist__cnn_scratch.json",
"sha256": "ac309b752caab9b09114ef594183239dc8b5fdb3732f0e694a9bf2aef0a9e46d"
},
{
"path": "results/records/organcmnist__cnn_scratch.json",
"sha256": "b00ac3f602f5687f42d5649af03dea831ae0f80f650542f3bbc20a3766d12225"
},
{
"path": "results/records/organsmnist__cnn_scratch.json",
"sha256": "83a9069311350c3760a04eb18c472285d1d26facd163ea735ef6448af4187a95"
},
{
"path": "results/records/pathmnist__cnn_scratch.json",
"sha256": "19764268891a48ac331ae275acc81c85b16828b02614ee8f32abda10afd4d909"
},
{
"path": "results/records/pneumoniamnist__cnn_scratch.json",
"sha256": "e15163c863218cf6062262c8bfc0c3800e7a3b6995751ffb2802e5a204876209"
},
{
"path": "results/records/retinamnist__cnn_scratch.json",
"sha256": "702b5c2292a18b58c0d1deb42fee6ea2dc74f3c6afa7e10203958a5ec14b8b01"
},
{
"path": "results/records/tissuemnist__cnn_scratch.json",
"sha256": "792813023724858e90cb26cab9ff90c5a00925ee563ecf98bccb4e4c08ae4396"
}
]
},
"declared": null,
"declared_note": "No vendor declaration. This is an open reference baseline implemented by NakedSignal, not a commercial product: there is no vendor to assert an intended use or to sign an attestation, so the declared block is null rather than filled in on the vendor's behalf.",
"expiry_basis": "the sealed held-out generation this evidence cycle is anchored to (gen2, set hash ad30d214daa285c5...) is published to expire 2026-09-20; per spec, a passport lapses when its generation retires or its spec version is superseded, whichever comes first",
"expiry_utc": "2026-09-20T00:00:00Z",
"issued_utc": "2026-08-03T16:03:42Z",
"issuer": {
"algo": "Ed25519",
"key_id": "ns-passport-2026-07",
"name": "NakedSignal",
"public_key_b64": "0PBCC6mjzcppR5QdPcK7vP/AamB6MP8l1T4ypMYbsMw="
},
"observed": null,
"observed_note": "Not available: this requires continuous monitoring. Production history and drift are observed fields, and nothing is deployed, so there is nothing to observe. The object is present and null on purpose. The shape is the roadmap, not a claim.",
"passport_version": "0.1",
"signature": "XcWDZfObYGpNaSmO3H0fx5rwHAJishZmoA52xWxcKeJB6L13mJrPoUPs/cdaja2oE03MAiOz883Cq6cwJJuDAw==",
"spec_version": "MedEval-1 v0.1",
"subject": {
"behaviour_version": "1.0.0",
"code_fingerprint": "b444c02112cc",
"code_fingerprints": [
"b444c02112cc"
],
"family": "trained",
"model_id": "cnn_scratch",
"model_name": "Small CNN, trained from scratch",
"params": "3 conv blocks, 20 epochs, adam 1e-3, early stop on val"
}
}Canonicalisation: RFC 8785 (JCS)-compatible: UTF-8, object keys sorted by code point, no insignificant whitespace, ECMAScript number formatting. every float is rounded once at emit to 6 decimal places (Python round(), banker's rounding); integers are left exact. Signed over the whole document with the `signature` member removed, canonicalised as above and encoded UTF-8. The public key for ns-passport-2026-07 is 0PBCC6mjzcppR5QdPcK7vP/AamB6MP8l1T4ypMYbsMw= — see Governance.
Every passport issued under this key
One per model on the leaderboard, emitted by python src/passport.py --all. Each was signature-checked when this page was built, by the same code the button above runs: 42 of 42 verify.
| Subject | Evaluations | Tasks | Expires | Signature |
|---|---|---|---|---|
| baseline_only_ancova | 1 | procova_validity | 2026-09-20 | verified |
| baseline_only_ancova_ad | 1 | procova_validity_ad | 2026-09-20 | verified |
| biomedbert_probe | 3 | jobpost_fraud, ledgar, medabstracts | 2026-09-20 | verified |
| biomedclip_ft | 1 | breastmnist_224 | 2026-09-20 | verified |
| biomedclip_probe | 12 | bloodmnist_224, bloodmnist, breastmnist_224, breastmnist, dermamnist_224, dermamnist, organcmnist_224, organcmnist, pneumoniamnist_224, pneumoniamnist, retinamnist_224, retinamnist | 2026-09-20 | verified |
| biomedclip_zeroshot | 12 | bloodmnist_224, bloodmnist, breastmnist_224, breastmnist, dermamnist_224, dermamnist, organcmnist_224, organcmnist, pneumoniamnist_224, pneumoniamnist, retinamnist_224, retinamnist | 2026-09-20 | verified |
| cnn_scratch | 11 | bloodmnist, breastmnist, dermamnist, octmnist, organamnist, organcmnist, organsmnist, pathmnist, pneumoniamnist, retinamnist, tissuemnist | 2026-09-20 | verified |
| covariate_shopping_20 | 1 | procova_validity | 2026-09-20 | verified |
| covariate_shopping_20_ad | 1 | procova_validity_ad | 2026-09-20 | verified |
| dinov2_probe | 12 | bloodmnist_224, bloodmnist, breastmnist_224, breastmnist, dermamnist_224, dermamnist, organcmnist_224, organcmnist, pneumoniamnist_224, pneumoniamnist, retinamnist_224, retinamnist | 2026-09-20 | verified |
| gte_probe | 3 | jobpost_fraud, ledgar, medabstracts | 2026-09-20 | verified |
| hog_logreg | 11 | bloodmnist, breastmnist, dermamnist, octmnist, organamnist, organcmnist, organsmnist, pathmnist, pneumoniamnist, retinamnist, tissuemnist | 2026-09-20 | verified |
| mlp_pixels | 11 | bloodmnist, breastmnist, dermamnist, octmnist, organamnist, organcmnist, organsmnist, pathmnist, pneumoniamnist, retinamnist, tissuemnist | 2026-09-20 | verified |
| pca_logreg | 11 | bloodmnist, breastmnist, dermamnist, octmnist, organamnist, organcmnist, organsmnist, pathmnist, pneumoniamnist, retinamnist, tissuemnist | 2026-09-20 | verified |
| pubmedbert_st_probe | 3 | jobpost_fraud, ledgar, medabstracts | 2026-09-20 | verified |
| resnet18_ft | 11 | bloodmnist, breastmnist, dermamnist, octmnist, organamnist, organcmnist, organsmnist, pathmnist, pneumoniamnist, retinamnist, tissuemnist | 2026-09-20 | verified |
| resnet18_in_probe | 11 | bloodmnist, breastmnist, dermamnist, octmnist, organamnist, organcmnist, organsmnist, pathmnist, pneumoniamnist, retinamnist, tissuemnist | 2026-09-20 | verified |
| rf_pixels | 11 | bloodmnist, breastmnist, dermamnist, octmnist, organamnist, organcmnist, organsmnist, pathmnist, pneumoniamnist, retinamnist, tissuemnist | 2026-09-20 | verified |
| tab_gbm | 15 | adult_income, anuran_mfcc, breast_wdbc, coil2000_caravan, diabetes_pima, dry_bean, german_credit, heart_cleveland, landsat_statlog, occupancy_detection_2, occupancy_detection, secom, steel_plate_faults, wine_quality_red, wine_quality_white | 2026-09-20 | verified |
| tab_logreg | 15 | adult_income, anuran_mfcc, breast_wdbc, coil2000_caravan, diabetes_pima, dry_bean, german_credit, heart_cleveland, landsat_statlog, occupancy_detection_2, occupancy_detection, secom, steel_plate_faults, wine_quality_red, wine_quality_white | 2026-09-20 | verified |
| tab_mlp | 15 | adult_income, anuran_mfcc, breast_wdbc, coil2000_caravan, diabetes_pima, dry_bean, german_credit, heart_cleveland, landsat_statlog, occupancy_detection_2, occupancy_detection, secom, steel_plate_faults, wine_quality_red, wine_quality_white | 2026-09-20 | verified |
| tab_rf | 15 | adult_income, anuran_mfcc, breast_wdbc, coil2000_caravan, diabetes_pima, dry_bean, german_credit, heart_cleveland, landsat_statlog, occupancy_detection_2, occupancy_detection, secom, steel_plate_faults, wine_quality_red, wine_quality_white | 2026-09-20 | verified |
| tfidf_logreg | 3 | jobpost_fraud, ledgar, medabstracts | 2026-09-20 | verified |
| tfidf_mlp | 3 | jobpost_fraud, ledgar, medabstracts | 2026-09-20 | verified |
| tfidf_nb | 3 | jobpost_fraud, ledgar, medabstracts | 2026-09-20 | verified |
| tfidf_svm | 3 | jobpost_fraud, ledgar, medabstracts | 2026-09-20 | verified |
| twin_broken_anticorrelated | 1 | procova_validity | 2026-09-20 | verified |
| twin_broken_anticorrelated_ad | 1 | procova_validity_ad | 2026-09-20 | verified |
| twin_broken_nonmonotone | 1 | procova_validity | 2026-09-20 | verified |
| twin_broken_nonmonotone_ad | 1 | procova_validity_ad | 2026-09-20 | verified |
| twin_constant | 1 | procova_validity | 2026-09-20 | verified |
| twin_constant_ad | 1 | procova_validity_ad | 2026-09-20 | verified |
| twin_r2_0.40 | 1 | procova_validity | 2026-09-20 | verified |
| twin_r2_0.40_ad | 1 | procova_validity_ad | 2026-09-20 | verified |
| twin_r2_0.60 | 1 | procova_validity | 2026-09-20 | verified |
| twin_r2_0.60_ad | 1 | procova_validity_ad | 2026-09-20 | verified |
| twin_r2_0.80 | 1 | procova_validity | 2026-09-20 | verified |
| twin_r2_0.80_ad | 1 | procova_validity_ad | 2026-09-20 | verified |
| twin_useless_r2_0.0 | 1 | procova_validity | 2026-09-20 | verified |
| twin_useless_r2_0.0_ad | 1 | procova_validity_ad | 2026-09-20 | verified |
| twin_wrong_population_age | 1 | procova_validity | 2026-09-20 | verified |
| twin_wrong_population_age_ad | 1 | procova_validity_ad | 2026-09-20 | verified |
Checking one yourself
The browser check above is the same check a hospital's CI runs, and the command that runs it is in the rail. The verifier trusts the public key published in plan_d/docs/GOVERNANCE.md and on Governance — never the key inside the document it is checking. It refuses a passport carrying a different key under the same key id, because a document that supplies its own trust anchor certifies itself and proves nothing.
What is not in these passports
The declared block is null on every one of them, and that is the honest answer rather than a gap: the models on this leaderboard are open reference implementations we wrote ourselves, so there is no vendor to assert an intended use or sign an attestation. The mechanism exists and is finished at both ends — a submitter fills it in at Vendor declaration, the emitter reads plan_d/declarations/<model>.yaml and refuses to emit a half-filled one. You can see the panel populated: example-declared.json in the sample bag is a signed specimen carrying a deliberately fictional vendor, kept because a panel nobody has seen with something in it is a panel nobody has checked. The observed block is null on every one of them too, and stays null until continuous monitoring exists. Neither is hidden; both are rendered as their own panel above, empty, because the shape of the document is part of what is being proposed.
Also deliberately absent: JWT and FHIR renderings, and a revocation list. Each is a permanent maintenance commitment nobody has asked for, and a registry is not useful before there are passports to register.