Back to the consoleDocumentation
Random forest on raw pixels
LIVEThe signed evidence record for rf_pixels. Everything below was read out of the passport file; the signature was checked when this page was built, by the same code the Verify button runs.
SIGNATURE VALIDexit 0signature valid, issued by NakedSignal under key ns-passport-2026-07, not expired
SubjectRandom forest on raw pixels
rf_pixelsConfiguration200 trees, sqrt features
Code fingerprintb444c02112cc
Spec versionMedEval-1 v0.1
Issued2026-08-03T16:03:42Z by NakedSignal
Expires2026-09-20T00:00:00Z — the sealed held-out generation this evidence cycle is anchored to (gen2, set hash ad30d214daa285c5...) is published to expire 2026-09-20; per spec, a passport lapses when its generation retires or its spec version is superseded, whichever comes first
key ns-passport-2026-07 · Ed25519
The check runs against the public key published on the governance page, using your browser's own Ed25519 implementation. The exit code shown is the exit code verify_passport.py returns for the same document.
Measured by NakedSignal
Computed in this repo by the MedEval-1 harness on data the model had never seen. Every number below was copied out of a file the harness wrote — 17 of them, each listed with its SHA-256 at the foot of this panel — and is printed exactly as it was signed, not re-rounded for display.
| Task | Domain | n test | accuracy | calibration | uncertainty | subgroup | corruption | acquisition | limited data | cost | spec sensitivity | Index |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bloodmnist | medicine | 3421 | 75.898528 | 2.59354 | 95.363039 | 42.454728 | 70.646134 | 79.754018 | 80.520515 | n/a | n/a | 62.852521 |
| breastmnist | medicine | 156 | 39.348371 | 73.141026 | 84.053008 | 0 | 67.955717 | 80.700637 | 71.33758 | n/a | n/a | 57.648922 |
| dermamnist | medicine | 2005 | 15.274405 | 78.990025 | 88.261491 | 0 | 80.280482 | 72.347973 | 100 | n/a | n/a | 55.708505 |
| octmnist | medicine | 1000 | 33.066667 | 67.717917 | 46.536214 | 0 | 53.43702 | 60.806452 | 39.919355 | n/a | n/a | 44.288353 |
| organamnist | medicine | 17778 | 74.30351 | 0 | 93.028 | 52.473282 | 72.145035 | 89.045305 | 81.937316 | n/a | n/a | 65.211789 |
| organcmnist | medicine | 8216 | 78.236078 | 0.815786 | 95.824864 | 57.047619 | 70.446135 | 88.776197 | 77.591393 | n/a | n/a | 66.264852 |
| organsmnist | medicine | 8827 | 57.691193 | 12.409935 | 83.836529 | 26.190476 | 69.077012 | 87.481896 | 73.516957 | n/a | n/a | 57.910437 |
| pathmnist | medicine | 7180 | 47.37752 | 0 | 76.336375 | 5.566802 | 75.305394 | 81.002982 | 75.701452 | n/a | n/a | 50.748238 |
| pneumoniamnist | medicine | 624 | 63.846154 | 82.5 | 89.044613 | 33.333333 | 71.626187 | 75.385988 | 76.662204 | n/a | n/a | 69.138178 |
| retinamnist | medicine | 400 | 18.131743 | 65.23125 | 62.789615 | 0 | 70.612274 | 69.973992 | 64.296455 | n/a | n/a | 47.790494 |
| tissuemnist | medicine | 47280 | 19.776074 | 61.618972 | 63.618374 | 0 | 71.929792 | 72.524243 | 78.647805 | n/a | n/a | 49.700267 |
Cross-site retention · organ_ct
retention is absolute retained skill (chance-corrected), per spec 2.2.1; the diagonal is same-site, every off-diagonal cell is train-on-row, test-on-column
Sitesorganamnist, organcmnist, organsmnist
Mean retention0.561744
Worst retention0.3626
Private held-out track
PUBLIC-DATA DEMONSTRATION OF THE PRIVATE PROTOCOL
| Generation | Public | Private (equated) | Gap | z | Verdict |
|---|---|---|---|---|---|
| gen1 | 75.628809 | 75.045419 | 0.58339 | 0.26155 | clear |
| gen1 | 76.768967 | 100 | -23.231033 | -15.114489 | FLAG |
| gen2 | 76.867888 | 72.938472 | 3.929416 | 0.86618 | clear |
| gen2 | 76.957169 | 74.10737 | 2.849799 | 0.633181 | clear |
Known limitations
Subgroup metadataabsent on 11 of 11 corpora, so the subgroup score falls back to the worst diagnostic class. That absence is a disclosure, not a pass.
Training set cappedyes on at least one task — see the raw JSON for which
Observed in deploymentnothing — see the third panel
Audit chain
Bundles re-derived48 of 48
Ledger100 entries, head 6b7b371ff650a8ed…, chain intact
Re-run it yourself
python src/heldout/audit.py 1f66c7719ab3943c6fcc17 source files, with hashes
| results/cross_site/organ_ct__rf_pixels.json | 38f26c32e987cb828498918b9cd641f3ff519c4cdc1ca72d3d0eb8f710e50089 |
| results/heldout/audit_report.json | df460b6d91a6512dfa1e7dc650637fce2b4dd3a1ec32e03446bf6e11f9c597d2 |
| results/heldout/cycle.json | e012b8978457b51b5c31d18d14d2cc92f100731897523306acae19934f943502 |
| results/heldout/manifests/gen1.seal.json | 91a7adfa29870a0719125cdd48f6dcb149f1f6485ac1cfa104c6e00057ad9b81 |
| results/heldout/manifests/gen2.seal.json | f61470918a09cf39e074ef14fb7ade4ac3a13d36d9385f18cf9b92e4b5329547 |
| results/heldout/manifests/public.seal.json | 1a4fd930d2e781ac7163483990f15d6694e28e507d1b3ca009ff09395e47497c |
| results/records/bloodmnist__rf_pixels.json | 76739a850ac1bb24c60dd2d71f085ebc8687952c990d088826191a4679dc21cf |
| results/records/breastmnist__rf_pixels.json | 9f68ecf6943e2f433882a2dc00714ba0723d2625aa17e575f918655715061ec9 |
| results/records/dermamnist__rf_pixels.json | 0f26064a0c7124531762f81a7bd76a87d3ce29944e079fa41c819fb9807dc0c4 |
| results/records/octmnist__rf_pixels.json | 079e28937fd8338438ab8d3fc585bcf4b1e6afed70890756e090a979f29dc71f |
| results/records/organamnist__rf_pixels.json | 4dd8dbaa4f0a51f0c227bdd032c7599976bbf8eca3972e4c2f71d22cb5e5a33a |
| results/records/organcmnist__rf_pixels.json | 44a2e26953aa481da6cccd08d770ef0ba647aea8a783d3f9c4caab02929f7e63 |
| results/records/organsmnist__rf_pixels.json | ef862848c2ebaad54ee489296f7b801cdc51cffa04063665aa9101ebf63565e8 |
| results/records/pathmnist__rf_pixels.json | cb5912f0c678cff6b2808c4f30c629d88e16143940f5a02b4ca5bfaa98bea1a1 |
| results/records/pneumoniamnist__rf_pixels.json | 44ccab0b3753b863b63b2ce32551ce7c9162e750428a4012e5571b0dc5646998 |
| results/records/retinamnist__rf_pixels.json | 2486e33a75838d9e8341f6a19660d7e739200531bc90b8cfec7bccc602d2f09b |
| results/records/tissuemnist__rf_pixels.json | ca169123c48759ae9540a809c778b653c38444f37b175d83da08ffb064986957 |
Declared by vendor
Five fields a vendor asserts about its own product — intended use, forbidden use, training cutoff, regulatory clearances, and who is personally attesting. NakedSignal never fills these in on a vendor's behalf.
No vendor declaration. This is an open reference baseline implemented by NakedSignal, not a commercial product: there is no vendor to assert an intended use or to sign an attestation, so the declared block is null rather than filled in on the vendor's behalf.
Observed in deployment
Production history and drift — how the model has actually behaved since it was deployed, on real traffic.
Not available: this requires continuous monitoring. Production history and drift are observed fields, and nothing is deployed, so there is nothing to observe. The object is present and null on purpose. The shape is the roadmap, not a claim.
Raw JSON — the whole signed document, 83,633 characters
{
"canonicalisation": {
"float_rounding": "every float is rounded once at emit to 6 decimal places (Python round(), banker's rounding); integers are left exact",
"form": "RFC 8785 (JCS)-compatible: UTF-8, object keys sorted by code point, no insignificant whitespace, ECMAScript number formatting",
"signed_over": "the whole document with the `signature` member removed, canonicalised as above and encoded UTF-8"
},
"computed": {
"audit": {
"audit_command": "python src/heldout/audit.py 1f66c7719ab3943c6fcc",
"bundles": 48,
"ledger_entries": 100,
"ledger_head": "6b7b371ff650a8edaf477490f2a5f9b043433f3ba9941989ee1962ed8f1b1d88",
"ledger_intact": true,
"rederived": 48
},
"cross_site": {
"code_fingerprint": "b444c02112cc",
"device": "mps",
"family": "organ_ct",
"matrix": {
"organamnist": {
"organamnist": {
"balanced_accuracy": 0.766396,
"chance_corrected": 0.743035,
"ece": 0.213978
},
"organcmnist": {
"balanced_accuracy": 0.552045,
"chance_corrected": 0.507249,
"ece": 0.102305,
"retention": 0.682672
},
"organsmnist": {
"balanced_accuracy": 0.33584,
"chance_corrected": 0.269424,
"ece": 0.042258,
"retention": 0.3626
}
},
"organcmnist": {
"organamnist": {
"balanced_accuracy": 0.535576,
"chance_corrected": 0.489134,
"ece": 0.09639,
"retention": 0.625202
},
"organcmnist": {
"balanced_accuracy": 0.802146,
"chance_corrected": 0.782361,
"ece": 0.198368
},
"organsmnist": {
"balanced_accuracy": 0.429023,
"chance_corrected": 0.371925,
"ece": 0.025896,
"retention": 0.475388
}
},
"organsmnist": {
"organamnist": {
"balanced_accuracy": 0.312728,
"chance_corrected": 0.244,
"ece": 0.052323,
"retention": 0.422942
},
"organcmnist": {
"balanced_accuracy": 0.511351,
"chance_corrected": 0.462486,
"ece": 0.140016,
"retention": 0.801658
},
"organsmnist": {
"balanced_accuracy": 0.615374,
"chance_corrected": 0.576912,
"ece": 0.17518
}
}
},
"mean_retention": 0.561744,
"note": "retention is absolute retained skill (chance-corrected), per spec 2.2.1; the diagonal is same-site, every off-diagonal cell is train-on-row, test-on-column",
"score": 56.174356,
"site_labels": {
"organamnist": "axial",
"organcmnist": "coronal",
"organsmnist": "sagittal"
},
"sites": [
"organamnist",
"organcmnist",
"organsmnist"
],
"spec_version": "MedEval-1 v0.1",
"worst_retention": 0.3626
},
"evaluations": [
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 62.852521
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.833967,
"auc": 0.976518,
"balanced_accuracy": 0.789112,
"ci95": [
0.773516,
0.801845
],
"n_test": 3421,
"score": 75.898528
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.776124,
"mean_agreement": 0.78,
"mean_kappa": 0.740822,
"mean_retention": 0.79754,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9692,
0.955,
0.8825
],
"mean_kappa": 0.922598,
"mean_retention": 0.960027,
"retention_by_severity": [
0.9872,
0.998,
0.8948
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8883,
0.7817,
0.585
],
"mean_kappa": 0.698529,
"mean_retention": 0.704772,
"retention_by_severity": [
0.9164,
0.7408,
0.4571
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.7542,
0.4742,
0.2375
],
"mean_kappa": 0.411042,
"mean_retention": 0.527577,
"retention_by_severity": [
0.8169,
0.5301,
0.2357
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9383,
0.9025,
0.8683
],
"mean_kappa": 0.883635,
"mean_retention": 0.93695,
"retention_by_severity": [
0.9947,
0.9345,
0.8816
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9283,
0.8375,
0.6975
],
"mean_kappa": 0.788305,
"mean_retention": 0.858374,
"retention_by_severity": [
0.9797,
0.8723,
0.7232
],
"scored": true
}
},
"score": 79.754018,
"worst_retention": 0.235697
},
"calibration": {
"accuracy": 0.833967,
"brier": 0.295686,
"ece": 0.194813,
"mean_confidence": 0.639244,
"overconfidence": -0.194722,
"score": 2.59354
},
"corruption": {
"clean_reference_cc": 0.776124,
"mean_agreement": 0.74127,
"mean_kappa": 0.687607,
"mean_retention": 0.706461,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.5542,
0.1858,
0.1317
],
"mean_kappa": 0.157262,
"mean_retention": 0.16699,
"retention_by_severity": [
0.4472,
0.0505,
0.0033
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.5492,
0.4442,
0.3717
],
"mean_kappa": 0.327161,
"mean_retention": 0.301469,
"retention_by_severity": [
0.4187,
0.2816,
0.2041
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9542,
0.8875,
0.7825
],
"mean_kappa": 0.848618,
"mean_retention": 0.881341,
"retention_by_severity": [
1,
0.9252,
0.7189
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9567,
0.8975,
0.7067
],
"mean_kappa": 0.82352,
"mean_retention": 0.845524,
"retention_by_severity": [
0.9854,
0.8958,
0.6553
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9458,
0.8925,
0.9025
],
"mean_kappa": 0.895958,
"mean_retention": 0.930667,
"retention_by_severity": [
0.9813,
0.8797,
0.931
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.98,
0.9667,
0.935
],
"mean_kappa": 0.952799,
"mean_retention": 0.987831,
"retention_by_severity": [
0.9972,
0.9958,
0.9705
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9242,
0.8683,
0.73
],
"mean_kappa": 0.807933,
"mean_retention": 0.831408,
"retention_by_severity": [
0.943,
0.88,
0.6713
],
"scored": true
}
},
"score": 70.646134,
"worst_retention": 0.00335
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.758985,
"mean_retention": 0.805205,
"per_budget": {
"n100": {
"chance_corrected": 0.743436,
"n_labels": 800,
"retention": 0.979513
},
"n20": {
"chance_corrected": 0.614774,
"n_labels": 160,
"retention": 0.809995
},
"n5": {
"chance_corrected": 0.475206,
"n_labels": 40,
"retention": 0.626108
}
},
"score": 80.520515
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 42.454728,
"worst_class_recall": 0.496479
},
"uncertainty": {
"full_accuracy": 0.833967,
"risk_coverage_auc": 0.959427,
"score": 95.363039,
"selective_acc_at_50": 0.988889,
"selective_acc_at_80": 0.911947
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 11959,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Microscopy",
"n_classes": 8,
"n_test": 3421,
"runtime_s": 7.7,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "bloodmnist",
"task_name": "Peripheral blood cells (8-class)",
"timestamp": "2026-08-01T19:18:34+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 57.648922
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.820513,
"auc": 0.854741,
"balanced_accuracy": 0.696742,
"ci95": [
0.627716,
0.767925
],
"n_test": 156,
"score": 39.348371
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.393484,
"mean_agreement": 0.911538,
"mean_kappa": 0.670223,
"mean_retention": 0.807006,
"n_cases": 156,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9744,
0.9615,
0.9615
],
"mean_kappa": 0.856798,
"mean_retention": 0.876858,
"retention_by_severity": [
0.9172,
0.879,
0.8344
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.9615,
0.9423,
0.8974
],
"mean_kappa": 0.722501,
"mean_retention": 0.703822,
"retention_by_severity": [
0.7516,
0.7293,
0.6306
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9103,
0.859,
0.8333
],
"mean_kappa": 0.432697,
"mean_retention": 0.726115,
"retention_by_severity": [
0.9554,
0.5924,
0.6306
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9872,
0.9808,
0.9872
],
"mean_kappa": 0.935271,
"mean_retention": 0.858811,
"retention_by_severity": [
0.879,
0.8185,
0.879
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9295,
0.7885,
0.6987
],
"mean_kappa": 0.403847,
"mean_retention": 0.869427,
"retention_by_severity": [
1,
1,
0.6083
],
"scored": true
}
},
"score": 80.700637,
"worst_retention": 0.592357
},
"calibration": {
"accuracy": 0.820513,
"brier": 0.26474,
"ece": 0.053718,
"mean_confidence": 0.770256,
"overconfidence": -0.050256,
"score": 73.141026
},
"corruption": {
"clean_reference_cc": 0.393484,
"mean_agreement": 0.924908,
"mean_kappa": 0.624834,
"mean_retention": 0.679557,
"n_cases": 156,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9103,
0.859,
0.859
],
"mean_kappa": 0.287779,
"mean_retention": 0.343949,
"retention_by_severity": [
0.6369,
0.3949,
0
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8974,
0.8526,
0.859
],
"mean_kappa": 0.136958,
"mean_retention": 0.181529,
"retention_by_severity": [
0.4841,
0.0605,
0
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9808,
0.9808,
0.9679
],
"mean_kappa": 0.898933,
"mean_retention": 0.803609,
"retention_by_severity": [
0.8185,
0.8185,
0.7739
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9679,
0.9423,
0.8269
],
"mean_kappa": 0.683167,
"mean_retention": 0.863057,
"retention_by_severity": [
0.9395,
0.8567,
0.793
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9615,
0.9551,
0.9615
],
"mean_kappa": 0.822961,
"mean_retention": 0.803609,
"retention_by_severity": [
1,
0.5764,
0.8344
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9872,
0.9615,
0.9487
],
"mean_kappa": 0.854011,
"mean_retention": 0.919321,
"retention_by_severity": [
1,
0.879,
0.879
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9679,
0.9359,
0.8397
],
"mean_kappa": 0.69003,
"mean_retention": 0.841826,
"retention_by_severity": [
0.8567,
0.6688,
1
],
"scored": true
}
},
"score": 67.955717,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.393484,
"mean_retention": 0.713376,
"per_budget": {
"n100": {
"chance_corrected": 0.570175,
"n_labels": 200,
"retention": 1
},
"n20": {
"chance_corrected": 0.055138,
"n_labels": 40,
"retention": 0.140127
},
"n5": {
"chance_corrected": 0.393484,
"n_labels": 10,
"retention": 1
}
},
"score": 71.33758
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0.428571
},
"uncertainty": {
"full_accuracy": 0.820513,
"risk_coverage_auc": 0.920265,
"score": 84.053008,
"selective_acc_at_50": 0.923077,
"selective_acc_at_80": 0.88
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 546,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Ultrasound",
"n_classes": 2,
"n_test": 156,
"runtime_s": 1.2,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "breastmnist",
"task_name": "Breast ultrasound (malignancy)",
"timestamp": "2026-08-01T19:12:55+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 55.708505
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.72419,
"auc": 0.884833,
"balanced_accuracy": 0.273781,
"ci95": [
0.252347,
0.295534
],
"n_test": 2005,
"score": 15.274405
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.164164,
"mean_agreement": 0.911167,
"mean_kappa": 0.625406,
"mean_retention": 0.72348,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9658,
0.95,
0.9225
],
"mean_kappa": 0.794588,
"mean_retention": 0.925746,
"retention_by_severity": [
0.9223,
0.9796,
0.8754
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.9692,
0.9442,
0.9175
],
"mean_kappa": 0.797335,
"mean_retention": 0.973084,
"retention_by_severity": [
1,
1,
0.9193
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8917,
0.8442,
0.8217
],
"mean_kappa": 0.311464,
"mean_retention": 0.256437,
"retention_by_severity": [
0.4792,
0.1986,
0.0915
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9708,
0.9592,
0.9575
],
"mean_kappa": 0.862174,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9108,
0.8592,
0.7833
],
"mean_kappa": 0.361468,
"mean_retention": 0.462132,
"retention_by_severity": [
0.7107,
0.3842,
0.2915
],
"scored": true
}
},
"score": 72.347973,
"worst_retention": 0.091458
},
"calibration": {
"accuracy": 0.72419,
"brier": 0.376641,
"ece": 0.04202,
"mean_confidence": 0.68217,
"overconfidence": -0.04202,
"score": 78.990025
},
"corruption": {
"clean_reference_cc": 0.164164,
"mean_agreement": 0.87246,
"mean_kappa": 0.586228,
"mean_retention": 0.802805,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.87,
0.84,
0.855
],
"mean_kappa": 0.218058,
"mean_retention": 0.188538,
"retention_by_severity": [
0.4456,
0.1132,
0.0067
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8967,
0.7775,
0.6083
],
"mean_kappa": 0.468439,
"mean_retention": 0.985968,
"retention_by_severity": [
1,
1,
0.9579
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9783,
0.965,
0.9375
],
"mean_kappa": 0.858416,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9467,
0.9017,
0.7017
],
"mean_kappa": 0.530655,
"mean_retention": 0.815515,
"retention_by_severity": [
1,
0.8653,
0.5812
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9667,
0.9033,
0.9633
],
"mean_kappa": 0.806943,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9825,
0.9692,
0.9417
],
"mean_kappa": 0.862569,
"mean_retention": 0.96492,
"retention_by_severity": [
0.9747,
0.9958,
0.9242
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.8975,
0.8308,
0.5883
],
"mean_kappa": 0.358518,
"mean_retention": 0.664693,
"retention_by_severity": [
0.8019,
0.7219,
0.4703
],
"scored": true
}
},
"score": 80.280482,
"worst_retention": 0.006738
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.152744,
"mean_retention": 1,
"per_budget": {
"n100": {
"chance_corrected": 0.385633,
"n_labels": 679,
"retention": 1
},
"n20": {
"chance_corrected": 0.253442,
"n_labels": 140,
"retention": 1
},
"n5": {
"chance_corrected": 0.213682,
"n_labels": 35,
"retention": 1
}
},
"score": 100
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0
},
"uncertainty": {
"full_accuracy": 0.72419,
"risk_coverage_auc": 0.899384,
"score": 88.261491,
"selective_acc_at_50": 0.924152,
"selective_acc_at_80": 0.807357
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 7007,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Dermatoscopy",
"n_classes": 7,
"n_test": 2005,
"runtime_s": 6.2,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "dermamnist",
"task_name": "Dermatoscopy (7-class skin lesion)",
"timestamp": "2026-08-01T19:17:03+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 44.288353
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.498,
"auc": 0.811123,
"balanced_accuracy": 0.498,
"ci95": [
0.482339,
0.516256
],
"n_test": 1000,
"score": 33.066667
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.330667,
"mean_agreement": 0.642267,
"mean_kappa": 0.415281,
"mean_retention": 0.608065,
"n_cases": 1000,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.863,
0.577,
0.405
],
"mean_kappa": 0.364328,
"mean_retention": 0.510753,
"retention_by_severity": [
0.9597,
0.4798,
0.0927
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.852,
0.698,
0.476
],
"mean_kappa": 0.450179,
"mean_retention": 0.715054,
"retention_by_severity": [
0.9637,
0.8105,
0.371
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.771,
0.429,
0.292
],
"mean_kappa": 0.216336,
"mean_retention": 0.413978,
"retention_by_severity": [
0.7581,
0.3065,
0.1774
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.911,
0.869,
0.851
],
"mean_kappa": 0.772154,
"mean_retention": 0.86828,
"retention_by_severity": [
0.9315,
0.8629,
0.8105
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.781,
0.496,
0.363
],
"mean_kappa": 0.273408,
"mean_retention": 0.532258,
"retention_by_severity": [
0.8548,
0.4315,
0.3105
],
"scored": true
}
},
"score": 60.806452,
"worst_retention": 0.092742
},
"calibration": {
"accuracy": 0.498,
"brier": 0.663535,
"ece": 0.064564,
"mean_confidence": 0.558984,
"overconfidence": 0.060984,
"score": 67.717917
},
"corruption": {
"clean_reference_cc": 0.330667,
"mean_agreement": 0.645143,
"mean_kappa": 0.411616,
"mean_retention": 0.53437,
"n_cases": 1000,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.38,
0.37,
0.37
],
"mean_kappa": 0.006804,
"mean_retention": 0.032258,
"retention_by_severity": [
0.0968,
0,
0
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.741,
0.541,
0.407
],
"mean_kappa": 0.271473,
"mean_retention": 0.415323,
"retention_by_severity": [
0.7742,
0.3831,
0.0887
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.922,
0.874,
0.79
],
"mean_kappa": 0.744806,
"mean_retention": 0.813172,
"retention_by_severity": [
0.8871,
0.8548,
0.6976
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.629,
0.43,
0.373
],
"mean_kappa": 0.162967,
"mean_retention": 0.317204,
"retention_by_severity": [
0.6452,
0.2097,
0.0968
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.902,
0.841,
0.792
],
"mean_kappa": 0.718661,
"mean_retention": 0.897849,
"retention_by_severity": [
0.9032,
0.9798,
0.8105
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.934,
0.824,
0.719
],
"mean_kappa": 0.686543,
"mean_retention": 0.848118,
"retention_by_severity": [
0.9798,
0.8871,
0.6774
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.783,
0.537,
0.389
],
"mean_kappa": 0.290056,
"mean_retention": 0.416667,
"retention_by_severity": [
0.8065,
0.3669,
0.0766
],
"scored": true
}
},
"score": 53.43702,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.330667,
"mean_retention": 0.399194,
"per_budget": {
"n100": {
"chance_corrected": 0.190667,
"n_labels": 400,
"retention": 0.576613
},
"n20": {
"chance_corrected": 0.205333,
"n_labels": 80,
"retention": 0.620968
},
"n5": {
"chance_corrected": 0,
"n_labels": 20,
"retention": 0
}
},
"score": 39.919355
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0
},
"uncertainty": {
"full_accuracy": 0.498,
"risk_coverage_auc": 0.599022,
"score": 46.536214,
"selective_acc_at_50": 0.608,
"selective_acc_at_80": 0.535
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 97477,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "OCT",
"n_classes": 4,
"n_test": 1000,
"runtime_s": 4.4,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "octmnist",
"task_name": "Retinal OCT (4-class)",
"timestamp": "2026-08-01T19:35:25+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 65.211789
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.778153,
"auc": 0.96717,
"balanced_accuracy": 0.766396,
"ci95": [
0.760192,
0.771997
],
"n_test": 17778,
"score": 74.30351
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.722394,
"mean_agreement": 0.803167,
"mean_kappa": 0.778998,
"mean_retention": 0.890453,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9425,
0.885,
0.8367
],
"mean_kappa": 0.873962,
"mean_retention": 0.953069,
"retention_by_severity": [
0.9789,
0.9631,
0.9172
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8575,
0.7358,
0.57
],
"mean_kappa": 0.688527,
"mean_retention": 0.816766,
"retention_by_severity": [
0.9391,
0.8401,
0.6711
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8533,
0.695,
0.5958
],
"mean_kappa": 0.679652,
"mean_retention": 0.803889,
"retention_by_severity": [
0.9774,
0.7894,
0.6448
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.8842,
0.8558,
0.8425
],
"mean_kappa": 0.843377,
"mean_retention": 0.957168,
"retention_by_severity": [
0.9649,
0.955,
0.9516
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.925,
0.8533,
0.715
],
"mean_kappa": 0.809475,
"mean_retention": 0.921373,
"retention_by_severity": [
0.9825,
0.9481,
0.8335
],
"scored": true
}
},
"score": 89.045305,
"worst_retention": 0.644818
},
"calibration": {
"accuracy": 0.778153,
"brier": 0.3771,
"ece": 0.213978,
"mean_confidence": 0.564175,
"overconfidence": -0.213978,
"score": 0
},
"corruption": {
"clean_reference_cc": 0.722394,
"mean_agreement": 0.683254,
"mean_kappa": 0.649576,
"mean_retention": 0.72145,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.3817,
0.2175,
0.095
],
"mean_kappa": 0.156518,
"mean_retention": 0.235527,
"retention_by_severity": [
0.3692,
0.2323,
0.1051
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.4333,
0.26,
0.165
],
"mean_kappa": 0.216635,
"mean_retention": 0.246518,
"retention_by_severity": [
0.3961,
0.2253,
0.1182
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.8683,
0.8183,
0.7292
],
"mean_kappa": 0.781488,
"mean_retention": 0.891971,
"retention_by_severity": [
0.9611,
0.9163,
0.7984
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9233,
0.87,
0.7617
],
"mean_kappa": 0.832291,
"mean_retention": 0.894337,
"retention_by_severity": [
0.9654,
0.927,
0.7907
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9033,
0.5583,
0.8867
],
"mean_kappa": 0.756948,
"mean_retention": 0.875984,
"retention_by_severity": [
0.9932,
0.6582,
0.9766
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9767,
0.96,
0.9267
],
"mean_kappa": 0.948641,
"mean_retention": 0.988776,
"retention_by_severity": [
1,
0.9834,
0.983
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9158,
0.8883,
0.8092
],
"mean_kappa": 0.854515,
"mean_retention": 0.91704,
"retention_by_severity": [
0.9624,
0.9456,
0.8431
],
"scored": true
}
},
"score": 72.145035,
"worst_retention": 0.105123
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.743035,
"mean_retention": 0.819373,
"per_budget": {
"n100": {
"chance_corrected": 0.697315,
"n_labels": 1100,
"retention": 0.938469
},
"n20": {
"chance_corrected": 0.601599,
"n_labels": 220,
"retention": 0.809651
},
"n5": {
"chance_corrected": 0.527554,
"n_labels": 55,
"retention": 0.709999
}
},
"score": 81.937316
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 52.473282,
"worst_class_recall": 0.567939
},
"uncertainty": {
"full_accuracy": 0.778153,
"risk_coverage_auc": 0.936618,
"score": 93.028,
"selective_acc_at_50": 0.976488,
"selective_acc_at_80": 0.867951
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 34561,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 17778,
"runtime_s": 5.2,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organamnist",
"task_name": "Abdominal CT organs — axial (A)",
"timestamp": "2026-08-01T19:32:34+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 66.264852
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.828627,
"auc": 0.975913,
"balanced_accuracy": 0.802146,
"ci95": [
0.794516,
0.809546
],
"n_test": 8216,
"score": 78.236078
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.770639,
"mean_agreement": 0.83,
"mean_kappa": 0.807671,
"mean_retention": 0.887762,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9608,
0.9083,
0.8467
],
"mean_kappa": 0.892391,
"mean_retention": 0.933229,
"retention_by_severity": [
0.9907,
0.9404,
0.8685
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.87,
0.7475,
0.5783
],
"mean_kappa": 0.699085,
"mean_retention": 0.796424,
"retention_by_severity": [
0.925,
0.8153,
0.6489
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8792,
0.7492,
0.6367
],
"mean_kappa": 0.722589,
"mean_retention": 0.815008,
"retention_by_severity": [
0.9346,
0.8141,
0.6963
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9108,
0.8958,
0.8617
],
"mean_kappa": 0.874587,
"mean_retention": 0.944266,
"retention_by_severity": [
0.9571,
0.9447,
0.9309
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9433,
0.885,
0.7767
],
"mean_kappa": 0.849702,
"mean_retention": 0.949883,
"retention_by_severity": [
0.9984,
0.9709,
0.8804
],
"scored": true
}
},
"score": 88.776197,
"worst_retention": 0.648943
},
"calibration": {
"accuracy": 0.828627,
"brier": 0.304929,
"ece": 0.198368,
"mean_confidence": 0.630259,
"overconfidence": -0.198368,
"score": 0.815786
},
"corruption": {
"clean_reference_cc": 0.770639,
"mean_agreement": 0.691984,
"mean_kappa": 0.6548,
"mean_retention": 0.704461,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.4317,
0.2133,
0.0942
],
"mean_kappa": 0.167994,
"mean_retention": 0.207053,
"retention_by_severity": [
0.4153,
0.1738,
0.0321
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.41,
0.2675,
0.1858
],
"mean_kappa": 0.212078,
"mean_retention": 0.247832,
"retention_by_severity": [
0.3981,
0.2295,
0.1159
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.905,
0.8367,
0.6883
],
"mean_kappa": 0.78622,
"mean_retention": 0.872642,
"retention_by_severity": [
0.9632,
0.9009,
0.7539
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9442,
0.8942,
0.8108
],
"mean_kappa": 0.866373,
"mean_retention": 0.913989,
"retention_by_severity": [
0.9759,
0.9327,
0.8333
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9342,
0.4467,
0.9158
],
"mean_kappa": 0.720663,
"mean_retention": 0.778838,
"retention_by_severity": [
0.9852,
0.3837,
0.9676
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9792,
0.9767,
0.9342
],
"mean_kappa": 0.958269,
"mean_retention": 0.995537,
"retention_by_severity": [
1,
0.9919,
0.9947
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9267,
0.8933,
0.8433
],
"mean_kappa": 0.872006,
"mean_retention": 0.915338,
"retention_by_severity": [
0.956,
0.9242,
0.8658
],
"scored": true
}
},
"score": 70.446135,
"worst_retention": 0.032079
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.782361,
"mean_retention": 0.775914,
"per_budget": {
"n100": {
"chance_corrected": 0.738553,
"n_labels": 1100,
"retention": 0.944006
},
"n20": {
"chance_corrected": 0.619821,
"n_labels": 220,
"retention": 0.792244
},
"n5": {
"chance_corrected": 0.46276,
"n_labels": 55,
"retention": 0.591492
}
},
"score": 77.591393
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 57.047619,
"worst_class_recall": 0.609524
},
"uncertainty": {
"full_accuracy": 0.828627,
"risk_coverage_auc": 0.962044,
"score": 95.824864,
"selective_acc_at_50": 0.99148,
"selective_acc_at_80": 0.922562
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 12975,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 8216,
"runtime_s": 8,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organcmnist",
"task_name": "Abdominal CT organs — coronal (C)",
"timestamp": "2026-08-01T19:27:36+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 57.910437
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.680299,
"auc": 0.948603,
"balanced_accuracy": 0.615374,
"ci95": [
0.604579,
0.624876
],
"n_test": 8827,
"score": 57.691193
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.571056,
"mean_agreement": 0.779333,
"mean_kappa": 0.736463,
"mean_retention": 0.874819,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9433,
0.8792,
0.82
],
"mean_kappa": 0.854646,
"mean_retention": 0.962883,
"retention_by_severity": [
1,
0.9912,
0.8974
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8542,
0.7683,
0.6267
],
"mean_kappa": 0.696278,
"mean_retention": 0.801995,
"retention_by_severity": [
0.9307,
0.8176,
0.6577
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8342,
0.61,
0.4617
],
"mean_kappa": 0.573225,
"mean_retention": 0.699221,
"retention_by_severity": [
0.9361,
0.678,
0.4835
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.8783,
0.8383,
0.8167
],
"mean_kappa": 0.811874,
"mean_retention": 0.947073,
"retention_by_severity": [
0.9693,
0.9549,
0.917
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9158,
0.8017,
0.6417
],
"mean_kappa": 0.74629,
"mean_retention": 0.962922,
"retention_by_severity": [
1,
1,
0.8888
],
"scored": true
}
},
"score": 87.481896,
"worst_retention": 0.483538
},
"calibration": {
"accuracy": 0.680299,
"brier": 0.476318,
"ece": 0.17518,
"mean_confidence": 0.505119,
"overconfidence": -0.17518,
"score": 12.409935
},
"corruption": {
"clean_reference_cc": 0.571056,
"mean_agreement": 0.679722,
"mean_kappa": 0.610639,
"mean_retention": 0.69077,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.5108,
0.305,
0.165
],
"mean_kappa": 0.175767,
"mean_retention": 0.210598,
"retention_by_severity": [
0.4027,
0.1743,
0.0548
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.4308,
0.3,
0.2283
],
"mean_kappa": 0.1842,
"mean_retention": 0.231625,
"retention_by_severity": [
0.374,
0.1924,
0.1284
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.8775,
0.8058,
0.6517
],
"mean_kappa": 0.732388,
"mean_retention": 0.877227,
"retention_by_severity": [
0.9828,
0.917,
0.7319
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.915,
0.8667,
0.71
],
"mean_kappa": 0.791338,
"mean_retention": 0.879947,
"retention_by_severity": [
0.982,
0.9527,
0.7052
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.8967,
0.4208,
0.86
],
"mean_kappa": 0.665136,
"mean_retention": 0.775724,
"retention_by_severity": [
0.9888,
0.4189,
0.9195
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9675,
0.9533,
0.925
],
"mean_kappa": 0.937586,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.8908,
0.86,
0.7333
],
"mean_kappa": 0.788058,
"mean_retention": 0.86027,
"retention_by_severity": [
0.9529,
0.8988,
0.729
],
"scored": true
}
},
"score": 69.077012,
"worst_retention": 0.054758
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.576912,
"mean_retention": 0.73517,
"per_budget": {
"n100": {
"chance_corrected": 0.540256,
"n_labels": 1100,
"retention": 0.936462
},
"n20": {
"chance_corrected": 0.432502,
"n_labels": 220,
"retention": 0.749684
},
"n5": {
"chance_corrected": 0.299626,
"n_labels": 55,
"retention": 0.519362
}
},
"score": 73.516957
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 26.190476,
"worst_class_recall": 0.329004
},
"uncertainty": {
"full_accuracy": 0.680299,
"risk_coverage_auc": 0.853059,
"score": 83.836529,
"selective_acc_at_50": 0.863842,
"selective_acc_at_80": 0.760125
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 13932,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 8827,
"runtime_s": 5.8,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organsmnist",
"task_name": "Abdominal CT organs — sagittal (S)",
"timestamp": "2026-08-01T19:30:19+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 50.748238
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.622423,
"auc": 0.881817,
"balanced_accuracy": 0.532245,
"ci95": [
0.521905,
0.54316
],
"n_test": 7180,
"score": 47.37752
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.482448,
"mean_agreement": 0.685444,
"mean_kappa": 0.626608,
"mean_retention": 0.81003,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.8775,
0.8358,
0.77
],
"mean_kappa": 0.791153,
"mean_retention": 0.894038,
"retention_by_severity": [
0.9837,
0.9165,
0.7819
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8717,
0.8567,
0.8308
],
"mean_kappa": 0.822883,
"mean_retention": 0.990839,
"retention_by_severity": [
1,
1,
0.9725
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.6467,
0.405,
0.3017
],
"mean_kappa": 0.337368,
"mean_retention": 0.646232,
"retention_by_severity": [
0.8192,
0.635,
0.4845
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.865,
0.86,
0.835
],
"mean_kappa": 0.823892,
"mean_retention": 0.987586,
"retention_by_severity": [
1,
0.9979,
0.9649
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.6442,
0.4125,
0.2692
],
"mean_kappa": 0.357745,
"mean_retention": 0.531454,
"retention_by_severity": [
0.8113,
0.5044,
0.2786
],
"scored": true
}
},
"score": 81.002982,
"worst_retention": 0.278615
},
"calibration": {
"accuracy": 0.622423,
"brier": 0.592572,
"ece": 0.213249,
"mean_confidence": 0.409175,
"overconfidence": -0.213249,
"score": 0
},
"corruption": {
"clean_reference_cc": 0.482448,
"mean_agreement": 0.687302,
"mean_kappa": 0.63301,
"mean_retention": 0.753054,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.3575,
0.1383,
0.1125
],
"mean_kappa": 0.098365,
"mean_retention": 0.221844,
"retention_by_severity": [
0.5631,
0.1025,
0
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8575,
0.74,
0.6358
],
"mean_kappa": 0.699788,
"mean_retention": 0.914823,
"retention_by_severity": [
1,
0.9306,
0.8139
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.88,
0.8283,
0.7842
],
"mean_kappa": 0.797676,
"mean_retention": 0.966823,
"retention_by_severity": [
1,
0.9659,
0.9346
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.8367,
0.735,
0.5833
],
"mean_kappa": 0.660172,
"mean_retention": 0.718793,
"retention_by_severity": [
0.8422,
0.7391,
0.575
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.8875,
0.7783,
0.8567
],
"mean_kappa": 0.808816,
"mean_retention": 0.959131,
"retention_by_severity": [
0.9767,
0.9177,
0.983
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9092,
0.8717,
0.7208
],
"mean_kappa": 0.799769,
"mean_retention": 0.87627,
"retention_by_severity": [
0.9417,
0.9769,
0.7102
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.7633,
0.6642,
0.4925
],
"mean_kappa": 0.566484,
"mean_retention": 0.613694,
"retention_by_severity": [
0.7554,
0.633,
0.4526
],
"scored": true
}
},
"score": 75.305394,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.473775,
"mean_retention": 0.757015,
"per_budget": {
"n100": {
"chance_corrected": 0.419653,
"n_labels": 900,
"retention": 0.885764
},
"n20": {
"chance_corrected": 0.294474,
"n_labels": 180,
"retention": 0.621549
},
"n5": {
"chance_corrected": 0.361837,
"n_labels": 45,
"retention": 0.76373
}
},
"score": 75.701452
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 5.566802,
"worst_class_recall": 0.160594
},
"uncertainty": {
"full_accuracy": 0.622423,
"risk_coverage_auc": 0.789657,
"score": 76.336375,
"selective_acc_at_50": 0.793593,
"selective_acc_at_80": 0.694638
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 89996,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "Histopathology",
"n_classes": 9,
"n_test": 7180,
"runtime_s": 8.4,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "pathmnist",
"task_name": "Colorectal pathology (9-class tissue)",
"timestamp": "2026-08-01T19:41:39+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 69.138178
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.857372,
"auc": 0.947726,
"balanced_accuracy": 0.819231,
"ci95": [
0.794788,
0.849786
],
"n_test": 624,
"score": 63.846154
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.638462,
"mean_agreement": 0.902564,
"mean_kappa": 0.706498,
"mean_retention": 0.75386,
"n_cases": 624,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9824,
0.9599,
0.9423
],
"mean_kappa": 0.897819,
"mean_retention": 0.904507,
"retention_by_severity": [
0.9612,
0.913,
0.8394
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8718,
0.7821,
0.7372
],
"mean_kappa": 0.296911,
"mean_retention": 0.266399,
"retention_by_severity": [
0.5716,
0.2075,
0.0201
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9167,
0.8494,
0.8301
],
"mean_kappa": 0.624832,
"mean_retention": 0.72066,
"retention_by_severity": [
0.9224,
0.6653,
0.5743
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9679,
0.9567,
0.9375
],
"mean_kappa": 0.877845,
"mean_retention": 0.895582,
"retention_by_severity": [
0.9411,
0.8969,
0.8487
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9583,
0.9359,
0.9103
],
"mean_kappa": 0.835084,
"mean_retention": 0.982151,
"retention_by_severity": [
1,
0.992,
0.9545
],
"scored": true
}
},
"score": 75.385988,
"worst_retention": 0.02008
},
"calibration": {
"accuracy": 0.857372,
"brier": 0.211774,
"ece": 0.035,
"mean_confidence": 0.858237,
"overconfidence": 0.000865,
"score": 82.5
},
"corruption": {
"clean_reference_cc": 0.638462,
"mean_agreement": 0.899115,
"mean_kappa": 0.684078,
"mean_retention": 0.716262,
"n_cases": 624,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.8558,
0.7596,
0.7564
],
"mean_kappa": 0.284213,
"mean_retention": 0.261937,
"retention_by_severity": [
0.5395,
0.1459,
0.1004
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.851,
0.7869,
0.7324
],
"mean_kappa": 0.270215,
"mean_retention": 0.237394,
"retention_by_severity": [
0.4846,
0.2276,
0
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9696,
0.9503,
0.8782
],
"mean_kappa": 0.8088,
"mean_retention": 0.809014,
"retention_by_severity": [
0.9505,
0.8648,
0.6118
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9696,
0.9535,
0.9054
],
"mean_kappa": 0.848412,
"mean_retention": 0.91477,
"retention_by_severity": [
0.9398,
0.9612,
0.8434
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9615,
0.9663,
0.9359
],
"mean_kappa": 0.882355,
"mean_retention": 0.950915,
"retention_by_severity": [
0.8715,
1,
0.9813
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9808,
0.976,
0.9679
],
"mean_kappa": 0.935699,
"mean_retention": 0.973672,
"retention_by_severity": [
0.9705,
0.9746,
0.9759
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9599,
0.9199,
0.8446
],
"mean_kappa": 0.758849,
"mean_retention": 0.866131,
"retention_by_severity": [
0.9371,
0.8983,
0.7631
],
"scored": true
}
},
"score": 71.626187,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.638462,
"mean_retention": 0.766622,
"per_budget": {
"n100": {
"chance_corrected": 0.671795,
"n_labels": 200,
"retention": 1
},
"n20": {
"chance_corrected": 0.624786,
"n_labels": 40,
"retention": 0.978581
},
"n5": {
"chance_corrected": 0.205128,
"n_labels": 10,
"retention": 0.321285
}
},
"score": 76.662204
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 33.333333,
"worst_class_recall": 0.666667
},
"uncertainty": {
"full_accuracy": 0.857372,
"risk_coverage_auc": 0.945223,
"score": 89.044613,
"selective_acc_at_50": 0.974359,
"selective_acc_at_80": 0.915832
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 4708,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Chest X-ray",
"n_classes": 2,
"n_test": 624,
"runtime_s": 2.7,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "pneumoniamnist",
"task_name": "Paediatric chest X-ray (pneumonia)",
"timestamp": "2026-08-01T19:16:14+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 47.790494
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.525,
"auc": 0.6908,
"balanced_accuracy": 0.345054,
"ci95": [
0.31081,
0.390562
],
"n_test": 400,
"score": 18.131743
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.181317,
"mean_agreement": 0.634,
"mean_kappa": 0.437845,
"mean_retention": 0.69974,
"n_cases": 400,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.8375,
0.7675,
0.6475
],
"mean_kappa": 0.572945,
"mean_retention": 0.723493,
"retention_by_severity": [
0.8017,
0.6695,
0.6993
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.6025,
0.555,
0.515
],
"mean_kappa": 0.325287,
"mean_retention": 0.781266,
"retention_by_severity": [
0.7268,
0.7225,
0.8945
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.69,
0.505,
0.38
],
"mean_kappa": 0.329807,
"mean_retention": 0.770596,
"retention_by_severity": [
0.9392,
0.8884,
0.4841
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.7825,
0.76,
0.7275
],
"mean_kappa": 0.572811,
"mean_retention": 0.544839,
"retention_by_severity": [
0.6686,
0.5152,
0.4508
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.7425,
0.5725,
0.425
],
"mean_kappa": 0.388378,
"mean_retention": 0.678506,
"retention_by_severity": [
0.6827,
0.8425,
0.5104
],
"scored": true
}
},
"score": 69.973992,
"worst_retention": 0.450773
},
"calibration": {
"accuracy": 0.525,
"brier": 0.603829,
"ece": 0.069538,
"mean_confidence": 0.515563,
"overconfidence": -0.009437,
"score": 65.23125
},
"corruption": {
"clean_reference_cc": 0.181317,
"mean_agreement": 0.659524,
"mean_kappa": 0.45671,
"mean_retention": 0.706123,
"n_cases": 400,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.56,
0.5425,
0.5975
],
"mean_kappa": 0.211014,
"mean_retention": 0.538553,
"retention_by_severity": [
1,
0.3961,
0.2196
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.52,
0.285,
0.2025
],
"mean_kappa": 0.126425,
"mean_retention": 0.274277,
"retention_by_severity": [
0.75,
0.0728,
0
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.79,
0.69,
0.6175
],
"mean_kappa": 0.484493,
"mean_retention": 0.612437,
"retention_by_severity": [
0.6298,
0.4993,
0.7083
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.7875,
0.6725,
0.5625
],
"mean_kappa": 0.455913,
"mean_retention": 0.71357,
"retention_by_severity": [
0.8599,
0.8607,
0.4201
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.77,
0.6775,
0.7175
],
"mean_kappa": 0.557721,
"mean_retention": 0.910406,
"retention_by_severity": [
0.7867,
1,
0.9445
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.8725,
0.85,
0.8175
],
"mean_kappa": 0.747161,
"mean_retention": 0.954752,
"retention_by_severity": [
1,
0.9868,
0.8775
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.8075,
0.8075,
0.7025
],
"mean_kappa": 0.614242,
"mean_retention": 0.938863,
"retention_by_severity": [
0.9894,
1,
0.8272
],
"scored": true
}
},
"score": 70.612274,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.181317,
"mean_retention": 0.642965,
"per_budget": {
"n100": {
"chance_corrected": 0.12909,
"n_labels": 466,
"retention": 0.711955
},
"n20": {
"chance_corrected": 0.164927,
"n_labels": 100,
"retention": 0.909602
},
"n5": {
"chance_corrected": 0.055725,
"n_labels": 25,
"retention": 0.307336
}
},
"score": 64.296455
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0
},
"uncertainty": {
"full_accuracy": 0.525,
"risk_coverage_auc": 0.702317,
"score": 62.789615,
"selective_acc_at_50": 0.705,
"selective_acc_at_80": 0.56875
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 1080,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Fundus",
"n_classes": 5,
"n_test": 400,
"runtime_s": 2.6,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "retinamnist",
"task_name": "Fundus photography (DR grade)",
"timestamp": "2026-08-01T19:13:08+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 49.700267
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.515419,
"auc": 0.802152,
"balanced_accuracy": 0.298041,
"ci95": [
0.295386,
0.301313
],
"n_test": 47280,
"score": 19.776074
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.187344,
"mean_agreement": 0.669833,
"mean_kappa": 0.52893,
"mean_retention": 0.725242,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.8492,
0.7308,
0.6433
],
"mean_kappa": 0.562766,
"mean_retention": 0.756831,
"retention_by_severity": [
0.9055,
0.7601,
0.6049
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8833,
0.8317,
0.7617
],
"mean_kappa": 0.717672,
"mean_retention": 0.875945,
"retention_by_severity": [
0.9521,
0.8655,
0.8103
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.5542,
0.4108,
0.3308
],
"mean_kappa": 0.246169,
"mean_retention": 0.45317,
"retention_by_severity": [
0.7625,
0.3857,
0.2113
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9208,
0.8967,
0.8783
],
"mean_kappa": 0.835149,
"mean_retention": 0.927454,
"retention_by_severity": [
0.9574,
0.9259,
0.8991
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.5567,
0.435,
0.3642
],
"mean_kappa": 0.282895,
"mean_retention": 0.612812,
"retention_by_severity": [
0.7562,
0.6286,
0.4536
],
"scored": true
}
},
"score": 72.524243,
"worst_retention": 0.211294
},
"calibration": {
"accuracy": 0.515419,
"brier": 0.632086,
"ece": 0.076762,
"mean_confidence": 0.440937,
"overconfidence": -0.074482,
"score": 61.618972
},
"corruption": {
"clean_reference_cc": 0.187344,
"mean_agreement": 0.742738,
"mean_kappa": 0.571902,
"mean_retention": 0.719298,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.6233,
0.5783,
0.475
],
"mean_kappa": 0.191401,
"mean_retention": 0.320239,
"retention_by_severity": [
0.3982,
0.3506,
0.2119
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.805,
0.7433,
0.6967
],
"mean_kappa": 0.558649,
"mean_retention": 0.465338,
"retention_by_severity": [
0.6324,
0.41,
0.3536
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9333,
0.885,
0.8283
],
"mean_kappa": 0.805215,
"mean_retention": 0.849713,
"retention_by_severity": [
0.9678,
0.9439,
0.6374
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.7308,
0.6583,
0.5117
],
"mean_kappa": 0.410407,
"mean_retention": 0.69662,
"retention_by_severity": [
0.8471,
0.6285,
0.6143
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9142,
0.8308,
0.8667
],
"mean_kappa": 0.796975,
"mean_retention": 0.964618,
"retention_by_severity": [
1,
0.9171,
0.9767
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.8917,
0.8258,
0.6217
],
"mean_kappa": 0.665893,
"mean_retention": 0.879524,
"retention_by_severity": [
0.9932,
0.8899,
0.7555
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.8275,
0.7308,
0.6192
],
"mean_kappa": 0.574776,
"mean_retention": 0.859033,
"retention_by_severity": [
0.9655,
0.83,
0.7815
],
"scored": true
}
},
"score": 71.929792,
"worst_retention": 0.211864
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.197761,
"mean_retention": 0.786478,
"per_budget": {
"n100": {
"chance_corrected": 0.218679,
"n_labels": 800,
"retention": 1
},
"n20": {
"chance_corrected": 0.1673,
"n_labels": 160,
"retention": 0.845971
},
"n5": {
"chance_corrected": 0.101543,
"n_labels": 40,
"retention": 0.513463
}
},
"score": 78.647805
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0
},
"uncertainty": {
"full_accuracy": 0.515419,
"risk_coverage_auc": 0.681661,
"score": 63.618374,
"selective_acc_at_50": 0.680541,
"selective_acc_at_80": 0.580875
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 165466,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "Microscopy",
"n_classes": 8,
"n_test": 47280,
"runtime_s": 5.6,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "tissuemnist",
"task_name": "Kidney cortex cells (8-class)",
"timestamp": "2026-08-01T19:37:55+00:00"
}
],
"heldout": {
"entries": [
{
"direction": "no gap beyond sampling error",
"equating_offset": 0,
"gap": 0.58339,
"gap_se": 2.230508,
"generation": "gen1",
"merkle_root": "61c508cbafed177bb1553f05aa2cc80d4157adc1bc1ff611d88581aab9ca61c5",
"model": "rf_pixels",
"score_private_equated": 75.045419,
"score_private_raw": 75.045419,
"score_public": 75.628809,
"set_hash": "641ab45baeaf445a1057c060eb9b35205bd55fcde4e629b2b88cfa38fba30047",
"submitter": "Submitter D",
"verdict": "clear",
"version_tag": "v1",
"z": 0.26155,
"z_crit": 3
},
{
"direction": "private-set leak",
"equating_offset": 0,
"gap": -23.231033,
"gap_se": 1.537004,
"generation": "gen1",
"merkle_root": "61c508cbafed177bb1553f05aa2cc80d4157adc1bc1ff611d88581aab9ca61c5",
"model": "rf_pixels",
"score_private_equated": 100,
"score_private_raw": 100,
"score_public": 76.768967,
"set_hash": "641ab45baeaf445a1057c060eb9b35205bd55fcde4e629b2b88cfa38fba30047",
"submitter": "Submitter F",
"verdict": "FLAG",
"version_tag": "v1",
"z": -15.114489,
"z_crit": 3
},
{
"direction": "no gap beyond sampling error",
"equating_offset": 13.753223,
"gap": 3.929416,
"gap_se": 4.536487,
"generation": "gen2",
"merkle_root": "4adc9bbb0bc17564ad2effaa4d6fc902a0f319bb3e26735ac67d1b988cbe6e2d",
"model": "rf_pixels",
"score_private_equated": 72.938472,
"score_private_raw": 59.185249,
"score_public": 76.867888,
"set_hash": "ad30d214daa285c5123df991dd0eae58038f99f3490b840299cb620279ff2e73",
"submitter": "Submitter D",
"verdict": "clear",
"version_tag": "v2",
"z": 0.86618,
"z_crit": 3
},
{
"direction": "no gap beyond sampling error",
"equating_offset": 13.753223,
"gap": 2.849799,
"gap_se": 4.500762,
"generation": "gen2",
"merkle_root": "4adc9bbb0bc17564ad2effaa4d6fc902a0f319bb3e26735ac67d1b988cbe6e2d",
"model": "rf_pixels",
"score_private_equated": 74.10737,
"score_private_raw": 60.354147,
"score_public": 76.957169,
"set_hash": "ad30d214daa285c5123df991dd0eae58038f99f3490b840299cb620279ff2e73",
"submitter": "Submitter F",
"verdict": "clear",
"version_tag": "v2",
"z": 0.633181,
"z_crit": 3
}
],
"labelling": "PUBLIC-DATA DEMONSTRATION OF THE PRIVATE PROTOCOL"
},
"sources": [
{
"path": "results/cross_site/organ_ct__rf_pixels.json",
"sha256": "38f26c32e987cb828498918b9cd641f3ff519c4cdc1ca72d3d0eb8f710e50089"
},
{
"path": "results/heldout/audit_report.json",
"sha256": "df460b6d91a6512dfa1e7dc650637fce2b4dd3a1ec32e03446bf6e11f9c597d2"
},
{
"path": "results/heldout/cycle.json",
"sha256": "e012b8978457b51b5c31d18d14d2cc92f100731897523306acae19934f943502"
},
{
"path": "results/heldout/manifests/gen1.seal.json",
"sha256": "91a7adfa29870a0719125cdd48f6dcb149f1f6485ac1cfa104c6e00057ad9b81"
},
{
"path": "results/heldout/manifests/gen2.seal.json",
"sha256": "f61470918a09cf39e074ef14fb7ade4ac3a13d36d9385f18cf9b92e4b5329547"
},
{
"path": "results/heldout/manifests/public.seal.json",
"sha256": "1a4fd930d2e781ac7163483990f15d6694e28e507d1b3ca009ff09395e47497c"
},
{
"path": "results/records/bloodmnist__rf_pixels.json",
"sha256": "76739a850ac1bb24c60dd2d71f085ebc8687952c990d088826191a4679dc21cf"
},
{
"path": "results/records/breastmnist__rf_pixels.json",
"sha256": "9f68ecf6943e2f433882a2dc00714ba0723d2625aa17e575f918655715061ec9"
},
{
"path": "results/records/dermamnist__rf_pixels.json",
"sha256": "0f26064a0c7124531762f81a7bd76a87d3ce29944e079fa41c819fb9807dc0c4"
},
{
"path": "results/records/octmnist__rf_pixels.json",
"sha256": "079e28937fd8338438ab8d3fc585bcf4b1e6afed70890756e090a979f29dc71f"
},
{
"path": "results/records/organamnist__rf_pixels.json",
"sha256": "4dd8dbaa4f0a51f0c227bdd032c7599976bbf8eca3972e4c2f71d22cb5e5a33a"
},
{
"path": "results/records/organcmnist__rf_pixels.json",
"sha256": "44a2e26953aa481da6cccd08d770ef0ba647aea8a783d3f9c4caab02929f7e63"
},
{
"path": "results/records/organsmnist__rf_pixels.json",
"sha256": "ef862848c2ebaad54ee489296f7b801cdc51cffa04063665aa9101ebf63565e8"
},
{
"path": "results/records/pathmnist__rf_pixels.json",
"sha256": "cb5912f0c678cff6b2808c4f30c629d88e16143940f5a02b4ca5bfaa98bea1a1"
},
{
"path": "results/records/pneumoniamnist__rf_pixels.json",
"sha256": "44ccab0b3753b863b63b2ce32551ce7c9162e750428a4012e5571b0dc5646998"
},
{
"path": "results/records/retinamnist__rf_pixels.json",
"sha256": "2486e33a75838d9e8341f6a19660d7e739200531bc90b8cfec7bccc602d2f09b"
},
{
"path": "results/records/tissuemnist__rf_pixels.json",
"sha256": "ca169123c48759ae9540a809c778b653c38444f37b175d83da08ffb064986957"
}
]
},
"declared": null,
"declared_note": "No vendor declaration. This is an open reference baseline implemented by NakedSignal, not a commercial product: there is no vendor to assert an intended use or to sign an attestation, so the declared block is null rather than filled in on the vendor's behalf.",
"expiry_basis": "the sealed held-out generation this evidence cycle is anchored to (gen2, set hash ad30d214daa285c5...) is published to expire 2026-09-20; per spec, a passport lapses when its generation retires or its spec version is superseded, whichever comes first",
"expiry_utc": "2026-09-20T00:00:00Z",
"issued_utc": "2026-08-03T16:03:42Z",
"issuer": {
"algo": "Ed25519",
"key_id": "ns-passport-2026-07",
"name": "NakedSignal",
"public_key_b64": "0PBCC6mjzcppR5QdPcK7vP/AamB6MP8l1T4ypMYbsMw="
},
"observed": null,
"observed_note": "Not available: this requires continuous monitoring. Production history and drift are observed fields, and nothing is deployed, so there is nothing to observe. The object is present and null on purpose. The shape is the roadmap, not a claim.",
"passport_version": "0.1",
"signature": "eP9h58x1fZvCqIoUtZYUUXUtD95TQstvFoCharEELEx2pczgcgjBTtlfY5tlclfrL8maATQCRmD3QvUDGAoPAQ==",
"spec_version": "MedEval-1 v0.1",
"subject": {
"behaviour_version": "1.0.0",
"code_fingerprint": "b444c02112cc",
"code_fingerprints": [
"b444c02112cc"
],
"family": "classical",
"model_id": "rf_pixels",
"model_name": "Random forest on raw pixels",
"params": "200 trees, sqrt features"
}
}Canonicalisation: RFC 8785 (JCS)-compatible: UTF-8, object keys sorted by code point, no insignificant whitespace, ECMAScript number formatting. every float is rounded once at emit to 6 decimal places (Python round(), banker's rounding); integers are left exact. Signed over the whole document with the `signature` member removed, canonicalised as above and encoded UTF-8. The public key for ns-passport-2026-07 is 0PBCC6mjzcppR5QdPcK7vP/AamB6MP8l1T4ypMYbsMw= — see Governance.