Back to the consoleDocumentation
ResNet-18 (ImageNet) frozen features + linear probe
LIVEThe signed evidence record for resnet18_in_probe. Everything below was read out of the passport file; the signature was checked when this page was built, by the same code the Verify button runs.
SIGNATURE VALIDexit 0signature valid, issued by NakedSignal under key ns-passport-2026-07, not expired
SubjectResNet-18 (ImageNet) frozen features + linear probe
resnet18_in_probeConfiguration512-d penultimate features at 64px, logistic probe
Code fingerprintb444c02112cc
Spec versionMedEval-1 v0.1
Issued2026-08-03T16:03:42Z by NakedSignal
Expires2026-09-20T00:00:00Z — the sealed held-out generation this evidence cycle is anchored to (gen2, set hash ad30d214daa285c5...) is published to expire 2026-09-20; per spec, a passport lapses when its generation retires or its spec version is superseded, whichever comes first
key ns-passport-2026-07 · Ed25519
The check runs against the public key published on the governance page, using your browser's own Ed25519 implementation. The exit code shown is the exit code verify_passport.py returns for the same document.
Measured by NakedSignal
Computed in this repo by the MedEval-1 harness on data the model had never seen. Every number below was copied out of a file the harness wrote — 17 of them, each listed with its SHA-256 at the foot of this panel — and is printed exactly as it was signed, not re-rounded for display.
| Task | Domain | n test | accuracy | calibration | uncertainty | subgroup | corruption | acquisition | limited data | cost | spec sensitivity | Index |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bloodmnist | medicine | 3421 | 76.014262 | 51.601034 | 94.02981 | 58.551308 | 50.155791 | 71.936651 | 76.149978 | n/a | n/a | 66.66397 |
| breastmnist | medicine | 156 | 46.616541 | 17.018414 | 73.519992 | 14.285714 | 50.128008 | 67.885305 | 98.74552 | n/a | n/a | 49.472661 |
| dermamnist | medicine | 2005 | 37.203262 | 35.214237 | 84.111838 | 13.768116 | 49.696555 | 70.777816 | 86.649961 | n/a | n/a | 49.737882 |
| octmnist | medicine | 1000 | 41.733333 | 4.540541 | 64.754927 | 0 | 54.647802 | 70.820021 | 51.224707 | n/a | n/a | 41.345418 |
| organamnist | medicine | 17778 | 80.26346 | 44.670084 | 95.187361 | 56.831395 | 70.617699 | 81.744086 | 80.01797 | n/a | n/a | 72.013472 |
| organcmnist | medicine | 8216 | 76.419873 | 31.008598 | 92.465257 | 53.397524 | 70.189475 | 81.636188 | 73.141207 | n/a | n/a | 67.823011 |
| organsmnist | medicine | 8827 | 60.57897 | 0 | 83.623543 | 34.126984 | 70.08264 | 81.944553 | 71.873184 | n/a | n/a | 56.644098 |
| pathmnist | medicine | 7180 | 73.893189 | 40.019436 | 93.513353 | 38.004751 | 60.477733 | 76.442541 | 82.498349 | n/a | n/a | 65.165341 |
| pneumoniamnist | medicine | 624 | 50.769231 | 23.487984 | 87.516522 | 7.692308 | 49.270483 | 71.245791 | 79.292929 | n/a | n/a | 50.248103 |
| retinamnist | medicine | 400 | 18.666476 | 0 | 52.780325 | 0 | 54.715221 | 61.130854 | 95.785021 | n/a | n/a | 36.47283 |
| tissuemnist | medicine | 47280 | 28.654465 | 47.096774 | 60.818367 | 0 | 52.111294 | 70.396442 | 67.975465 | n/a | n/a | 45.13314 |
Cross-site retention · organ_ct
retention is absolute retained skill (chance-corrected), per spec 2.2.1; the diagonal is same-site, every off-diagonal cell is train-on-row, test-on-column
Sitesorganamnist, organcmnist, organsmnist
Mean retention0.582483
Worst retention0.29153
Private held-out track
PUBLIC-DATA DEMONSTRATION OF THE PRIVATE PROTOCOL
| Generation | Public | Private (equated) | Gap | z | Verdict |
|---|---|---|---|---|---|
| gen1 | 75.589367 | 73.653825 | 1.935542 | 0.852613 | clear |
| gen2 | 76.051965 | 71.896509 | 4.155456 | 0.896918 | clear |
Known limitations
Subgroup metadataabsent on 11 of 11 corpora, so the subgroup score falls back to the worst diagnostic class. That absence is a disclosure, not a pass.
Training set cappedyes on at least one task — see the raw JSON for which
Observed in deploymentnothing — see the third panel
Audit chain
Bundles re-derived48 of 48
Ledger100 entries, head 6b7b371ff650a8ed…, chain intact
Re-run it yourself
python src/heldout/audit.py 1f66c7719ab3943c6fcc17 source files, with hashes
| results/cross_site/organ_ct__resnet18_in_probe.json | 68677d9c913e5d847958e0305faec568d2996c8d19f1fe52be0627fe705156d9 |
| results/heldout/audit_report.json | df460b6d91a6512dfa1e7dc650637fce2b4dd3a1ec32e03446bf6e11f9c597d2 |
| results/heldout/cycle.json | e012b8978457b51b5c31d18d14d2cc92f100731897523306acae19934f943502 |
| results/heldout/manifests/gen1.seal.json | 91a7adfa29870a0719125cdd48f6dcb149f1f6485ac1cfa104c6e00057ad9b81 |
| results/heldout/manifests/gen2.seal.json | f61470918a09cf39e074ef14fb7ade4ac3a13d36d9385f18cf9b92e4b5329547 |
| results/heldout/manifests/public.seal.json | 1a4fd930d2e781ac7163483990f15d6694e28e507d1b3ca009ff09395e47497c |
| results/records/bloodmnist__resnet18_in_probe.json | 01f721d601f4ecead0c97433da3db021100fe9c975d6ba693f41048994457bde |
| results/records/breastmnist__resnet18_in_probe.json | fbf955dda1a3f4f8b9f3611a11e61e9315eb861003691a4874bf16bfe018c222 |
| results/records/dermamnist__resnet18_in_probe.json | d43b735accf0d98fffa4534bf64982a3f0b77d9610c98402ede918e3850dc09a |
| results/records/octmnist__resnet18_in_probe.json | 1000cfff207b3362d8091fd765020b773fb5f875c2518dc4e6ee796c96aea247 |
| results/records/organamnist__resnet18_in_probe.json | a33f772c2f2ff3a13104b26938fe237275bd68f5e3e0564cf73baf9293762c0e |
| results/records/organcmnist__resnet18_in_probe.json | 530e771869ca253c46c8189dd669743ace86b0fa3666dc4479952a06e48cf9b6 |
| results/records/organsmnist__resnet18_in_probe.json | 149ef26f0ad2e390498ae1ae79e8b1756781fd7555088143f5e63e34daeb8bbd |
| results/records/pathmnist__resnet18_in_probe.json | b7675cabaaca35d926aa55257f82c0a92284937b06caf50e410a0b6590655e99 |
| results/records/pneumoniamnist__resnet18_in_probe.json | 665edd6539224d12ba59351db40ab6c0b3c2ee642d9d4ad87664c28b2980233a |
| results/records/retinamnist__resnet18_in_probe.json | cd4e738933a1a1cc6113c93cc104b8a2ff25d57eba779f7fac3c9639dab5dc14 |
| results/records/tissuemnist__resnet18_in_probe.json | 2eb99a936383b66360c7cf20436277405e93dc484a159f3520339a82cfc08aad |
Declared by vendor
Five fields a vendor asserts about its own product — intended use, forbidden use, training cutoff, regulatory clearances, and who is personally attesting. NakedSignal never fills these in on a vendor's behalf.
No vendor declaration. This is an open reference baseline implemented by NakedSignal, not a commercial product: there is no vendor to assert an intended use or to sign an attestation, so the declared block is null rather than filled in on the vendor's behalf.
Observed in deployment
Production history and drift — how the model has actually behaved since it was deployed, on real traffic.
Not available: this requires continuous monitoring. Production history and drift are observed fields, and nothing is deployed, so there is nothing to observe. The object is present and null on purpose. The shape is the roadmap, not a claim.
Raw JSON — the whole signed document, 82,805 characters
{
"canonicalisation": {
"float_rounding": "every float is rounded once at emit to 6 decimal places (Python round(), banker's rounding); integers are left exact",
"form": "RFC 8785 (JCS)-compatible: UTF-8, object keys sorted by code point, no insignificant whitespace, ECMAScript number formatting",
"signed_over": "the whole document with the `signature` member removed, canonicalised as above and encoded UTF-8"
},
"computed": {
"audit": {
"audit_command": "python src/heldout/audit.py 1f66c7719ab3943c6fcc",
"bundles": 48,
"ledger_entries": 100,
"ledger_head": "6b7b371ff650a8edaf477490f2a5f9b043433f3ba9941989ee1962ed8f1b1d88",
"ledger_intact": true,
"rederived": 48
},
"cross_site": {
"code_fingerprint": "b444c02112cc",
"device": "mps",
"family": "organ_ct",
"matrix": {
"organamnist": {
"organamnist": {
"balanced_accuracy": 0.820577,
"chance_corrected": 0.802635,
"ece": 0.11066
},
"organcmnist": {
"balanced_accuracy": 0.486749,
"chance_corrected": 0.435424,
"ece": 0.402993,
"retention": 0.542493
},
"organsmnist": {
"balanced_accuracy": 0.303629,
"chance_corrected": 0.233992,
"ece": 0.559318,
"retention": 0.29153
}
},
"organcmnist": {
"organamnist": {
"balanced_accuracy": 0.643157,
"chance_corrected": 0.607473,
"ece": 0.24354,
"retention": 0.794915
},
"organcmnist": {
"balanced_accuracy": 0.785635,
"chance_corrected": 0.764199,
"ece": 0.137983
},
"organsmnist": {
"balanced_accuracy": 0.42455,
"chance_corrected": 0.367005,
"ece": 0.437225,
"retention": 0.480248
}
},
"organsmnist": {
"organamnist": {
"balanced_accuracy": 0.443634,
"chance_corrected": 0.387997,
"ece": 0.376913,
"retention": 0.640482
},
"organcmnist": {
"balanced_accuracy": 0.501321,
"chance_corrected": 0.451454,
"ece": 0.302387,
"retention": 0.745232
},
"organsmnist": {
"balanced_accuracy": 0.641627,
"chance_corrected": 0.60579,
"ece": 0.203339
}
}
},
"mean_retention": 0.582483,
"note": "retention is absolute retained skill (chance-corrected), per spec 2.2.1; the diagonal is same-site, every off-diagonal cell is train-on-row, test-on-column",
"score": 58.248312,
"site_labels": {
"organamnist": "axial",
"organcmnist": "coronal",
"organsmnist": "sagittal"
},
"sites": [
"organamnist",
"organcmnist",
"organsmnist"
],
"spec_version": "MedEval-1 v0.1",
"worst_retention": 0.29153
},
"evaluations": [
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 66.66397
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.817305,
"auc": 0.976321,
"balanced_accuracy": 0.790125,
"ci95": [
0.776337,
0.80602
],
"n_test": 3421,
"score": 76.014262
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.778086,
"mean_agreement": 0.656444,
"mean_kappa": 0.597769,
"mean_retention": 0.719367,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.8492,
0.6175,
0.3025
],
"mean_kappa": 0.515273,
"mean_retention": 0.612114,
"retention_by_severity": [
0.9466,
0.6703,
0.2194
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.7775,
0.7008,
0.5833
],
"mean_kappa": 0.632114,
"mean_retention": 0.750158,
"retention_by_severity": [
0.8602,
0.8001,
0.5902
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8758,
0.715,
0.5558
],
"mean_kappa": 0.672558,
"mean_retention": 0.793662,
"retention_by_severity": [
0.9501,
0.81,
0.6209
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.7742,
0.7233,
0.6217
],
"mean_kappa": 0.658238,
"mean_retention": 0.80187,
"retention_by_severity": [
0.8469,
0.8144,
0.7443
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.7667,
0.5292,
0.4542
],
"mean_kappa": 0.510661,
"mean_retention": 0.639028,
"retention_by_severity": [
0.8677,
0.5628,
0.4867
],
"scored": true
}
},
"score": 71.936651,
"worst_retention": 0.219424
},
"calibration": {
"accuracy": 0.817305,
"brier": 0.282774,
"ece": 0.096798,
"mean_confidence": 0.912862,
"overconfidence": 0.095557,
"score": 51.601034
},
"corruption": {
"clean_reference_cc": 0.778086,
"mean_agreement": 0.508849,
"mean_kappa": 0.422201,
"mean_retention": 0.501558,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.7108,
0.4783,
0.3467
],
"mean_kappa": 0.424774,
"mean_retention": 0.50196,
"retention_by_severity": [
0.7688,
0.4807,
0.2564
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.7517,
0.5167,
0.2758
],
"mean_kappa": 0.43225,
"mean_retention": 0.475318,
"retention_by_severity": [
0.7957,
0.4662,
0.1641
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.81,
0.6925,
0.5117
],
"mean_kappa": 0.613227,
"mean_retention": 0.712588,
"retention_by_severity": [
0.8793,
0.7504,
0.5081
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.6308,
0.3642,
0.2108
],
"mean_kappa": 0.288097,
"mean_retention": 0.369806,
"retention_by_severity": [
0.6893,
0.3395,
0.0806
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.6425,
0.2483,
0.2267
],
"mean_kappa": 0.278446,
"mean_retention": 0.35645,
"retention_by_severity": [
0.7459,
0.0981,
0.2253
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9425,
0.8558,
0.6858
],
"mean_kappa": 0.798515,
"mean_retention": 0.905567,
"retention_by_severity": [
1,
0.9616,
0.7552
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.3558,
0.2358,
0.1925
],
"mean_kappa": 0.120095,
"mean_retention": 0.189217,
"retention_by_severity": [
0.3437,
0.1802,
0.0437
],
"scored": true
}
},
"score": 50.155791,
"worst_retention": 0.043716
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.760143,
"mean_retention": 0.7615,
"per_budget": {
"n100": {
"chance_corrected": 0.70225,
"n_labels": 800,
"retention": 0.92384
},
"n20": {
"chance_corrected": 0.620178,
"n_labels": 160,
"retention": 0.815871
},
"n5": {
"chance_corrected": 0.414117,
"n_labels": 40,
"retention": 0.544789
}
},
"score": 76.149978
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 58.551308,
"worst_class_recall": 0.637324
},
"uncertainty": {
"full_accuracy": 0.817305,
"risk_coverage_auc": 0.947761,
"score": 94.02981,
"selective_acc_at_50": 0.980702,
"selective_acc_at_80": 0.89441
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 11959,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Microscopy",
"n_classes": 8,
"n_test": 3421,
"runtime_s": 10.4,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "bloodmnist",
"task_name": "Peripheral blood cells (8-class)",
"timestamp": "2026-08-01T19:19:00+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 49.472661
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.807692,
"auc": 0.790936,
"balanced_accuracy": 0.733083,
"ci95": [
0.659618,
0.804042
],
"n_test": 156,
"score": 46.616541
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.466165,
"mean_agreement": 0.768376,
"mean_kappa": 0.450623,
"mean_retention": 0.678853,
"n_cases": 156,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.8846,
0.6218,
0.5192
],
"mean_kappa": 0.341984,
"mean_retention": 0.501792,
"retention_by_severity": [
0.7473,
0.5081,
0.25
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.7885,
0.7628,
0.6923
],
"mean_kappa": 0.411384,
"mean_retention": 0.704301,
"retention_by_severity": [
0.75,
0.8898,
0.4731
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8846,
0.8141,
0.7564
],
"mean_kappa": 0.567146,
"mean_retention": 0.916667,
"retention_by_severity": [
1,
0.9597,
0.7903
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.8141,
0.7436,
0.6731
],
"mean_kappa": 0.388814,
"mean_retention": 0.697133,
"retention_by_severity": [
0.5887,
0.7688,
0.7339
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9487,
0.8462,
0.7756
],
"mean_kappa": 0.543787,
"mean_retention": 0.574373,
"retention_by_severity": [
1,
0.5914,
0.1317
],
"scored": true
}
},
"score": 67.885305,
"worst_retention": 0.13172
},
"calibration": {
"accuracy": 0.807692,
"brier": 0.348172,
"ece": 0.165963,
"mean_confidence": 0.93107,
"overconfidence": 0.123377,
"score": 17.018414
},
"corruption": {
"clean_reference_cc": 0.466165,
"mean_agreement": 0.746032,
"mean_kappa": 0.364619,
"mean_retention": 0.50128,
"n_cases": 156,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9167,
0.8526,
0.7885
],
"mean_kappa": 0.461856,
"mean_retention": 0.357527,
"retention_by_severity": [
0.6478,
0.3468,
0.078
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.9103,
0.8077,
0.5705
],
"mean_kappa": 0.50344,
"mean_retention": 0.796595,
"retention_by_severity": [
0.9624,
0.9032,
0.5242
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9038,
0.7308,
0.6154
],
"mean_kappa": 0.476907,
"mean_retention": 0.733871,
"retention_by_severity": [
0.7984,
0.6505,
0.7527
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.6667,
0.7051,
0.7628
],
"mean_kappa": 0.215821,
"mean_retention": 0.387993,
"retention_by_severity": [
0.7419,
0.4086,
0.0134
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.7821,
0.4551,
0.4872
],
"mean_kappa": 0.0753,
"mean_retention": 0.102151,
"retention_by_severity": [
0.3065,
0,
0
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9423,
0.8269,
0.7821
],
"mean_kappa": 0.53241,
"mean_retention": 0.629928,
"retention_by_severity": [
1,
0.7016,
0.1882
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.6795,
0.7051,
0.7756
],
"mean_kappa": 0.286598,
"mean_retention": 0.500896,
"retention_by_severity": [
0.5054,
0.7258,
0.2715
],
"scored": true
}
},
"score": 50.128008,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.466165,
"mean_retention": 0.987455,
"per_budget": {
"n100": {
"chance_corrected": 0.448622,
"n_labels": 200,
"retention": 0.962366
},
"n20": {
"chance_corrected": 0.528822,
"n_labels": 40,
"retention": 1
},
"n5": {
"chance_corrected": 0.5,
"n_labels": 10,
"retention": 1
}
},
"score": 98.74552
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 14.285714,
"worst_class_recall": 0.571429
},
"uncertainty": {
"full_accuracy": 0.807692,
"risk_coverage_auc": 0.8676,
"score": 73.519992,
"selective_acc_at_50": 0.871795,
"selective_acc_at_80": 0.824
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 546,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Ultrasound",
"n_classes": 2,
"n_test": 156,
"runtime_s": 1.8,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "breastmnist",
"task_name": "Breast ultrasound (malignancy)",
"timestamp": "2026-08-01T19:12:57+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 49.737882
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.710723,
"auc": 0.884679,
"balanced_accuracy": 0.461742,
"ci95": [
0.411725,
0.510574
],
"n_test": 2005,
"score": 37.203262
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.380896,
"mean_agreement": 0.793722,
"mean_kappa": 0.54349,
"mean_retention": 0.707778,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.7883,
0.7225,
0.6208
],
"mean_kappa": 0.369184,
"mean_retention": 0.583778,
"retention_by_severity": [
0.7967,
0.4906,
0.464
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8458,
0.8083,
0.7533
],
"mean_kappa": 0.55531,
"mean_retention": 0.758143,
"retention_by_severity": [
0.8132,
0.8169,
0.6443
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8875,
0.7967,
0.7467
],
"mean_kappa": 0.598681,
"mean_retention": 0.590008,
"retention_by_severity": [
0.7789,
0.5405,
0.4506
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.8425,
0.8075,
0.7808
],
"mean_kappa": 0.55222,
"mean_retention": 0.750207,
"retention_by_severity": [
0.754,
0.7782,
0.7184
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9217,
0.8325,
0.7508
],
"mean_kappa": 0.642053,
"mean_retention": 0.856755,
"retention_by_severity": [
0.9959,
0.8356,
0.7387
],
"scored": true
}
},
"score": 70.777816,
"worst_retention": 0.450641
},
"calibration": {
"accuracy": 0.710723,
"brier": 0.426254,
"ece": 0.129572,
"mean_confidence": 0.838829,
"overconfidence": 0.128105,
"score": 35.214237
},
"corruption": {
"clean_reference_cc": 0.380896,
"mean_agreement": 0.670714,
"mean_kappa": 0.342061,
"mean_retention": 0.496966,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.8775,
0.7433,
0.5325
],
"mean_kappa": 0.485281,
"mean_retention": 0.710425,
"retention_by_severity": [
1,
0.6973,
0.434
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8317,
0.7258,
0.6875
],
"mean_kappa": 0.42266,
"mean_retention": 0.509969,
"retention_by_severity": [
0.8114,
0.4836,
0.2349
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.8717,
0.785,
0.73
],
"mean_kappa": 0.484752,
"mean_retention": 0.630005,
"retention_by_severity": [
0.8072,
0.6909,
0.392
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.7133,
0.5425,
0.4875
],
"mean_kappa": 0.162105,
"mean_retention": 0.320188,
"retention_by_severity": [
0.5392,
0.3756,
0.0457
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.7517,
0.2317,
0.5867
],
"mean_kappa": 0.25528,
"mean_retention": 0.419226,
"retention_by_severity": [
0.8525,
0.1564,
0.2488
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.8975,
0.7642,
0.64
],
"mean_kappa": 0.523017,
"mean_retention": 0.680946,
"retention_by_severity": [
0.9702,
0.6244,
0.4482
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.6133,
0.4892,
0.5825
],
"mean_kappa": 0.06133,
"mean_retention": 0.208,
"retention_by_severity": [
0.4462,
0.1778,
0
],
"scored": true
}
},
"score": 49.696555,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.372033,
"mean_retention": 0.8665,
"per_budget": {
"n100": {
"chance_corrected": 0.38409,
"n_labels": 679,
"retention": 1
},
"n20": {
"chance_corrected": 0.349731,
"n_labels": 140,
"retention": 0.940055
},
"n5": {
"chance_corrected": 0.245335,
"n_labels": 35,
"retention": 0.659444
}
},
"score": 86.649961
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 13.768116,
"worst_class_recall": 0.26087
},
"uncertainty": {
"full_accuracy": 0.710723,
"risk_coverage_auc": 0.863816,
"score": 84.111838,
"selective_acc_at_50": 0.893214,
"selective_acc_at_80": 0.788653
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 7007,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Dermatoscopy",
"n_classes": 7,
"n_test": 2005,
"runtime_s": 9.4,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "dermamnist",
"task_name": "Dermatoscopy (7-class skin lesion)",
"timestamp": "2026-08-01T19:17:22+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 41.345418
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.563,
"auc": 0.83976,
"balanced_accuracy": 0.563,
"ci95": [
0.539559,
0.586237
],
"n_test": 1000,
"score": 41.733333
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.417333,
"mean_agreement": 0.6384,
"mean_kappa": 0.445302,
"mean_retention": 0.7082,
"n_cases": 1000,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.53,
0.444,
0.448
],
"mean_kappa": 0.186201,
"mean_retention": 0.368477,
"retention_by_severity": [
0.5559,
0.2556,
0.2939
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.762,
0.689,
0.638
],
"mean_kappa": 0.528683,
"mean_retention": 0.903088,
"retention_by_severity": [
0.9329,
0.8978,
0.8786
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.838,
0.741,
0.665
],
"mean_kappa": 0.59493,
"mean_retention": 0.839191,
"retention_by_severity": [
0.9521,
0.8658,
0.6997
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.7,
0.639,
0.58
],
"mean_kappa": 0.430932,
"mean_retention": 0.637913,
"retention_by_severity": [
0.7316,
0.5815,
0.6006
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.85,
0.583,
0.469
],
"mean_kappa": 0.485764,
"mean_retention": 0.792332,
"retention_by_severity": [
0.984,
0.8211,
0.5719
],
"scored": true
}
},
"score": 70.820021,
"worst_retention": 0.255591
},
"calibration": {
"accuracy": 0.563,
"brier": 0.612211,
"ece": 0.190919,
"mean_confidence": 0.753919,
"overconfidence": 0.190919,
"score": 4.540541
},
"corruption": {
"clean_reference_cc": 0.417333,
"mean_agreement": 0.567143,
"mean_kappa": 0.345154,
"mean_retention": 0.546478,
"n_cases": 1000,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.82,
0.706,
0.564
],
"mean_kappa": 0.535004,
"mean_retention": 0.8541,
"retention_by_severity": [
1,
0.9553,
0.607
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.824,
0.731,
0.633
],
"mean_kappa": 0.575702,
"mean_retention": 0.863685,
"retention_by_severity": [
1,
0.8914,
0.6997
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.785,
0.613,
0.466
],
"mean_kappa": 0.391396,
"mean_retention": 0.531416,
"retention_by_severity": [
0.8658,
0.5591,
0.1693
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.449,
0.504,
0.489
],
"mean_kappa": 0.185437,
"mean_retention": 0.353568,
"retention_by_severity": [
0.3131,
0.3642,
0.3834
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.667,
0.269,
0.323
],
"mean_kappa": 0.211718,
"mean_retention": 0.357827,
"retention_by_severity": [
0.7604,
0.2396,
0.0735
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.763,
0.578,
0.472
],
"mean_kappa": 0.39302,
"mean_retention": 0.610224,
"retention_by_severity": [
0.8882,
0.607,
0.3355
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.407,
0.414,
0.433
],
"mean_kappa": 0.123803,
"mean_retention": 0.254526,
"retention_by_severity": [
0.2875,
0.2492,
0.2268
],
"scored": true
}
},
"score": 54.647802,
"worst_retention": 0.073482
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.417333,
"mean_retention": 0.512247,
"per_budget": {
"n100": {
"chance_corrected": 0.333333,
"n_labels": 400,
"retention": 0.798722
},
"n20": {
"chance_corrected": 0.308,
"n_labels": 80,
"retention": 0.738019
},
"n5": {
"chance_corrected": 0,
"n_labels": 20,
"retention": 0
}
},
"score": 51.224707
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0.072
},
"uncertainty": {
"full_accuracy": 0.563,
"risk_coverage_auc": 0.735662,
"score": 64.754927,
"selective_acc_at_50": 0.73,
"selective_acc_at_80": 0.62
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 97477,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "OCT",
"n_classes": 4,
"n_test": 1000,
"runtime_s": 7.7,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "octmnist",
"task_name": "Retinal OCT (4-class)",
"timestamp": "2026-08-01T19:35:39+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 72.013472
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.832208,
"auc": 0.980734,
"balanced_accuracy": 0.820577,
"ci95": [
0.814944,
0.826262
],
"n_test": 17778,
"score": 80.26346
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.820456,
"mean_agreement": 0.728056,
"mean_kappa": 0.697271,
"mean_retention": 0.817441,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9033,
0.7417,
0.4325
],
"mean_kappa": 0.657868,
"mean_retention": 0.776918,
"retention_by_severity": [
1,
0.859,
0.4717
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.7392,
0.6392,
0.5025
],
"mean_kappa": 0.586228,
"mean_retention": 0.733524,
"retention_by_severity": [
0.8771,
0.7417,
0.5818
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9092,
0.855,
0.8083
],
"mean_kappa": 0.840235,
"mean_retention": 0.957598,
"retention_by_severity": [
0.9864,
0.9719,
0.9145
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.6992,
0.6142,
0.485
],
"mean_kappa": 0.55505,
"mean_retention": 0.679752,
"retention_by_severity": [
0.7849,
0.7091,
0.5453
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9492,
0.8658,
0.7767
],
"mean_kappa": 0.846973,
"mean_retention": 0.939412,
"retention_by_severity": [
1,
0.9542,
0.8641
],
"scored": true
}
},
"score": 81.744086,
"worst_retention": 0.471725
},
"calibration": {
"accuracy": 0.832208,
"brier": 0.275722,
"ece": 0.11066,
"mean_confidence": 0.942804,
"overconfidence": 0.110595,
"score": 44.670084
},
"corruption": {
"clean_reference_cc": 0.820456,
"mean_agreement": 0.651111,
"mean_kappa": 0.61292,
"mean_retention": 0.706177,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.8817,
0.7483,
0.4817
],
"mean_kappa": 0.673022,
"mean_retention": 0.779452,
"retention_by_severity": [
0.9803,
0.8426,
0.5154
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8267,
0.6742,
0.4758
],
"mean_kappa": 0.622777,
"mean_retention": 0.744783,
"retention_by_severity": [
0.9171,
0.768,
0.5492
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.7392,
0.555,
0.3542
],
"mean_kappa": 0.500182,
"mean_retention": 0.607432,
"retention_by_severity": [
0.8465,
0.6061,
0.3697
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.8392,
0.7275,
0.4442
],
"mean_kappa": 0.627876,
"mean_retention": 0.711534,
"retention_by_severity": [
0.9559,
0.7829,
0.3958
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.7283,
0.1858,
0.1642
],
"mean_kappa": 0.299786,
"mean_retention": 0.385365,
"retention_by_severity": [
0.8254,
0.1731,
0.1576
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9542,
0.935,
0.8667
],
"mean_kappa": 0.908789,
"mean_retention": 0.991613,
"retention_by_severity": [
0.997,
1,
0.9778
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.805,
0.7067,
0.58
],
"mean_kappa": 0.65801,
"mean_retention": 0.723061,
"retention_by_severity": [
0.8865,
0.7626,
0.5201
],
"scored": true
}
},
"score": 70.617699,
"worst_retention": 0.157612
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.802635,
"mean_retention": 0.80018,
"per_budget": {
"n100": {
"chance_corrected": 0.759946,
"n_labels": 1100,
"retention": 0.946814
},
"n20": {
"chance_corrected": 0.665681,
"n_labels": 220,
"retention": 0.82937
},
"n5": {
"chance_corrected": 0.501129,
"n_labels": 55,
"retention": 0.624355
}
},
"score": 80.01797
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 56.831395,
"worst_class_recall": 0.607558
},
"uncertainty": {
"full_accuracy": 0.832208,
"risk_coverage_auc": 0.956249,
"score": 95.187361,
"selective_acc_at_50": 0.979638,
"selective_acc_at_80": 0.915835
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 34561,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 17778,
"runtime_s": 14,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organamnist",
"task_name": "Abdominal CT organs — axial (A)",
"timestamp": "2026-08-01T19:32:59+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 67.823011
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.801485,
"auc": 0.972205,
"balanced_accuracy": 0.785635,
"ci95": [
0.775595,
0.794254
],
"n_test": 8216,
"score": 76.419873
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.73275,
"mean_agreement": 0.690778,
"mean_kappa": 0.655874,
"mean_retention": 0.816362,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.8833,
0.7117,
0.3417
],
"mean_kappa": 0.606968,
"mean_retention": 0.749069,
"retention_by_severity": [
0.9906,
0.8442,
0.4124
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.6967,
0.6267,
0.4542
],
"mean_kappa": 0.54834,
"mean_retention": 0.774917,
"retention_by_severity": [
0.8901,
0.8221,
0.6125
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9142,
0.8375,
0.7708
],
"mean_kappa": 0.82038,
"mean_retention": 0.929962,
"retention_by_severity": [
0.968,
0.9344,
0.8874
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.6442,
0.545,
0.4292
],
"mean_kappa": 0.489452,
"mean_retention": 0.667261,
"retention_by_severity": [
0.7819,
0.6804,
0.5395
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9242,
0.8442,
0.7383
],
"mean_kappa": 0.814231,
"mean_retention": 0.960601,
"retention_by_severity": [
0.9914,
0.9789,
0.9116
],
"scored": true
}
},
"score": 81.636188,
"worst_retention": 0.412437
},
"calibration": {
"accuracy": 0.801485,
"brier": 0.332896,
"ece": 0.137983,
"mean_confidence": 0.939468,
"overconfidence": 0.137983,
"score": 31.008598
},
"corruption": {
"clean_reference_cc": 0.73275,
"mean_agreement": 0.612897,
"mean_kappa": 0.570245,
"mean_retention": 0.701895,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.8575,
0.7217,
0.4983
],
"mean_kappa": 0.65633,
"mean_retention": 0.77668,
"retention_by_severity": [
0.9607,
0.8236,
0.5458
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8008,
0.6125,
0.3775
],
"mean_kappa": 0.552201,
"mean_retention": 0.66931,
"retention_by_severity": [
0.8991,
0.6901,
0.4188
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.735,
0.4683,
0.24
],
"mean_kappa": 0.429484,
"mean_retention": 0.568585,
"retention_by_severity": [
0.8756,
0.5619,
0.2683
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.8083,
0.6908,
0.4208
],
"mean_kappa": 0.592771,
"mean_retention": 0.741363,
"retention_by_severity": [
0.9683,
0.8362,
0.4197
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.695,
0.21,
0.1708
],
"mean_kappa": 0.30352,
"mean_retention": 0.435881,
"retention_by_severity": [
0.8908,
0.2594,
0.1574
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9333,
0.8933,
0.7975
],
"mean_kappa": 0.858294,
"mean_retention": 0.969784,
"retention_by_severity": [
1,
0.9893,
0.9201
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.77,
0.6658,
0.5033
],
"mean_kappa": 0.599117,
"mean_retention": 0.751659,
"retention_by_severity": [
0.9114,
0.7797,
0.5639
],
"scored": true
}
},
"score": 70.189475,
"worst_retention": 0.15739
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.764199,
"mean_retention": 0.731412,
"per_budget": {
"n100": {
"chance_corrected": 0.703241,
"n_labels": 1100,
"retention": 0.920233
},
"n20": {
"chance_corrected": 0.599575,
"n_labels": 220,
"retention": 0.78458
},
"n5": {
"chance_corrected": 0.374016,
"n_labels": 55,
"retention": 0.489423
}
},
"score": 73.141207
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 53.397524,
"worst_class_recall": 0.576341
},
"uncertainty": {
"full_accuracy": 0.801485,
"risk_coverage_auc": 0.931502,
"score": 92.465257,
"selective_acc_at_50": 0.957157,
"selective_acc_at_80": 0.877681
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 12975,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 8216,
"runtime_s": 17.6,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organcmnist",
"task_name": "Abdominal CT organs — coronal (C)",
"timestamp": "2026-08-01T19:28:02+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 56.644098
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.684377,
"auc": 0.949369,
"balanced_accuracy": 0.641627,
"ci95": [
0.632043,
0.651526
],
"n_test": 8827,
"score": 60.57897
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.591739,
"mean_agreement": 0.644444,
"mean_kappa": 0.5963,
"mean_retention": 0.819446,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.85,
0.6417,
0.385
],
"mean_kappa": 0.574283,
"mean_retention": 0.798388,
"retention_by_severity": [
1,
0.8921,
0.503
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.6317,
0.5658,
0.4275
],
"mean_kappa": 0.482668,
"mean_retention": 0.750947,
"retention_by_severity": [
0.8379,
0.8079,
0.607
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8808,
0.79,
0.7017
],
"mean_kappa": 0.758118,
"mean_retention": 0.90262,
"retention_by_severity": [
0.9694,
0.9297,
0.8088
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.5908,
0.5058,
0.3975
],
"mean_kappa": 0.437681,
"mean_retention": 0.743901,
"retention_by_severity": [
0.8686,
0.7386,
0.6245
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.8975,
0.7683,
0.6325
],
"mean_kappa": 0.72875,
"mean_retention": 0.901372,
"retention_by_severity": [
1,
0.9107,
0.7934
],
"scored": true
}
},
"score": 81.944553,
"worst_retention": 0.503034
},
"calibration": {
"accuracy": 0.684377,
"brier": 0.503015,
"ece": 0.203339,
"mean_confidence": 0.887607,
"overconfidence": 0.20323,
"score": 0
},
"corruption": {
"clean_reference_cc": 0.591739,
"mean_agreement": 0.574683,
"mean_kappa": 0.516624,
"mean_retention": 0.700826,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.82,
0.6642,
0.4358
],
"mean_kappa": 0.588498,
"mean_retention": 0.753957,
"retention_by_severity": [
0.9438,
0.8133,
0.5048
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.7125,
0.53,
0.3725
],
"mean_kappa": 0.478767,
"mean_retention": 0.655402,
"retention_by_severity": [
0.8652,
0.6587,
0.4423
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.6808,
0.44,
0.245
],
"mean_kappa": 0.395474,
"mean_retention": 0.632721,
"retention_by_severity": [
0.9066,
0.6506,
0.341
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.7325,
0.625,
0.4708
],
"mean_kappa": 0.541008,
"mean_retention": 0.730405,
"retention_by_severity": [
0.9213,
0.8158,
0.4541
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.6133,
0.1833,
0.1908
],
"mean_kappa": 0.25597,
"mean_retention": 0.408171,
"retention_by_severity": [
0.8655,
0.185,
0.174
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.93,
0.8583,
0.735
],
"mean_kappa": 0.815995,
"mean_retention": 0.967735,
"retention_by_severity": [
1,
0.9848,
0.9185
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.7075,
0.6125,
0.5083
],
"mean_kappa": 0.540657,
"mean_retention": 0.757392,
"retention_by_severity": [
0.9115,
0.7733,
0.5874
],
"scored": true
}
},
"score": 70.08264,
"worst_retention": 0.174001
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.60579,
"mean_retention": 0.718732,
"per_budget": {
"n100": {
"chance_corrected": 0.54773,
"n_labels": 1100,
"retention": 0.904158
},
"n20": {
"chance_corrected": 0.45734,
"n_labels": 220,
"retention": 0.754948
},
"n5": {
"chance_corrected": 0.301131,
"n_labels": 55,
"retention": 0.497089
}
},
"score": 71.873184
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 34.126984,
"worst_class_recall": 0.401154
},
"uncertainty": {
"full_accuracy": 0.684377,
"risk_coverage_auc": 0.851123,
"score": 83.623543,
"selective_acc_at_50": 0.865202,
"selective_acc_at_80": 0.754602
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 13932,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 8827,
"runtime_s": 12.1,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organsmnist",
"task_name": "Abdominal CT organs — sagittal (S)",
"timestamp": "2026-08-01T19:30:44+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 65.165341
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.807799,
"auc": 0.972101,
"balanced_accuracy": 0.767939,
"ci95": [
0.757743,
0.778933
],
"n_test": 7180,
"score": 73.893189
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.721493,
"mean_agreement": 0.686056,
"mean_kappa": 0.643897,
"mean_retention": 0.764425,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.8683,
0.6117,
0.3183
],
"mean_kappa": 0.548397,
"mean_retention": 0.684796,
"retention_by_severity": [
0.9723,
0.7105,
0.3716
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.7475,
0.7258,
0.6508
],
"mean_kappa": 0.666108,
"mean_retention": 0.814737,
"retention_by_severity": [
0.8324,
0.8559,
0.7559
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9242,
0.8667,
0.8067
],
"mean_kappa": 0.847869,
"mean_retention": 0.963946,
"retention_by_severity": [
0.9927,
0.9737,
0.9254
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.665,
0.5817,
0.5175
],
"mean_kappa": 0.527184,
"mean_retention": 0.618775,
"retention_by_severity": [
0.7239,
0.6065,
0.5259
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.8783,
0.7058,
0.4225
],
"mean_kappa": 0.629926,
"mean_retention": 0.739873,
"retention_by_severity": [
0.9639,
0.8009,
0.4548
],
"scored": true
}
},
"score": 76.442541,
"worst_retention": 0.371598
},
"calibration": {
"accuracy": 0.807799,
"brier": 0.304667,
"ece": 0.119961,
"mean_confidence": 0.927511,
"overconfidence": 0.119712,
"score": 40.019436
},
"corruption": {
"clean_reference_cc": 0.721493,
"mean_agreement": 0.547698,
"mean_kappa": 0.489726,
"mean_retention": 0.604777,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.8808,
0.7258,
0.4375
],
"mean_kappa": 0.643912,
"mean_retention": 0.78467,
"retention_by_severity": [
0.9863,
0.8448,
0.5229
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8525,
0.7158,
0.4992
],
"mean_kappa": 0.647489,
"mean_retention": 0.784816,
"retention_by_severity": [
0.9516,
0.8474,
0.5554
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.7767,
0.525,
0.3008
],
"mean_kappa": 0.466277,
"mean_retention": 0.535515,
"retention_by_severity": [
0.8458,
0.499,
0.2617
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.6658,
0.3942,
0.1858
],
"mean_kappa": 0.346713,
"mean_retention": 0.479033,
"retention_by_severity": [
0.8044,
0.4773,
0.1554
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.7192,
0.1558,
0.2683
],
"mean_kappa": 0.293011,
"mean_retention": 0.364972,
"retention_by_severity": [
0.8139,
0.0381,
0.2429
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9417,
0.8833,
0.6658
],
"mean_kappa": 0.808304,
"mean_retention": 0.919747,
"retention_by_severity": [
0.996,
0.9826,
0.7807
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.4492,
0.29,
0.1683
],
"mean_kappa": 0.222377,
"mean_retention": 0.364688,
"retention_by_severity": [
0.5795,
0.349,
0.1656
],
"scored": true
}
},
"score": 60.477733,
"worst_retention": 0.038051
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.738932,
"mean_retention": 0.824983,
"per_budget": {
"n100": {
"chance_corrected": 0.670346,
"n_labels": 900,
"retention": 0.907182
},
"n20": {
"chance_corrected": 0.636972,
"n_labels": 180,
"retention": 0.862017
},
"n5": {
"chance_corrected": 0.521502,
"n_labels": 45,
"retention": 0.705751
}
},
"score": 82.498349
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 38.004751,
"worst_class_recall": 0.448931
},
"uncertainty": {
"full_accuracy": 0.807799,
"risk_coverage_auc": 0.942341,
"score": 93.513353,
"selective_acc_at_50": 0.971031,
"selective_acc_at_80": 0.889276
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 89996,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "Histopathology",
"n_classes": 9,
"n_test": 7180,
"runtime_s": 13.5,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "pathmnist",
"task_name": "Colorectal pathology (9-class tissue)",
"timestamp": "2026-08-01T19:42:10+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 50.248103
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.807692,
"auc": 0.928852,
"balanced_accuracy": 0.753846,
"ci95": [
0.726653,
0.780446
],
"n_test": 624,
"score": 50.769231
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.507692,
"mean_agreement": 0.859615,
"mean_kappa": 0.538375,
"mean_retention": 0.712458,
"n_cases": 624,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9038,
0.8157,
0.7869
],
"mean_kappa": 0.49916,
"mean_retention": 0.717172,
"retention_by_severity": [
1,
0.7391,
0.4125
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8686,
0.8061,
0.7821
],
"mean_kappa": 0.258143,
"mean_retention": 0.289001,
"retention_by_severity": [
0.6229,
0.2273,
0.0168
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9471,
0.9022,
0.867
],
"mean_kappa": 0.750083,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.8654,
0.8654,
0.8157
],
"mean_kappa": 0.462851,
"mean_retention": 0.556117,
"retention_by_severity": [
0.5791,
0.7003,
0.3889
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9455,
0.8926,
0.8301
],
"mean_kappa": 0.721638,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
}
},
"score": 71.245791,
"worst_retention": 0.016835
},
"calibration": {
"accuracy": 0.807692,
"brier": 0.33157,
"ece": 0.153024,
"mean_confidence": 0.960716,
"overconfidence": 0.153024,
"score": 23.487984
},
"corruption": {
"clean_reference_cc": 0.507692,
"mean_agreement": 0.83692,
"mean_kappa": 0.405052,
"mean_retention": 0.492705,
"n_cases": 624,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9295,
0.899,
0.867
],
"mean_kappa": 0.716126,
"mean_retention": 0.932099,
"retention_by_severity": [
1,
1,
0.7963
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.9022,
0.8429,
0.7804
],
"mean_kappa": 0.362397,
"mean_retention": 0.352413,
"retention_by_severity": [
0.6616,
0.3704,
0.0253
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.8862,
0.8301,
0.7869
],
"mean_kappa": 0.348729,
"mean_retention": 0.391695,
"retention_by_severity": [
0.7256,
0.3872,
0.0623
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.8429,
0.8141,
0.75
],
"mean_kappa": 0.326811,
"mean_retention": 0.445567,
"retention_by_severity": [
0.798,
0.4579,
0.0808
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.8253,
0.7837,
0.7869
],
"mean_kappa": 0.129735,
"mean_retention": 0.109428,
"retention_by_severity": [
0.2441,
0.0253,
0.0589
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9391,
0.9071,
0.8734
],
"mean_kappa": 0.726215,
"mean_retention": 0.934343,
"retention_by_severity": [
1,
1,
0.803
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.8269,
0.7997,
0.7019
],
"mean_kappa": 0.225348,
"mean_retention": 0.283389,
"retention_by_severity": [
0.5825,
0.2306,
0.037
],
"scored": true
}
},
"score": 49.270483,
"worst_retention": 0.025253
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.507692,
"mean_retention": 0.792929,
"per_budget": {
"n100": {
"chance_corrected": 0.525641,
"n_labels": 200,
"retention": 1
},
"n20": {
"chance_corrected": 0.563248,
"n_labels": 40,
"retention": 1
},
"n5": {
"chance_corrected": 0.192308,
"n_labels": 10,
"retention": 0.378788
}
},
"score": 79.292929
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 7.692308,
"worst_class_recall": 0.538462
},
"uncertainty": {
"full_accuracy": 0.807692,
"risk_coverage_auc": 0.937583,
"score": 87.516522,
"selective_acc_at_50": 0.961538,
"selective_acc_at_80": 0.875752
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 4708,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Chest X-ray",
"n_classes": 2,
"n_test": 624,
"runtime_s": 3.7,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "pneumoniamnist",
"task_name": "Paediatric chest X-ray (pneumonia)",
"timestamp": "2026-08-01T19:16:22+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 36.47283
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.485,
"auc": 0.675656,
"balanced_accuracy": 0.349332,
"ci95": [
0.303953,
0.398363
],
"n_test": 400,
"score": 18.666476
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.186665,
"mean_agreement": 0.531833,
"mean_kappa": 0.341695,
"mean_retention": 0.611309,
"n_cases": 400,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.5625,
0.3925,
0.35
],
"mean_kappa": 0.254891,
"mean_retention": 0.546066,
"retention_by_severity": [
0.6842,
0.4379,
0.516
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.61,
0.4475,
0.3
],
"mean_kappa": 0.264942,
"mean_retention": 0.533772,
"retention_by_severity": [
0.7019,
0.6199,
0.2795
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.72,
0.6,
0.5175
],
"mean_kappa": 0.421798,
"mean_retention": 0.543142,
"retention_by_severity": [
0.8691,
0.4132,
0.3472
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.6475,
0.5875,
0.575
],
"mean_kappa": 0.410276,
"mean_retention": 0.840263,
"retention_by_severity": [
0.8524,
0.8647,
0.8037
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.6825,
0.53,
0.455
],
"mean_kappa": 0.356569,
"mean_retention": 0.593299,
"retention_by_severity": [
0.8007,
0.4964,
0.4828
],
"scored": true
}
},
"score": 61.130854,
"worst_retention": 0.279537
},
"calibration": {
"accuracy": 0.485,
"brier": 0.835986,
"ece": 0.367064,
"mean_confidence": 0.852064,
"overconfidence": 0.367064,
"score": 0
},
"corruption": {
"clean_reference_cc": 0.186665,
"mean_agreement": 0.443571,
"mean_kappa": 0.25864,
"mean_retention": 0.547152,
"n_cases": 400,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.63,
0.54,
0.335
],
"mean_kappa": 0.281859,
"mean_retention": 0.638437,
"retention_by_severity": [
0.8381,
0.7241,
0.3531
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.6375,
0.42,
0.2625
],
"mean_kappa": 0.253403,
"mean_retention": 0.488613,
"retention_by_severity": [
0.6293,
0.5576,
0.2789
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.6925,
0.585,
0.5275
],
"mean_kappa": 0.399615,
"mean_retention": 0.772976,
"retention_by_severity": [
0.8715,
0.6401,
0.8073
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.34,
0.335,
0.185
],
"mean_kappa": 0.097021,
"mean_retention": 0.24328,
"retention_by_severity": [
0.182,
0.5229,
0.025
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.515,
0.3425,
0.3075
],
"mean_kappa": 0.205359,
"mean_retention": 0.572987,
"retention_by_severity": [
0.6965,
0.8328,
0.1897
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.8,
0.61,
0.34
],
"mean_kappa": 0.440684,
"mean_retention": 0.784808,
"retention_by_severity": [
0.8494,
0.7972,
0.7079
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.38,
0.335,
0.195
],
"mean_kappa": 0.132539,
"mean_retention": 0.328964,
"retention_by_severity": [
0.4415,
0.3547,
0.1907
],
"scored": true
}
},
"score": 54.715221,
"worst_retention": 0.024956
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.186665,
"mean_retention": 0.95785,
"per_budget": {
"n100": {
"chance_corrected": 0.174173,
"n_labels": 466,
"retention": 0.93308
},
"n20": {
"chance_corrected": 0.231233,
"n_labels": 100,
"retention": 1
},
"n5": {
"chance_corrected": 0.175553,
"n_labels": 25,
"retention": 0.94047
}
},
"score": 95.785021
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0.05
},
"uncertainty": {
"full_accuracy": 0.485,
"risk_coverage_auc": 0.622243,
"score": 52.780325,
"selective_acc_at_50": 0.61,
"selective_acc_at_80": 0.55
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 1080,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Fundus",
"n_classes": 5,
"n_test": 400,
"runtime_s": 2.9,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "retinamnist",
"task_name": "Fundus photography (DR grade)",
"timestamp": "2026-08-01T19:13:15+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 45.13314
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.513283,
"auc": 0.821287,
"balanced_accuracy": 0.375727,
"ci95": [
0.370862,
0.380974
],
"n_test": 47280,
"score": 28.654465
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.299673,
"mean_agreement": 0.502722,
"mean_kappa": 0.378805,
"mean_retention": 0.703964,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.5958,
0.4558,
0.3458
],
"mean_kappa": 0.298093,
"mean_retention": 0.528401,
"retention_by_severity": [
0.7387,
0.6118,
0.2347
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.6892,
0.5967,
0.4808
],
"mean_kappa": 0.47115,
"mean_retention": 0.818499,
"retention_by_severity": [
0.9568,
0.8417,
0.657
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.69,
0.5083,
0.3983
],
"mean_kappa": 0.419119,
"mean_retention": 0.745299,
"retention_by_severity": [
0.9588,
0.7165,
0.5606
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.5875,
0.5117,
0.4692
],
"mean_kappa": 0.419635,
"mean_retention": 0.809108,
"retention_by_severity": [
0.8175,
0.8122,
0.7976
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.515,
0.3567,
0.34
],
"mean_kappa": 0.286026,
"mean_retention": 0.618516,
"retention_by_severity": [
0.7385,
0.6356,
0.4815
],
"scored": true
}
},
"score": 70.396442,
"worst_retention": 0.234686
},
"calibration": {
"accuracy": 0.513283,
"brier": 0.647385,
"ece": 0.105806,
"mean_confidence": 0.619089,
"overconfidence": 0.105806,
"score": 47.096774
},
"corruption": {
"clean_reference_cc": 0.299673,
"mean_agreement": 0.472857,
"mean_kappa": 0.328924,
"mean_retention": 0.521113,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.675,
0.5717,
0.3917
],
"mean_kappa": 0.411071,
"mean_retention": 0.592861,
"retention_by_severity": [
0.7404,
0.5887,
0.4495
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.7683,
0.615,
0.4625
],
"mean_kappa": 0.480164,
"mean_retention": 0.646347,
"retention_by_severity": [
0.8317,
0.6927,
0.4146
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.6483,
0.415,
0.2575
],
"mean_kappa": 0.340688,
"mean_retention": 0.685459,
"retention_by_severity": [
0.8929,
0.7147,
0.4488
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.4992,
0.3583,
0.2908
],
"mean_kappa": 0.178157,
"mean_retention": 0.303975,
"retention_by_severity": [
0.5624,
0.2525,
0.097
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.6283,
0.3217,
0.235
],
"mean_kappa": 0.273254,
"mean_retention": 0.476172,
"retention_by_severity": [
0.7163,
0.3552,
0.3571
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.72,
0.4867,
0.2533
],
"mean_kappa": 0.366219,
"mean_retention": 0.557079,
"retention_by_severity": [
0.8111,
0.5825,
0.2776
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.5233,
0.4517,
0.3567
],
"mean_kappa": 0.252916,
"mean_retention": 0.385898,
"retention_by_severity": [
0.6014,
0.3702,
0.1861
],
"scored": true
}
},
"score": 52.111294,
"worst_retention": 0.097004
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.286545,
"mean_retention": 0.679755,
"per_budget": {
"n100": {
"chance_corrected": 0.234441,
"n_labels": 800,
"retention": 0.818167
},
"n20": {
"chance_corrected": 0.205531,
"n_labels": 160,
"retention": 0.717272
},
"n5": {
"chance_corrected": 0.144368,
"n_labels": 40,
"retention": 0.503824
}
},
"score": 67.975465
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0.081057
},
"uncertainty": {
"full_accuracy": 0.513283,
"risk_coverage_auc": 0.657161,
"score": 60.818367,
"selective_acc_at_50": 0.651988,
"selective_acc_at_80": 0.565911
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 165466,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "Microscopy",
"n_classes": 8,
"n_test": 47280,
"runtime_s": 30.7,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "tissuemnist",
"task_name": "Kidney cortex cells (8-class)",
"timestamp": "2026-08-01T19:38:31+00:00"
}
],
"heldout": {
"entries": [
{
"direction": "no gap beyond sampling error",
"equating_offset": 0,
"gap": 1.935542,
"gap_se": 2.270128,
"generation": "gen1",
"merkle_root": "61c508cbafed177bb1553f05aa2cc80d4157adc1bc1ff611d88581aab9ca61c5",
"model": "resnet18_in_probe",
"score_private_equated": 73.653825,
"score_private_raw": 73.653825,
"score_public": 75.589367,
"set_hash": "641ab45baeaf445a1057c060eb9b35205bd55fcde4e629b2b88cfa38fba30047",
"submitter": "Submitter E",
"verdict": "clear",
"version_tag": "v1",
"z": 0.852613,
"z_crit": 3
},
{
"direction": "no gap beyond sampling error",
"equating_offset": 13.753223,
"gap": 4.155456,
"gap_se": 4.633041,
"generation": "gen2",
"merkle_root": "4adc9bbb0bc17564ad2effaa4d6fc902a0f319bb3e26735ac67d1b988cbe6e2d",
"model": "resnet18_in_probe",
"score_private_equated": 71.896509,
"score_private_raw": 58.143286,
"score_public": 76.051965,
"set_hash": "ad30d214daa285c5123df991dd0eae58038f99f3490b840299cb620279ff2e73",
"submitter": "Submitter E",
"verdict": "clear",
"version_tag": "v2",
"z": 0.896918,
"z_crit": 3
}
],
"labelling": "PUBLIC-DATA DEMONSTRATION OF THE PRIVATE PROTOCOL"
},
"sources": [
{
"path": "results/cross_site/organ_ct__resnet18_in_probe.json",
"sha256": "68677d9c913e5d847958e0305faec568d2996c8d19f1fe52be0627fe705156d9"
},
{
"path": "results/heldout/audit_report.json",
"sha256": "df460b6d91a6512dfa1e7dc650637fce2b4dd3a1ec32e03446bf6e11f9c597d2"
},
{
"path": "results/heldout/cycle.json",
"sha256": "e012b8978457b51b5c31d18d14d2cc92f100731897523306acae19934f943502"
},
{
"path": "results/heldout/manifests/gen1.seal.json",
"sha256": "91a7adfa29870a0719125cdd48f6dcb149f1f6485ac1cfa104c6e00057ad9b81"
},
{
"path": "results/heldout/manifests/gen2.seal.json",
"sha256": "f61470918a09cf39e074ef14fb7ade4ac3a13d36d9385f18cf9b92e4b5329547"
},
{
"path": "results/heldout/manifests/public.seal.json",
"sha256": "1a4fd930d2e781ac7163483990f15d6694e28e507d1b3ca009ff09395e47497c"
},
{
"path": "results/records/bloodmnist__resnet18_in_probe.json",
"sha256": "01f721d601f4ecead0c97433da3db021100fe9c975d6ba693f41048994457bde"
},
{
"path": "results/records/breastmnist__resnet18_in_probe.json",
"sha256": "fbf955dda1a3f4f8b9f3611a11e61e9315eb861003691a4874bf16bfe018c222"
},
{
"path": "results/records/dermamnist__resnet18_in_probe.json",
"sha256": "d43b735accf0d98fffa4534bf64982a3f0b77d9610c98402ede918e3850dc09a"
},
{
"path": "results/records/octmnist__resnet18_in_probe.json",
"sha256": "1000cfff207b3362d8091fd765020b773fb5f875c2518dc4e6ee796c96aea247"
},
{
"path": "results/records/organamnist__resnet18_in_probe.json",
"sha256": "a33f772c2f2ff3a13104b26938fe237275bd68f5e3e0564cf73baf9293762c0e"
},
{
"path": "results/records/organcmnist__resnet18_in_probe.json",
"sha256": "530e771869ca253c46c8189dd669743ace86b0fa3666dc4479952a06e48cf9b6"
},
{
"path": "results/records/organsmnist__resnet18_in_probe.json",
"sha256": "149ef26f0ad2e390498ae1ae79e8b1756781fd7555088143f5e63e34daeb8bbd"
},
{
"path": "results/records/pathmnist__resnet18_in_probe.json",
"sha256": "b7675cabaaca35d926aa55257f82c0a92284937b06caf50e410a0b6590655e99"
},
{
"path": "results/records/pneumoniamnist__resnet18_in_probe.json",
"sha256": "665edd6539224d12ba59351db40ab6c0b3c2ee642d9d4ad87664c28b2980233a"
},
{
"path": "results/records/retinamnist__resnet18_in_probe.json",
"sha256": "cd4e738933a1a1cc6113c93cc104b8a2ff25d57eba779f7fac3c9639dab5dc14"
},
{
"path": "results/records/tissuemnist__resnet18_in_probe.json",
"sha256": "2eb99a936383b66360c7cf20436277405e93dc484a159f3520339a82cfc08aad"
}
]
},
"declared": null,
"declared_note": "No vendor declaration. This is an open reference baseline implemented by NakedSignal, not a commercial product: there is no vendor to assert an intended use or to sign an attestation, so the declared block is null rather than filled in on the vendor's behalf.",
"expiry_basis": "the sealed held-out generation this evidence cycle is anchored to (gen2, set hash ad30d214daa285c5...) is published to expire 2026-09-20; per spec, a passport lapses when its generation retires or its spec version is superseded, whichever comes first",
"expiry_utc": "2026-09-20T00:00:00Z",
"issued_utc": "2026-08-03T16:03:42Z",
"issuer": {
"algo": "Ed25519",
"key_id": "ns-passport-2026-07",
"name": "NakedSignal",
"public_key_b64": "0PBCC6mjzcppR5QdPcK7vP/AamB6MP8l1T4ypMYbsMw="
},
"observed": null,
"observed_note": "Not available: this requires continuous monitoring. Production history and drift are observed fields, and nothing is deployed, so there is nothing to observe. The object is present and null on purpose. The shape is the roadmap, not a claim.",
"passport_version": "0.1",
"signature": "ohfn8V9esnTqRLylCx/01/3wfcDtuN+K3cQL/RVqi5YV+aydjQpieHTyF5nFjrjrx7oxnqdApJqOCBp4WJl0AA==",
"spec_version": "MedEval-1 v0.1",
"subject": {
"behaviour_version": "1.0.0",
"code_fingerprint": "b444c02112cc",
"code_fingerprints": [
"b444c02112cc"
],
"family": "transfer",
"model_id": "resnet18_in_probe",
"model_name": "ResNet-18 (ImageNet) frozen features + linear probe",
"params": "512-d penultimate features at 64px, logistic probe"
}
}Canonicalisation: RFC 8785 (JCS)-compatible: UTF-8, object keys sorted by code point, no insignificant whitespace, ECMAScript number formatting. every float is rounded once at emit to 6 decimal places (Python round(), banker's rounding); integers are left exact. Signed over the whole document with the `signature` member removed, canonicalised as above and encoded UTF-8. The public key for ns-passport-2026-07 is 0PBCC6mjzcppR5QdPcK7vP/AamB6MP8l1T4ypMYbsMw= — see Governance.