Back to the consoleDocumentation
MLP (256) on raw pixels
LIVEThe signed evidence record for mlp_pixels. Everything below was read out of the passport file; the signature was checked when this page was built, by the same code the Verify button runs.
SIGNATURE VALIDexit 0signature valid, issued by NakedSignal under key ns-passport-2026-07, not expired
SubjectMLP (256) on raw pixels
mlp_pixelsConfiguration1 hidden layer of 256, adam, early stopping when n>=200
Code fingerprintb444c02112cc
Spec versionMedEval-1 v0.1
Issued2026-08-03T16:03:42Z by NakedSignal
Expires2026-09-20T00:00:00Z — the sealed held-out generation this evidence cycle is anchored to (gen2, set hash ad30d214daa285c5...) is published to expire 2026-09-20; per spec, a passport lapses when its generation retires or its spec version is superseded, whichever comes first
key ns-passport-2026-07 · Ed25519
The check runs against the public key published on the governance page, using your browser's own Ed25519 implementation. The exit code shown is the exit code verify_passport.py returns for the same document.
Measured by NakedSignal
Computed in this repo by the MedEval-1 harness on data the model had never seen. Every number below was copied out of a file the harness wrote — 17 of them, each listed with its SHA-256 at the foot of this panel — and is printed exactly as it was signed, not re-rounded for display.
| Task | Domain | n test | accuracy | calibration | uncertainty | subgroup | corruption | acquisition | limited data | cost | spec sensitivity | Index |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bloodmnist | medicine | 3421 | 73.990302 | 82.533586 | 93.34369 | 54.124748 | 78.131153 | 79.534695 | 43.556288 | n/a | n/a | 73.068151 |
| breastmnist | medicine | 156 | 11.654135 | 54.795949 | 68.095123 | 0 | 68.970814 | 98.27957 | 100 | n/a | n/a | 53.320877 |
| dermamnist | medicine | 2005 | 0.391384 | 52.146775 | 82.472369 | 0 | 88.89872 | 91.186503 | 66.666667 | n/a | n/a | 49.741195 |
| octmnist | medicine | 1000 | 30 | 17.454492 | 48.132562 | 0 | 63.809524 | 64.414815 | 20.592593 | n/a | n/a | 36.861766 |
| organamnist | medicine | 17778 | 68.726978 | 30.269202 | 89.485153 | 41.389313 | 86.500527 | 93.282372 | 66.116462 | n/a | n/a | 68.455634 |
| organcmnist | medicine | 8216 | 69.970575 | 64.622649 | 91.817404 | 38.491879 | 87.296222 | 92.846464 | 66.881797 | n/a | n/a | 73.697127 |
| organsmnist | medicine | 8827 | 50.40961 | 2.932971 | 73.799774 | 9.52809 | 83.56815 | 92.141656 | 48.13376 | n/a | n/a | 53.178114 |
| pathmnist | medicine | 7180 | 30.166573 | 54.099742 | 52.318536 | 0 | 88.002195 | 92.080472 | 16.964136 | n/a | n/a | 51.254672 |
| pneumoniamnist | medicine | 624 | 52.564103 | 40.325065 | 81.667547 | 7.692308 | 76.933798 | 80.910569 | 96.585366 | n/a | n/a | 60.574525 |
| retinamnist | medicine | 400 | 11.240703 | 29.864852 | 59.921234 | 0 | 75.361618 | 75.257709 | 82.028005 | n/a | n/a | 44.110252 |
| tissuemnist | medicine | 47280 | 22.196069 | 87.290194 | 59.303506 | 0 | 77.392035 | 84.598821 | 46.847188 | n/a | n/a | 54.112916 |
Cross-site retention · organ_ct
retention is absolute retained skill (chance-corrected), per spec 2.2.1; the diagonal is same-site, every off-diagonal cell is train-on-row, test-on-column
Sitesorganamnist, organcmnist, organsmnist
Mean retention0.36125
Worst retention0.121186
Private held-out track
PUBLIC-DATA DEMONSTRATION OF THE PRIVATE PROTOCOL
| Generation | Public | Private (equated) | Gap | z | Verdict |
|---|---|---|---|---|---|
| gen1 | 77.599323 | 76.332275 | 1.267048 | 0.566464 | clear |
| gen2 | 72.526125 | 73.499084 | -0.972959 | -0.211184 | clear |
Known limitations
Subgroup metadataabsent on 11 of 11 corpora, so the subgroup score falls back to the worst diagnostic class. That absence is a disclosure, not a pass.
Training set cappedyes on at least one task — see the raw JSON for which
Observed in deploymentnothing — see the third panel
Audit chain
Bundles re-derived48 of 48
Ledger100 entries, head 6b7b371ff650a8ed…, chain intact
Re-run it yourself
python src/heldout/audit.py 1f66c7719ab3943c6fcc17 source files, with hashes
| results/cross_site/organ_ct__mlp_pixels.json | 869f0d2a4d961737b0a27288f70f9167072589a491721c277d58f12faeac5888 |
| results/heldout/audit_report.json | df460b6d91a6512dfa1e7dc650637fce2b4dd3a1ec32e03446bf6e11f9c597d2 |
| results/heldout/cycle.json | e012b8978457b51b5c31d18d14d2cc92f100731897523306acae19934f943502 |
| results/heldout/manifests/gen1.seal.json | 91a7adfa29870a0719125cdd48f6dcb149f1f6485ac1cfa104c6e00057ad9b81 |
| results/heldout/manifests/gen2.seal.json | f61470918a09cf39e074ef14fb7ade4ac3a13d36d9385f18cf9b92e4b5329547 |
| results/heldout/manifests/public.seal.json | 1a4fd930d2e781ac7163483990f15d6694e28e507d1b3ca009ff09395e47497c |
| results/records/bloodmnist__mlp_pixels.json | e14d9bdc08bfcde6cc18e93dc8fd0ee7370f59af233a9d295e77e9fff171ffef |
| results/records/breastmnist__mlp_pixels.json | 772826e38c3f9799f79a69686b4feb90e6d58ac07445fafd3ade87c43049e8f5 |
| results/records/dermamnist__mlp_pixels.json | 7be4a950f9027ce59ef71a5c886f31bc7b5d913ac9e6162277c5c320ea22d441 |
| results/records/octmnist__mlp_pixels.json | 76ec19f8902df27db6395ec1f9d2a299680210e01e52499c8201b15b796d51d8 |
| results/records/organamnist__mlp_pixels.json | d62d5c9c40d95e997a1b398505500fcee110d0fb489634e36c1f8c42202b47d8 |
| results/records/organcmnist__mlp_pixels.json | 5d0c04e85398fb67d366103492ba2fa9d00dbd8447df6b4cf0d64a5052b04093 |
| results/records/organsmnist__mlp_pixels.json | ea2be64d056b8150744fc86e760a64a3d04866ae8c08969975a0de6971d607b0 |
| results/records/pathmnist__mlp_pixels.json | baaf8a0b2375f8a85936da60790bfdd6c840c0915ea6dcc5e10e073fd17d3c97 |
| results/records/pneumoniamnist__mlp_pixels.json | c616b3eb79beef2dd991494401891920bd5c247525419d7a541878deaf287fd8 |
| results/records/retinamnist__mlp_pixels.json | 9a3851a94329f5971adfeaaa16c14e16c789a7dc14f2877520699340f982b03e |
| results/records/tissuemnist__mlp_pixels.json | d78e92ffdb7f185a4c3fca5d357cf8a48cba277fa37e532be18d772689a83cfb |
Declared by vendor
Five fields a vendor asserts about its own product — intended use, forbidden use, training cutoff, regulatory clearances, and who is personally attesting. NakedSignal never fills these in on a vendor's behalf.
No vendor declaration. This is an open reference baseline implemented by NakedSignal, not a commercial product: there is no vendor to assert an intended use or to sign an attestation, so the declared block is null rather than filled in on the vendor's behalf.
Observed in deployment
Production history and drift — how the model has actually behaved since it was deployed, on real traffic.
Not available: this requires continuous monitoring. Production history and drift are observed fields, and nothing is deployed, so there is nothing to observe. The object is present and null on purpose. The shape is the roadmap, not a claim.
Raw JSON — the whole signed document, 81,842 characters
{
"canonicalisation": {
"float_rounding": "every float is rounded once at emit to 6 decimal places (Python round(), banker's rounding); integers are left exact",
"form": "RFC 8785 (JCS)-compatible: UTF-8, object keys sorted by code point, no insignificant whitespace, ECMAScript number formatting",
"signed_over": "the whole document with the `signature` member removed, canonicalised as above and encoded UTF-8"
},
"computed": {
"audit": {
"audit_command": "python src/heldout/audit.py 1f66c7719ab3943c6fcc",
"bundles": 48,
"ledger_entries": 100,
"ledger_head": "6b7b371ff650a8edaf477490f2a5f9b043433f3ba9941989ee1962ed8f1b1d88",
"ledger_intact": true,
"rederived": 48
},
"cross_site": {
"code_fingerprint": "b444c02112cc",
"device": "mps",
"family": "organ_ct",
"matrix": {
"organamnist": {
"organamnist": {
"balanced_accuracy": 0.7157,
"chance_corrected": 0.68727,
"ece": 0.139462
},
"organcmnist": {
"balanced_accuracy": 0.369012,
"chance_corrected": 0.305913,
"ece": 0.418607,
"retention": 0.445113
},
"organsmnist": {
"balanced_accuracy": 0.166625,
"chance_corrected": 0.083288,
"ece": 0.586667,
"retention": 0.121186
}
},
"organcmnist": {
"organamnist": {
"balanced_accuracy": 0.401401,
"chance_corrected": 0.341541,
"ece": 0.340649,
"retention": 0.488121
},
"organcmnist": {
"balanced_accuracy": 0.727005,
"chance_corrected": 0.699706,
"ece": 0.070755
},
"organsmnist": {
"balanced_accuracy": 0.30037,
"chance_corrected": 0.230407,
"ece": 0.374365,
"retention": 0.329291
}
},
"organsmnist": {
"organamnist": {
"balanced_accuracy": 0.188681,
"chance_corrected": 0.10755,
"ece": 0.53204,
"retention": 0.213351
},
"organcmnist": {
"balanced_accuracy": 0.352323,
"chance_corrected": 0.287555,
"ece": 0.371878,
"retention": 0.570437
},
"organsmnist": {
"balanced_accuracy": 0.549178,
"chance_corrected": 0.504096,
"ece": 0.194134
}
}
},
"mean_retention": 0.36125,
"note": "retention is absolute retained skill (chance-corrected), per spec 2.2.1; the diagonal is same-site, every off-diagonal cell is train-on-row, test-on-column",
"score": 36.124998,
"site_labels": {
"organamnist": "axial",
"organcmnist": "coronal",
"organsmnist": "sagittal"
},
"sites": [
"organamnist",
"organcmnist",
"organsmnist"
],
"spec_version": "MedEval-1 v0.1",
"worst_retention": 0.121186
},
"evaluations": [
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 73.068151
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.801812,
"auc": 0.969099,
"balanced_accuracy": 0.772415,
"ci95": [
0.758323,
0.787642
],
"n_test": 3421,
"score": 73.990302
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.756109,
"mean_agreement": 0.737056,
"mean_kappa": 0.698281,
"mean_retention": 0.795347,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.985,
0.9667,
0.9233
],
"mean_kappa": 0.950946,
"mean_retention": 0.995698,
"retention_by_severity": [
0.9998,
1,
0.9873
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8308,
0.6667,
0.53
],
"mean_kappa": 0.61984,
"mean_retention": 0.677153,
"retention_by_severity": [
0.8902,
0.6738,
0.4675
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.6667,
0.4658,
0.3642
],
"mean_kappa": 0.433568,
"mean_retention": 0.603145,
"retention_by_severity": [
0.7924,
0.5752,
0.4418
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9642,
0.9358,
0.9117
],
"mean_kappa": 0.925942,
"mean_retention": 0.971865,
"retention_by_severity": [
0.9986,
0.9754,
0.9415
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.83,
0.5833,
0.4317
],
"mean_kappa": 0.56111,
"mean_retention": 0.728874,
"retention_by_severity": [
0.9379,
0.7185,
0.5302
],
"scored": true
}
},
"score": 79.534695,
"worst_retention": 0.441812
},
"calibration": {
"accuracy": 0.801812,
"brier": 0.277393,
"ece": 0.034933,
"mean_confidence": 0.766879,
"overconfidence": -0.034933,
"score": 82.533586
},
"corruption": {
"clean_reference_cc": 0.756109,
"mean_agreement": 0.772937,
"mean_kappa": 0.728682,
"mean_retention": 0.781312,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.7092,
0.4775,
0.3783
],
"mean_kappa": 0.430192,
"mean_retention": 0.519139,
"retention_by_severity": [
0.7794,
0.4652,
0.3128
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.5208,
0.4242,
0.2642
],
"mean_kappa": 0.285662,
"mean_retention": 0.318015,
"retention_by_severity": [
0.4859,
0.3202,
0.148
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.975,
0.93,
0.8267
],
"mean_kappa": 0.893996,
"mean_retention": 0.926994,
"retention_by_severity": [
1,
0.9711,
0.8098
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9767,
0.9517,
0.9
],
"mean_kappa": 0.932641,
"mean_retention": 0.989399,
"retention_by_severity": [
0.9959,
1,
0.9723
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9608,
0.3558,
0.8825
],
"mean_kappa": 0.676675,
"mean_retention": 0.725032,
"retention_by_severity": [
0.985,
0.2302,
0.96
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9925,
0.985,
0.9392
],
"mean_kappa": 0.967351,
"mean_retention": 0.99901,
"retention_by_severity": [
1,
1,
0.997
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9533,
0.9417,
0.8867
],
"mean_kappa": 0.914259,
"mean_retention": 0.991591,
"retention_by_severity": [
1,
1,
0.9748
],
"scored": true
}
},
"score": 78.131153,
"worst_retention": 0.148014
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.739903,
"mean_retention": 0.435563,
"per_budget": {
"n100": {
"chance_corrected": 0.471691,
"n_labels": 800,
"retention": 0.637504
},
"n20": {
"chance_corrected": 0,
"n_labels": 160,
"retention": 0
},
"n5": {
"chance_corrected": 0.495132,
"n_labels": 40,
"retention": 0.669184
}
},
"score": 43.556288
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 54.124748,
"worst_class_recall": 0.598592
},
"uncertainty": {
"full_accuracy": 0.801812,
"risk_coverage_auc": 0.941757,
"score": 93.34369,
"selective_acc_at_50": 0.973684,
"selective_acc_at_80": 0.882353
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 11959,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Microscopy",
"n_classes": 8,
"n_test": 3421,
"runtime_s": 8.8,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "bloodmnist",
"task_name": "Peripheral blood cells (8-class)",
"timestamp": "2026-08-01T19:18:52+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 53.320877
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.75,
"auc": 0.71721,
"balanced_accuracy": 0.558271,
"ci95": [
0.509907,
0.629035
],
"n_test": 156,
"score": 11.654135
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.116541,
"mean_agreement": 0.932906,
"mean_kappa": 0.679063,
"mean_retention": 0.982796,
"n_cases": 156,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9936,
0.9744,
0.9872
],
"mean_kappa": 0.856775,
"mean_retention": 0.956989,
"retention_by_severity": [
1,
1,
0.871
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.9872,
0.9808,
0.9679
],
"mean_kappa": 0.794826,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9103,
0.8654,
0.7949
],
"mean_kappa": 0.408382,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
1,
0.9872,
0.9936
],
"mean_kappa": 0.935384,
"mean_retention": 0.956989,
"retention_by_severity": [
1,
0.871,
1
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9167,
0.8462,
0.7885
],
"mean_kappa": 0.399948,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
}
},
"score": 98.27957,
"worst_retention": 0.870968
},
"calibration": {
"accuracy": 0.75,
"brier": 0.350305,
"ece": 0.090408,
"mean_confidence": 0.70023,
"overconfidence": -0.04977,
"score": 54.795949
},
"corruption": {
"clean_reference_cc": 0.116541,
"mean_agreement": 0.975275,
"mean_kappa": 0.642147,
"mean_retention": 0.689708,
"n_cases": 156,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9423,
0.9423,
0.9423
],
"mean_kappa": 0,
"mean_retention": 0,
"retention_by_severity": [
0,
0,
0
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.9551,
0.9423,
0.9423
],
"mean_kappa": 0.116667,
"mean_retention": 0.136201,
"retention_by_severity": [
0.4086,
0,
0
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
1,
1,
0.9936
],
"mean_kappa": 0.979266,
"mean_retention": 0.9319,
"retention_by_severity": [
1,
1,
0.7957
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9808,
0.9679,
0.9679
],
"mean_kappa": 0.688387,
"mean_retention": 0.827957,
"retention_by_severity": [
1,
0.9462,
0.5376
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
1,
0.9808,
1
],
"mean_kappa": 0.943966,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
1,
0.9936,
0.9808
],
"mean_kappa": 0.928286,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9744,
0.9808,
0.9936
],
"mean_kappa": 0.838456,
"mean_retention": 0.9319,
"retention_by_severity": [
1,
0.7957,
1
],
"scored": true
}
},
"score": 68.970814,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.116541,
"mean_retention": 1,
"per_budget": {
"n100": {
"chance_corrected": 0.274436,
"n_labels": 200,
"retention": 1
},
"n20": {
"chance_corrected": 0.180451,
"n_labels": 40,
"retention": 1
},
"n5": {
"chance_corrected": 0.339599,
"n_labels": 10,
"retention": 1
}
},
"score": 100
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0.142857
},
"uncertainty": {
"full_accuracy": 0.75,
"risk_coverage_auc": 0.840476,
"score": 68.095123,
"selective_acc_at_50": 0.846154,
"selective_acc_at_80": 0.8
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 546,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Ultrasound",
"n_classes": 2,
"n_test": 156,
"runtime_s": 0.3,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "breastmnist",
"task_name": "Breast ultrasound (malignancy)",
"timestamp": "2026-08-01T19:12:57+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 49.741195
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.669327,
"auc": 0.705258,
"balanced_accuracy": 0.146212,
"ci95": [
0.143777,
0.149052
],
"n_test": 2005,
"score": 0.391384
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.005997,
"mean_agreement": 0.997778,
"mean_kappa": 0.857108,
"mean_retention": 0.911865,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
1,
0.9975,
0.9958
],
"mean_kappa": 0.864231,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.9975,
0.9942,
0.9942
],
"mean_kappa": 0.65403,
"mean_retention": 0.593721,
"retention_by_severity": [
1,
0.3906,
0.3906
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9992,
0.9958,
0.995
],
"mean_kappa": 0.820211,
"mean_retention": 0.965604,
"retention_by_severity": [
1,
0.9656,
0.9312
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
1,
1,
1
],
"mean_kappa": 1,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9992,
1,
0.9983
],
"mean_kappa": 0.947067,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
}
},
"score": 91.186503,
"worst_retention": 0.390582
},
"calibration": {
"accuracy": 0.669327,
"brier": 0.475724,
"ece": 0.095706,
"mean_confidence": 0.620736,
"overconfidence": -0.04859,
"score": 52.146775
},
"corruption": {
"clean_reference_cc": 0.005997,
"mean_agreement": 0.990317,
"mean_kappa": 0.790337,
"mean_retention": 0.888987,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9975,
0.9975,
0.9975
],
"mean_kappa": 0.832371,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.9975,
0.9967,
0.9958
],
"mean_kappa": 0.745614,
"mean_retention": 0.575022,
"retention_by_severity": [
0.8047,
0.575,
0.3453
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
1,
1,
0.9992
],
"mean_kappa": 0.983228,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9958,
0.9958,
0.9958
],
"mean_kappa": 0.759205,
"mean_retention": 0.83541,
"retention_by_severity": [
0.7703,
0.7359,
1
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9983,
0.8525,
0.9975
],
"mean_kappa": 0.604985,
"mean_retention": 0.988535,
"retention_by_severity": [
1,
1,
0.9656
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9992,
0.9992,
0.9967
],
"mean_kappa": 0.890214,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9975,
0.9925,
0.9942
],
"mean_kappa": 0.716741,
"mean_retention": 0.823944,
"retention_by_severity": [
0.9312,
1,
0.5406
],
"scored": true
}
},
"score": 88.89872,
"worst_retention": 0.345334
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.003914,
"mean_retention": 0.666667,
"per_budget": {
"n100": {
"chance_corrected": 0,
"n_labels": 679,
"retention": 0
},
"n20": {
"chance_corrected": 0.212286,
"n_labels": 140,
"retention": 1
},
"n5": {
"chance_corrected": 0.199683,
"n_labels": 35,
"retention": 1
}
},
"score": 66.666667
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0
},
"uncertainty": {
"full_accuracy": 0.669327,
"risk_coverage_auc": 0.849763,
"score": 82.472369,
"selective_acc_at_50": 0.867265,
"selective_acc_at_80": 0.724439
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 7007,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Dermatoscopy",
"n_classes": 7,
"n_test": 2005,
"runtime_s": 3.6,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "dermamnist",
"task_name": "Dermatoscopy (7-class skin lesion)",
"timestamp": "2026-08-01T19:17:18+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 36.861766
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.475,
"auc": 0.7913,
"balanced_accuracy": 0.475,
"ci95": [
0.457097,
0.497986
],
"n_test": 1000,
"score": 30
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.3,
"mean_agreement": 0.738067,
"mean_kappa": 0.520142,
"mean_retention": 0.644148,
"n_cases": 1000,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.94,
0.875,
0.739
],
"mean_kappa": 0.741177,
"mean_retention": 0.837037,
"retention_by_severity": [
0.9778,
0.8533,
0.68
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.842,
0.729,
0.55
],
"mean_kappa": 0.504927,
"mean_retention": 0.757037,
"retention_by_severity": [
0.9467,
0.8844,
0.44
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.795,
0.68,
0.609
],
"mean_kappa": 0.403753,
"mean_retention": 0.512593,
"retention_by_severity": [
0.7644,
0.4933,
0.28
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.843,
0.729,
0.727
],
"mean_kappa": 0.598458,
"mean_retention": 0.642963,
"retention_by_severity": [
0.8044,
0.5511,
0.5733
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.767,
0.647,
0.599
],
"mean_kappa": 0.352397,
"mean_retention": 0.471111,
"retention_by_severity": [
0.7378,
0.4044,
0.2711
],
"scored": true
}
},
"score": 64.414815,
"worst_retention": 0.271111
},
"calibration": {
"accuracy": 0.475,
"brier": 0.692227,
"ece": 0.165091,
"mean_confidence": 0.640091,
"overconfidence": 0.165091,
"score": 17.454492
},
"corruption": {
"clean_reference_cc": 0.3,
"mean_agreement": 0.701714,
"mean_kappa": 0.512175,
"mean_retention": 0.638095,
"n_cases": 1000,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.551,
0.454,
0.424
],
"mean_kappa": 0.158017,
"mean_retention": 0.235556,
"retention_by_severity": [
0.3956,
0.1733,
0.1378
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.665,
0.468,
0.411
],
"mean_kappa": 0.225338,
"mean_retention": 0.376296,
"retention_by_severity": [
0.6844,
0.3067,
0.1378
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.898,
0.741,
0.594
],
"mean_kappa": 0.566186,
"mean_retention": 0.602963,
"retention_by_severity": [
0.9067,
0.6044,
0.2978
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.903,
0.809,
0.665
],
"mean_kappa": 0.642617,
"mean_retention": 0.823704,
"retention_by_severity": [
0.8978,
0.8489,
0.7244
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.877,
0.448,
0.631
],
"mean_kappa": 0.468473,
"mean_retention": 0.659259,
"retention_by_severity": [
0.8933,
0.6311,
0.4533
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.964,
0.926,
0.788
],
"mean_kappa": 0.808769,
"mean_retention": 0.912593,
"retention_by_severity": [
0.9733,
0.9378,
0.8267
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.911,
0.847,
0.761
],
"mean_kappa": 0.715826,
"mean_retention": 0.856296,
"retention_by_severity": [
0.9467,
0.84,
0.7822
],
"scored": true
}
},
"score": 63.809524,
"worst_retention": 0.137778
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.3,
"mean_retention": 0.205926,
"per_budget": {
"n100": {
"chance_corrected": 0.057333,
"n_labels": 400,
"retention": 0.191111
},
"n20": {
"chance_corrected": 0.128,
"n_labels": 80,
"retention": 0.426667
},
"n5": {
"chance_corrected": 0,
"n_labels": 20,
"retention": 0
}
},
"score": 20.592593
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0
},
"uncertainty": {
"full_accuracy": 0.475,
"risk_coverage_auc": 0.610994,
"score": 48.132562,
"selective_acc_at_50": 0.588,
"selective_acc_at_80": 0.51125
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 97477,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "OCT",
"n_classes": 4,
"n_test": 1000,
"runtime_s": 3.2,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "octmnist",
"task_name": "Retinal OCT (4-class)",
"timestamp": "2026-08-01T19:35:36+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 68.455634
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.720047,
"auc": 0.952208,
"balanced_accuracy": 0.7157,
"ci95": [
0.709042,
0.722787
],
"n_test": 17778,
"score": 68.726978
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.661566,
"mean_agreement": 0.836722,
"mean_kappa": 0.817456,
"mean_retention": 0.932824,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9792,
0.9408,
0.8792
],
"mean_kappa": 0.925098,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.7867,
0.5967,
0.4208
],
"mean_kappa": 0.555001,
"mean_retention": 0.736308,
"retention_by_severity": [
0.925,
0.7525,
0.5313
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.92,
0.8525,
0.8025
],
"mean_kappa": 0.841413,
"mean_retention": 0.952312,
"retention_by_severity": [
0.9929,
0.9569,
0.9071
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9092,
0.8992,
0.8758
],
"mean_kappa": 0.882128,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9575,
0.8958,
0.835
],
"mean_kappa": 0.883638,
"mean_retention": 0.975499,
"retention_by_severity": [
1,
0.9798,
0.9467
],
"scored": true
}
},
"score": 93.282372,
"worst_retention": 0.531345
},
"calibration": {
"accuracy": 0.720047,
"brier": 0.414812,
"ece": 0.139462,
"mean_confidence": 0.859509,
"overconfidence": 0.139462,
"score": 30.269202
},
"corruption": {
"clean_reference_cc": 0.661566,
"mean_agreement": 0.801825,
"mean_kappa": 0.77912,
"mean_retention": 0.865005,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.8875,
0.7483,
0.4775
],
"mean_kappa": 0.670499,
"mean_retention": 0.747608,
"retention_by_severity": [
0.9619,
0.8222,
0.4588
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.805,
0.5308,
0.24
],
"mean_kappa": 0.477256,
"mean_retention": 0.558927,
"retention_by_severity": [
0.8853,
0.5867,
0.2049
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9483,
0.9042,
0.8533
],
"mean_kappa": 0.890345,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.955,
0.9142,
0.8233
],
"mean_kappa": 0.885341,
"mean_retention": 0.979002,
"retention_by_severity": [
1,
0.9983,
0.9387
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9208,
0.4108,
0.8167
],
"mean_kappa": 0.678733,
"mean_retention": 0.794796,
"retention_by_severity": [
1,
0.3844,
1
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9867,
0.9833,
0.955
],
"mean_kappa": 0.972015,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9392,
0.8975,
0.8408
],
"mean_kappa": 0.879651,
"mean_retention": 0.974704,
"retention_by_severity": [
1,
0.9989,
0.9252
],
"scored": true
}
},
"score": 86.500527,
"worst_retention": 0.204851
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.68727,
"mean_retention": 0.661165,
"per_budget": {
"n100": {
"chance_corrected": 0.563699,
"n_labels": 1100,
"retention": 0.820201
},
"n20": {
"chance_corrected": 0.40183,
"n_labels": 220,
"retention": 0.584675
},
"n5": {
"chance_corrected": 0.397666,
"n_labels": 55,
"retention": 0.578617
}
},
"score": 66.116462
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 41.389313,
"worst_class_recall": 0.467176
},
"uncertainty": {
"full_accuracy": 0.720047,
"risk_coverage_auc": 0.90441,
"score": 89.485153,
"selective_acc_at_50": 0.944988,
"selective_acc_at_80": 0.813106
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 34561,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 17778,
"runtime_s": 5.3,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organamnist",
"task_name": "Abdominal CT organs — axial (A)",
"timestamp": "2026-08-01T19:32:54+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 73.697127
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.755599,
"auc": 0.956537,
"balanced_accuracy": 0.727005,
"ci95": [
0.716058,
0.736978
],
"n_test": 8216,
"score": 69.970575
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.683963,
"mean_agreement": 0.859722,
"mean_kappa": 0.841022,
"mean_retention": 0.928465,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9792,
0.9608,
0.9208
],
"mean_kappa": 0.947244,
"mean_retention": 0.996327,
"retention_by_severity": [
1,
1,
0.989
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.81,
0.6558,
0.4658
],
"mean_kappa": 0.59764,
"mean_retention": 0.736928,
"retention_by_severity": [
0.9078,
0.7527,
0.5503
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9242,
0.8617,
0.8025
],
"mean_kappa": 0.844079,
"mean_retention": 0.946746,
"retention_by_severity": [
0.983,
0.9574,
0.8998
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9558,
0.9258,
0.92
],
"mean_kappa": 0.924814,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9583,
0.9075,
0.8475
],
"mean_kappa": 0.891333,
"mean_retention": 0.962323,
"retention_by_severity": [
0.9794,
0.969,
0.9386
],
"scored": true
}
},
"score": 92.846464,
"worst_retention": 0.550287
},
"calibration": {
"accuracy": 0.755599,
"brier": 0.347083,
"ece": 0.070755,
"mean_confidence": 0.826354,
"overconfidence": 0.070755,
"score": 64.622649
},
"corruption": {
"clean_reference_cc": 0.683963,
"mean_agreement": 0.837341,
"mean_kappa": 0.815001,
"mean_retention": 0.872962,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.91,
0.8133,
0.66
],
"mean_kappa": 0.765532,
"mean_retention": 0.814176,
"retention_by_severity": [
0.9461,
0.8616,
0.6349
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8542,
0.6658,
0.345
],
"mean_kappa": 0.573414,
"mean_retention": 0.628451,
"retention_by_severity": [
0.9164,
0.6867,
0.2822
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9667,
0.9317,
0.8833
],
"mean_kappa": 0.917322,
"mean_retention": 0.987973,
"retention_by_severity": [
1,
1,
0.9639
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9708,
0.9258,
0.8558
],
"mean_kappa": 0.906087,
"mean_retention": 0.965121,
"retention_by_severity": [
0.9983,
0.9841,
0.9129
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9433,
0.3242,
0.8742
],
"mean_kappa": 0.671616,
"mean_retention": 0.759513,
"retention_by_severity": [
0.999,
0.3046,
0.975
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9942,
0.9858,
0.9642
],
"mean_kappa": 0.978829,
"mean_retention": 0.993067,
"retention_by_severity": [
0.9996,
0.9956,
0.984
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.945,
0.9142,
0.8567
],
"mean_kappa": 0.89221,
"mean_retention": 0.962435,
"retention_by_severity": [
0.9882,
0.9578,
0.9413
],
"scored": true
}
},
"score": 87.296222,
"worst_retention": 0.282234
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.699706,
"mean_retention": 0.668818,
"per_budget": {
"n100": {
"chance_corrected": 0.560797,
"n_labels": 1100,
"retention": 0.801476
},
"n20": {
"chance_corrected": 0.500415,
"n_labels": 220,
"retention": 0.715179
},
"n5": {
"chance_corrected": 0.342715,
"n_labels": 55,
"retention": 0.489799
}
},
"score": 66.881797
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 38.491879,
"worst_class_recall": 0.440835
},
"uncertainty": {
"full_accuracy": 0.755599,
"risk_coverage_auc": 0.925613,
"score": 91.817404,
"selective_acc_at_50": 0.961295,
"selective_acc_at_80": 0.851818
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 12975,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 8216,
"runtime_s": 5.7,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organcmnist",
"task_name": "Abdominal CT organs — coronal (C)",
"timestamp": "2026-08-01T19:27:56+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 53.178114
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.578113,
"auc": 0.911655,
"balanced_accuracy": 0.549178,
"ci95": [
0.536972,
0.558425
],
"n_test": 8827,
"score": 50.40961
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.530145,
"mean_agreement": 0.813778,
"mean_kappa": 0.786836,
"mean_retention": 0.921417,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9725,
0.9258,
0.8567
],
"mean_kappa": 0.906055,
"mean_retention": 0.978278,
"retention_by_severity": [
0.988,
0.9828,
0.9641
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.7583,
0.6142,
0.4358
],
"mean_kappa": 0.545062,
"mean_retention": 0.768014,
"retention_by_severity": [
0.9287,
0.7977,
0.5776
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8967,
0.8008,
0.7267
],
"mean_kappa": 0.781411,
"mean_retention": 0.913457,
"retention_by_severity": [
0.9632,
0.9203,
0.8569
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9092,
0.8942,
0.8508
],
"mean_kappa": 0.867593,
"mean_retention": 0.979041,
"retention_by_severity": [
0.9838,
0.9871,
0.9662
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9325,
0.86,
0.7725
],
"mean_kappa": 0.834062,
"mean_retention": 0.968292,
"retention_by_severity": [
0.9928,
0.9769,
0.9351
],
"scored": true
}
},
"score": 92.141656,
"worst_retention": 0.577611
},
"calibration": {
"accuracy": 0.578113,
"brier": 0.611665,
"ece": 0.194134,
"mean_confidence": 0.772247,
"overconfidence": 0.194134,
"score": 2.932971
},
"corruption": {
"clean_reference_cc": 0.530145,
"mean_agreement": 0.785159,
"mean_kappa": 0.753375,
"mean_retention": 0.835681,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.8608,
0.7342,
0.5575
],
"mean_kappa": 0.673754,
"mean_retention": 0.729381,
"retention_by_severity": [
0.8939,
0.7521,
0.5421
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.7775,
0.5208,
0.2867
],
"mean_kappa": 0.464007,
"mean_retention": 0.54015,
"retention_by_severity": [
0.8286,
0.5647,
0.2271
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9408,
0.8792,
0.8075
],
"mean_kappa": 0.857203,
"mean_retention": 0.971562,
"retention_by_severity": [
0.9891,
0.9791,
0.9465
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9475,
0.8925,
0.7583
],
"mean_kappa": 0.845999,
"mean_retention": 0.914054,
"retention_by_severity": [
0.9733,
0.9252,
0.8437
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9092,
0.3067,
0.795
],
"mean_kappa": 0.619041,
"mean_retention": 0.74351,
"retention_by_severity": [
0.9818,
0.3092,
0.9395
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9867,
0.9742,
0.9542
],
"mean_kappa": 0.967439,
"mean_retention": 0.994532,
"retention_by_severity": [
0.9926,
1,
0.991
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9192,
0.8767,
0.8033
],
"mean_kappa": 0.846183,
"mean_retention": 0.956581,
"retention_by_severity": [
0.9813,
0.9618,
0.9266
],
"scored": true
}
},
"score": 83.56815,
"worst_retention": 0.227132
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.504096,
"mean_retention": 0.481338,
"per_budget": {
"n100": {
"chance_corrected": 0.346637,
"n_labels": 1100,
"retention": 0.687641
},
"n20": {
"chance_corrected": 0.190689,
"n_labels": 220,
"retention": 0.378279
},
"n5": {
"chance_corrected": 0.190595,
"n_labels": 55,
"retention": 0.378092
}
},
"score": 48.13376
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 9.52809,
"worst_class_recall": 0.177528
},
"uncertainty": {
"full_accuracy": 0.578113,
"risk_coverage_auc": 0.761816,
"score": 73.799774,
"selective_acc_at_50": 0.754644,
"selective_acc_at_80": 0.644293
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 13932,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 8827,
"runtime_s": 7.1,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organsmnist",
"task_name": "Abdominal CT organs — sagittal (S)",
"timestamp": "2026-08-01T19:30:37+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 51.254672
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.454596,
"auc": 0.784423,
"balanced_accuracy": 0.379258,
"ci95": [
0.369867,
0.389158
],
"n_test": 7180,
"score": 30.166573
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.316474,
"mean_agreement": 0.750833,
"mean_kappa": 0.714905,
"mean_retention": 0.920805,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9783,
0.9475,
0.9317
],
"mean_kappa": 0.944339,
"mean_retention": 0.96933,
"retention_by_severity": [
1,
0.9775,
0.9305
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.93,
0.8875,
0.8292
],
"mean_kappa": 0.861944,
"mean_retention": 0.941491,
"retention_by_severity": [
0.9596,
0.949,
0.9159
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.54,
0.3408,
0.2233
],
"mean_kappa": 0.283699,
"mean_retention": 0.817267,
"retention_by_severity": [
0.9018,
0.8578,
0.6922
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.96,
0.9542,
0.94
],
"mean_kappa": 0.943002,
"mean_retention": 0.992805,
"retention_by_severity": [
0.9938,
0.9851,
0.9995
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.8083,
0.5517,
0.44
],
"mean_kappa": 0.541541,
"mean_retention": 0.883132,
"retention_by_severity": [
0.8801,
0.9378,
0.8315
],
"scored": true
}
},
"score": 92.080472,
"worst_retention": 0.692166
},
"calibration": {
"accuracy": 0.454596,
"brier": 0.707651,
"ece": 0.091801,
"mean_confidence": 0.4443,
"overconfidence": -0.010296,
"score": 54.099742
},
"corruption": {
"clean_reference_cc": 0.316474,
"mean_agreement": 0.822103,
"mean_kappa": 0.792881,
"mean_retention": 0.880022,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.6175,
0.3783,
0.185
],
"mean_kappa": 0.288281,
"mean_retention": 0.573525,
"retention_by_severity": [
0.7828,
0.6368,
0.301
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.9575,
0.9317,
0.91
],
"mean_kappa": 0.921476,
"mean_retention": 0.985271,
"retention_by_severity": [
0.9894,
0.9845,
0.9819
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9667,
0.9508,
0.9367
],
"mean_kappa": 0.942989,
"mean_retention": 0.991841,
"retention_by_severity": [
0.9889,
0.9941,
0.9925
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9617,
0.9292,
0.8383
],
"mean_kappa": 0.894123,
"mean_retention": 0.970778,
"retention_by_severity": [
1,
0.972,
0.9404
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9625,
0.4708,
0.9067
],
"mean_kappa": 0.753502,
"mean_retention": 0.764243,
"retention_by_severity": [
1,
0.2976,
0.9951
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.985,
0.9625,
0.8367
],
"mean_kappa": 0.915711,
"mean_retention": 0.940616,
"retention_by_severity": [
1,
1,
0.8218
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9225,
0.8758,
0.7783
],
"mean_kappa": 0.834086,
"mean_retention": 0.933881,
"retention_by_severity": [
0.9724,
0.9159,
0.9133
],
"scored": true
}
},
"score": 88.002195,
"worst_retention": 0.297637
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.301666,
"mean_retention": 0.169641,
"per_budget": {
"n100": {
"chance_corrected": 0.153066,
"n_labels": 900,
"retention": 0.507402
},
"n20": {
"chance_corrected": 0.000366,
"n_labels": 180,
"retention": 0.001213
},
"n5": {
"chance_corrected": 0.000093,
"n_labels": 45,
"retention": 0.00031
}
},
"score": 16.964136
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0.062078
},
"uncertainty": {
"full_accuracy": 0.454596,
"risk_coverage_auc": 0.576165,
"score": 52.318536,
"selective_acc_at_50": 0.574095,
"selective_acc_at_80": 0.506267
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 89996,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "Histopathology",
"n_classes": 9,
"n_test": 7180,
"runtime_s": 10.1,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "pathmnist",
"task_name": "Colorectal pathology (9-class tissue)",
"timestamp": "2026-08-01T19:42:00+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 60.574525
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.81891,
"auc": 0.913774,
"balanced_accuracy": 0.762821,
"ci95": [
0.737545,
0.79718
],
"n_test": 624,
"score": 52.564103
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.525641,
"mean_agreement": 0.932265,
"mean_kappa": 0.75533,
"mean_retention": 0.809106,
"n_cases": 624,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9968,
0.9824,
0.9663
],
"mean_kappa": 0.943113,
"mean_retention": 0.95664,
"retention_by_severity": [
1,
0.9984,
0.8715
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8926,
0.8141,
0.7917
],
"mean_kappa": 0.261055,
"mean_retention": 0.216802,
"retention_by_severity": [
0.5203,
0.122,
0.0081
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9647,
0.9247,
0.8846
],
"mean_kappa": 0.803278,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9776,
0.9647,
0.9519
],
"mean_kappa": 0.886716,
"mean_retention": 0.872087,
"retention_by_severity": [
0.9285,
0.8634,
0.8244
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9856,
0.9583,
0.9279
],
"mean_kappa": 0.882489,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
}
},
"score": 80.910569,
"worst_retention": 0.00813
},
"calibration": {
"accuracy": 0.81891,
"brier": 0.297466,
"ece": 0.11935,
"mean_confidence": 0.93237,
"overconfidence": 0.113459,
"score": 40.325065
},
"corruption": {
"clean_reference_cc": 0.525641,
"mean_agreement": 0.939789,
"mean_kappa": 0.767602,
"mean_retention": 0.769338,
"n_cases": 624,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9583,
0.9054,
0.8349
],
"mean_kappa": 0.607835,
"mean_retention": 0.546883,
"retention_by_severity": [
0.8276,
0.5854,
0.2276
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.9038,
0.8221,
0.7901
],
"mean_kappa": 0.291053,
"mean_retention": 0.246612,
"retention_by_severity": [
0.5772,
0.1626,
0
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9824,
0.9599,
0.8734
],
"mean_kappa": 0.776572,
"mean_retention": 0.743631,
"retention_by_severity": [
0.9398,
0.852,
0.439
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9952,
0.9856,
0.9663
],
"mean_kappa": 0.946572,
"mean_retention": 0.993496,
"retention_by_severity": [
1,
0.9919,
0.9886
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9647,
0.9503,
0.9439
],
"mean_kappa": 0.853588,
"mean_retention": 0.889973,
"retention_by_severity": [
0.8602,
0.8098,
1
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9952,
0.9952,
0.9872
],
"mean_kappa": 0.977409,
"mean_retention": 0.987534,
"retention_by_severity": [
1,
0.9919,
0.9707
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.984,
0.976,
0.9615
],
"mean_kappa": 0.920184,
"mean_retention": 0.977236,
"retention_by_severity": [
0.9967,
1,
0.935
],
"scored": true
}
},
"score": 76.933798,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.525641,
"mean_retention": 0.965854,
"per_budget": {
"n100": {
"chance_corrected": 0.557265,
"n_labels": 200,
"retention": 1
},
"n20": {
"chance_corrected": 0.676068,
"n_labels": 40,
"retention": 1
},
"n5": {
"chance_corrected": 0.471795,
"n_labels": 10,
"retention": 0.897561
}
},
"score": 96.585366
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 7.692308,
"worst_class_recall": 0.538462
},
"uncertainty": {
"full_accuracy": 0.81891,
"risk_coverage_auc": 0.908338,
"score": 81.667547,
"selective_acc_at_50": 0.935897,
"selective_acc_at_80": 0.867735
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 4708,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Chest X-ray",
"n_classes": 2,
"n_test": 624,
"runtime_s": 1.6,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "pneumoniamnist",
"task_name": "Paediatric chest X-ray (pneumonia)",
"timestamp": "2026-08-01T19:16:20+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 44.110252
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.4825,
"auc": 0.70677,
"balanced_accuracy": 0.289926,
"ci95": [
0.265703,
0.316194
],
"n_test": 400,
"score": 11.240703
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.112407,
"mean_agreement": 0.799,
"mean_kappa": 0.539266,
"mean_retention": 0.752577,
"n_cases": 400,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9775,
0.955,
0.9
],
"mean_kappa": 0.860764,
"mean_retention": 0.929323,
"retention_by_severity": [
1,
0.9602,
0.8278
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.7,
0.6975,
0.6625
],
"mean_kappa": 0.018536,
"mean_retention": 0.042209,
"retention_by_severity": [
0.0327,
0,
0.0939
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8725,
0.75,
0.6675
],
"mean_kappa": 0.567679,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9475,
0.915,
0.8875
],
"mean_kappa": 0.787519,
"mean_retention": 0.791353,
"retention_by_severity": [
0.8165,
0.7695,
0.788
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.8125,
0.6525,
0.5875
],
"mean_kappa": 0.461833,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
}
},
"score": 75.257709,
"worst_retention": 0
},
"calibration": {
"accuracy": 0.4825,
"brier": 0.629846,
"ece": 0.14027,
"mean_confidence": 0.426315,
"overconfidence": -0.056185,
"score": 29.864852
},
"corruption": {
"clean_reference_cc": 0.112407,
"mean_agreement": 0.862976,
"mean_kappa": 0.624887,
"mean_retention": 0.753616,
"n_cases": 400,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.845,
0.745,
0.7075
],
"mean_kappa": 0.274314,
"mean_retention": 0.358396,
"retention_by_severity": [
0.6955,
0.3143,
0.0654
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.85,
0.7625,
0.7175
],
"mean_kappa": 0.329936,
"mean_retention": 0.498246,
"retention_by_severity": [
0.7154,
0.4977,
0.2816
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9375,
0.8425,
0.7425
],
"mean_kappa": 0.534358,
"mean_retention": 0.613033,
"retention_by_severity": [
0.8293,
0.6955,
0.3143
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.97,
0.925,
0.87
],
"mean_kappa": 0.811302,
"mean_retention": 0.904261,
"retention_by_severity": [
0.9872,
1,
0.7256
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9075,
0.725,
0.895
],
"mean_kappa": 0.669615,
"mean_retention": 0.901378,
"retention_by_severity": [
0.7951,
1,
0.909
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9825,
0.99,
0.92
],
"mean_kappa": 0.919543,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.965,
0.9375,
0.885
],
"mean_kappa": 0.835139,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
}
},
"score": 75.361618,
"worst_retention": 0.065414
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.112407,
"mean_retention": 0.82028,
"per_budget": {
"n100": {
"chance_corrected": 0.178438,
"n_labels": 466,
"retention": 1
},
"n20": {
"chance_corrected": 0.21132,
"n_labels": 100,
"retention": 1
},
"n5": {
"chance_corrected": 0.051802,
"n_labels": 25,
"retention": 0.46084
}
},
"score": 82.028005
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0
},
"uncertainty": {
"full_accuracy": 0.4825,
"risk_coverage_auc": 0.67937,
"score": 59.921234,
"selective_acc_at_50": 0.72,
"selective_acc_at_80": 0.55625
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 1080,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Fundus",
"n_classes": 5,
"n_test": 400,
"runtime_s": 1.4,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "retinamnist",
"task_name": "Fundus photography (DR grade)",
"timestamp": "2026-08-01T19:13:14+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 54.112916
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.500952,
"auc": 0.798994,
"balanced_accuracy": 0.319216,
"ci95": [
0.315098,
0.323094
],
"n_test": 47280,
"score": 22.196069
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.202348,
"mean_agreement": 0.631556,
"mean_kappa": 0.550479,
"mean_retention": 0.845988,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9392,
0.8858,
0.755
],
"mean_kappa": 0.79777,
"mean_retention": 0.962995,
"retention_by_severity": [
1,
0.9947,
0.8943
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8542,
0.7508,
0.6267
],
"mean_kappa": 0.632711,
"mean_retention": 0.944307,
"retention_by_severity": [
1,
0.9543,
0.8786
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.595,
0.3025,
0.1733
],
"mean_kappa": 0.249669,
"mean_retention": 0.640765,
"retention_by_severity": [
0.9038,
0.6135,
0.405
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9317,
0.91,
0.8717
],
"mean_kappa": 0.863529,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.435,
0.2492,
0.1933
],
"mean_kappa": 0.208718,
"mean_retention": 0.681875,
"retention_by_severity": [
0.8666,
0.6781,
0.5009
],
"scored": true
}
},
"score": 84.598821,
"worst_retention": 0.40502
},
"calibration": {
"accuracy": 0.500952,
"brier": 0.64555,
"ece": 0.02542,
"mean_confidence": 0.526371,
"overconfidence": 0.02542,
"score": 87.290194
},
"corruption": {
"clean_reference_cc": 0.202348,
"mean_agreement": 0.731786,
"mean_kappa": 0.598,
"mean_retention": 0.77392,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.5333,
0.4792,
0.4633
],
"mean_kappa": 0.175774,
"mean_retention": 0.325818,
"retention_by_severity": [
0.4539,
0.2927,
0.2308
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.7492,
0.6158,
0.505
],
"mean_kappa": 0.393281,
"mean_retention": 0.456113,
"retention_by_severity": [
0.7864,
0.4395,
0.1425
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9517,
0.8917,
0.7792
],
"mean_kappa": 0.815747,
"mean_retention": 0.951523,
"retention_by_severity": [
1,
1,
0.8546
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.8892,
0.7733,
0.5767
],
"mean_kappa": 0.620497,
"mean_retention": 0.83956,
"retention_by_severity": [
1,
0.9106,
0.6081
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9283,
0.5483,
0.7925
],
"mean_kappa": 0.664697,
"mean_retention": 0.901001,
"retention_by_severity": [
1,
0.7488,
0.9542
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.945,
0.8767,
0.535
],
"mean_kappa": 0.729464,
"mean_retention": 0.943428,
"retention_by_severity": [
1,
1,
0.8303
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9083,
0.8608,
0.765
],
"mean_kappa": 0.786537,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
}
},
"score": 77.392035,
"worst_retention": 0.142494
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.221961,
"mean_retention": 0.468472,
"per_budget": {
"n100": {
"chance_corrected": 0.113911,
"n_labels": 800,
"retention": 0.513203
},
"n20": {
"chance_corrected": 0.114884,
"n_labels": 160,
"retention": 0.517586
},
"n5": {
"chance_corrected": 0.083153,
"n_labels": 40,
"retention": 0.374628
}
},
"score": 46.847188
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0.002239
},
"uncertainty": {
"full_accuracy": 0.500952,
"risk_coverage_auc": 0.643906,
"score": 59.303506,
"selective_acc_at_50": 0.638156,
"selective_acc_at_80": 0.554621
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 165466,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "Microscopy",
"n_classes": 8,
"n_test": 47280,
"runtime_s": 4.1,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "tissuemnist",
"task_name": "Kidney cortex cells (8-class)",
"timestamp": "2026-08-01T19:38:27+00:00"
}
],
"heldout": {
"entries": [
{
"direction": "no gap beyond sampling error",
"equating_offset": 0,
"gap": 1.267048,
"gap_se": 2.236768,
"generation": "gen1",
"merkle_root": "61c508cbafed177bb1553f05aa2cc80d4157adc1bc1ff611d88581aab9ca61c5",
"model": "mlp_pixels",
"score_private_equated": 76.332275,
"score_private_raw": 76.332275,
"score_public": 77.599323,
"set_hash": "641ab45baeaf445a1057c060eb9b35205bd55fcde4e629b2b88cfa38fba30047",
"submitter": "Submitter C",
"verdict": "clear",
"version_tag": "v1",
"z": 0.566464,
"z_crit": 3
},
{
"direction": "no gap beyond sampling error",
"equating_offset": 13.753223,
"gap": -0.972959,
"gap_se": 4.607169,
"generation": "gen2",
"merkle_root": "4adc9bbb0bc17564ad2effaa4d6fc902a0f319bb3e26735ac67d1b988cbe6e2d",
"model": "mlp_pixels",
"score_private_equated": 73.499084,
"score_private_raw": 59.745861,
"score_public": 72.526125,
"set_hash": "ad30d214daa285c5123df991dd0eae58038f99f3490b840299cb620279ff2e73",
"submitter": "Submitter C",
"verdict": "clear",
"version_tag": "v2",
"z": -0.211184,
"z_crit": 3
}
],
"labelling": "PUBLIC-DATA DEMONSTRATION OF THE PRIVATE PROTOCOL"
},
"sources": [
{
"path": "results/cross_site/organ_ct__mlp_pixels.json",
"sha256": "869f0d2a4d961737b0a27288f70f9167072589a491721c277d58f12faeac5888"
},
{
"path": "results/heldout/audit_report.json",
"sha256": "df460b6d91a6512dfa1e7dc650637fce2b4dd3a1ec32e03446bf6e11f9c597d2"
},
{
"path": "results/heldout/cycle.json",
"sha256": "e012b8978457b51b5c31d18d14d2cc92f100731897523306acae19934f943502"
},
{
"path": "results/heldout/manifests/gen1.seal.json",
"sha256": "91a7adfa29870a0719125cdd48f6dcb149f1f6485ac1cfa104c6e00057ad9b81"
},
{
"path": "results/heldout/manifests/gen2.seal.json",
"sha256": "f61470918a09cf39e074ef14fb7ade4ac3a13d36d9385f18cf9b92e4b5329547"
},
{
"path": "results/heldout/manifests/public.seal.json",
"sha256": "1a4fd930d2e781ac7163483990f15d6694e28e507d1b3ca009ff09395e47497c"
},
{
"path": "results/records/bloodmnist__mlp_pixels.json",
"sha256": "e14d9bdc08bfcde6cc18e93dc8fd0ee7370f59af233a9d295e77e9fff171ffef"
},
{
"path": "results/records/breastmnist__mlp_pixels.json",
"sha256": "772826e38c3f9799f79a69686b4feb90e6d58ac07445fafd3ade87c43049e8f5"
},
{
"path": "results/records/dermamnist__mlp_pixels.json",
"sha256": "7be4a950f9027ce59ef71a5c886f31bc7b5d913ac9e6162277c5c320ea22d441"
},
{
"path": "results/records/octmnist__mlp_pixels.json",
"sha256": "76ec19f8902df27db6395ec1f9d2a299680210e01e52499c8201b15b796d51d8"
},
{
"path": "results/records/organamnist__mlp_pixels.json",
"sha256": "d62d5c9c40d95e997a1b398505500fcee110d0fb489634e36c1f8c42202b47d8"
},
{
"path": "results/records/organcmnist__mlp_pixels.json",
"sha256": "5d0c04e85398fb67d366103492ba2fa9d00dbd8447df6b4cf0d64a5052b04093"
},
{
"path": "results/records/organsmnist__mlp_pixels.json",
"sha256": "ea2be64d056b8150744fc86e760a64a3d04866ae8c08969975a0de6971d607b0"
},
{
"path": "results/records/pathmnist__mlp_pixels.json",
"sha256": "baaf8a0b2375f8a85936da60790bfdd6c840c0915ea6dcc5e10e073fd17d3c97"
},
{
"path": "results/records/pneumoniamnist__mlp_pixels.json",
"sha256": "c616b3eb79beef2dd991494401891920bd5c247525419d7a541878deaf287fd8"
},
{
"path": "results/records/retinamnist__mlp_pixels.json",
"sha256": "9a3851a94329f5971adfeaaa16c14e16c789a7dc14f2877520699340f982b03e"
},
{
"path": "results/records/tissuemnist__mlp_pixels.json",
"sha256": "d78e92ffdb7f185a4c3fca5d357cf8a48cba277fa37e532be18d772689a83cfb"
}
]
},
"declared": null,
"declared_note": "No vendor declaration. This is an open reference baseline implemented by NakedSignal, not a commercial product: there is no vendor to assert an intended use or to sign an attestation, so the declared block is null rather than filled in on the vendor's behalf.",
"expiry_basis": "the sealed held-out generation this evidence cycle is anchored to (gen2, set hash ad30d214daa285c5...) is published to expire 2026-09-20; per spec, a passport lapses when its generation retires or its spec version is superseded, whichever comes first",
"expiry_utc": "2026-09-20T00:00:00Z",
"issued_utc": "2026-08-03T16:03:42Z",
"issuer": {
"algo": "Ed25519",
"key_id": "ns-passport-2026-07",
"name": "NakedSignal",
"public_key_b64": "0PBCC6mjzcppR5QdPcK7vP/AamB6MP8l1T4ypMYbsMw="
},
"observed": null,
"observed_note": "Not available: this requires continuous monitoring. Production history and drift are observed fields, and nothing is deployed, so there is nothing to observe. The object is present and null on purpose. The shape is the roadmap, not a claim.",
"passport_version": "0.1",
"signature": "7G1iJlJlzhiNlNSu4b8dnTaGzlU2gFHIEigxULao5WY3ZBWmdvR9qXezYwdIaCUVUZhDYXeY3dH76j1npsiMCA==",
"spec_version": "MedEval-1 v0.1",
"subject": {
"behaviour_version": "1.0.0",
"code_fingerprint": "b444c02112cc",
"code_fingerprints": [
"b444c02112cc"
],
"family": "shallow",
"model_id": "mlp_pixels",
"model_name": "MLP (256) on raw pixels",
"params": "1 hidden layer of 256, adam, early stopping when n>=200"
}
}Canonicalisation: RFC 8785 (JCS)-compatible: UTF-8, object keys sorted by code point, no insignificant whitespace, ECMAScript number formatting. every float is rounded once at emit to 6 decimal places (Python round(), banker's rounding); integers are left exact. Signed over the whole document with the `signature` member removed, canonicalised as above and encoded UTF-8. The public key for ns-passport-2026-07 is 0PBCC6mjzcppR5QdPcK7vP/AamB6MP8l1T4ypMYbsMw= — see Governance.