Back to the consoleDocumentation
Small CNN, trained from scratch
LIVEThe signed evidence record for cnn_scratch. Everything below was read out of the passport file; the signature was checked when this page was built, by the same code the Verify button runs.
SIGNATURE VALIDexit 0signature valid, issued by NakedSignal under key ns-passport-2026-07, not expired
SubjectSmall CNN, trained from scratch
cnn_scratchConfiguration3 conv blocks, 20 epochs, adam 1e-3, early stop on val
Code fingerprintb444c02112cc
Spec versionMedEval-1 v0.1
Issued2026-08-03T16:03:42Z by NakedSignal
Expires2026-09-20T00:00:00Z — the sealed held-out generation this evidence cycle is anchored to (gen2, set hash ad30d214daa285c5...) is published to expire 2026-09-20; per spec, a passport lapses when its generation retires or its spec version is superseded, whichever comes first
key ns-passport-2026-07 · Ed25519
The check runs against the public key published on the governance page, using your browser's own Ed25519 implementation. The exit code shown is the exit code verify_passport.py returns for the same document.
Measured by NakedSignal
Computed in this repo by the MedEval-1 harness on data the model had never seen. Every number below was copied out of a file the harness wrote — 17 of them, each listed with its SHA-256 at the foot of this panel — and is printed exactly as it was signed, not re-rounded for display.
| Task | Domain | n test | accuracy | calibration | uncertainty | subgroup | corruption | acquisition | limited data | cost | spec sensitivity | Index |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bloodmnist | medicine | 3421 | 95.670132 | 92.839737 | 99.539304 | 91.549296 | 48.281147 | 62.262611 | 36.159496 | n/a | n/a | 75.826128 |
| breastmnist | medicine | 156 | 59.649123 | 57.62214 | 89.000119 | 33.333333 | 49.329732 | 73.445378 | 36.904762 | n/a | n/a | 57.012677 |
| dermamnist | medicine | 2005 | 44.349021 | 88.548352 | 91.054933 | 0 | 59.039044 | 76.717358 | 41.153195 | n/a | n/a | 56.652235 |
| octmnist | medicine | 1000 | 57.066667 | 0 | 78.10626 | 1.866667 | 31.964842 | 52.305296 | 32.476636 | n/a | n/a | 36.642422 |
| organamnist | medicine | 17778 | 91.496937 | 84.485694 | 99.13683 | 77.776163 | 72.660343 | 83.556087 | 43.447091 | n/a | n/a | 80.561432 |
| organcmnist | medicine | 8216 | 90.014245 | 80.353307 | 98.791999 | 75.639615 | 75.19459 | 83.541043 | 41.967047 | n/a | n/a | 79.594617 |
| organsmnist | medicine | 8827 | 72.475324 | 53.915875 | 93.222958 | 37.6082 | 65.178139 | 78.199663 | 40.691244 | n/a | n/a | 64.100882 |
| pathmnist | medicine | 7180 | 78.7404 | 61.2935 | 93.718695 | 38.539192 | 38.292373 | 46.412525 | 42.077896 | n/a | n/a | 56.68336 |
| pneumoniamnist | medicine | 624 | 74.871795 | 64.538792 | 95.083525 | 53.846154 | 55.300065 | 72.709285 | 0 | n/a | n/a | 61.913727 |
| retinamnist | medicine | 400 | 17.238807 | 80.109489 | 64.736128 | 0 | 94.321353 | 89.328189 | 36.139308 | n/a | n/a | 54.747757 |
| tissuemnist | medicine | 47280 | 41.647253 | 93.716198 | 72.569134 | 8.43836 | 48.226827 | 58.390874 | 30.629144 | n/a | n/a | 50.501494 |
Cross-site retention · organ_ct
retention is absolute retained skill (chance-corrected), per spec 2.2.1; the diagonal is same-site, every off-diagonal cell is train-on-row, test-on-column
Sitesorganamnist, organcmnist, organsmnist
Mean retention0.63221
Worst retention0.374032
Private held-out track
this subject was not submitted to a sealed generation.Known limitations
Subgroup metadataabsent on 11 of 11 corpora, so the subgroup score falls back to the worst diagnostic class. That absence is a disclosure, not a pass.
Training set cappedyes on at least one task — see the raw JSON for which
Observed in deploymentnothing — see the third panel
Audit chain
Bundles re-derived48 of 48
Ledger100 entries, head 6b7b371ff650a8ed…, chain intact
Re-run it yourself
python src/heldout/audit.py 1f66c7719ab3943c6fcc17 source files, with hashes
| results/cross_site/organ_ct__cnn_scratch.json | 2cb33bd19620e036d679332fd717024b5b74f1cae25c506645558f82ba8018ac |
| results/heldout/audit_report.json | df460b6d91a6512dfa1e7dc650637fce2b4dd3a1ec32e03446bf6e11f9c597d2 |
| results/heldout/cycle.json | e012b8978457b51b5c31d18d14d2cc92f100731897523306acae19934f943502 |
| results/heldout/manifests/gen1.seal.json | 91a7adfa29870a0719125cdd48f6dcb149f1f6485ac1cfa104c6e00057ad9b81 |
| results/heldout/manifests/gen2.seal.json | f61470918a09cf39e074ef14fb7ade4ac3a13d36d9385f18cf9b92e4b5329547 |
| results/heldout/manifests/public.seal.json | 1a4fd930d2e781ac7163483990f15d6694e28e507d1b3ca009ff09395e47497c |
| results/records/bloodmnist__cnn_scratch.json | 658e3e5f7cfa4ad5fd6c7b39ba0aeafd63cd24b954494a2ea543b80339ff90ee |
| results/records/breastmnist__cnn_scratch.json | 2212aec836e7f76f734e4219f2440104ddf80ee2db0595211786509d77fc9ad6 |
| results/records/dermamnist__cnn_scratch.json | ecafb8abee712c7ae14cefcbc3e079ecdbbec6ca0eaf1915cd70d66b033443c4 |
| results/records/octmnist__cnn_scratch.json | 051221f9e35353e03d86cf3fc99a668fc38a9dde7a433eeee6c08430ff7577d7 |
| results/records/organamnist__cnn_scratch.json | ac309b752caab9b09114ef594183239dc8b5fdb3732f0e694a9bf2aef0a9e46d |
| results/records/organcmnist__cnn_scratch.json | b00ac3f602f5687f42d5649af03dea831ae0f80f650542f3bbc20a3766d12225 |
| results/records/organsmnist__cnn_scratch.json | 83a9069311350c3760a04eb18c472285d1d26facd163ea735ef6448af4187a95 |
| results/records/pathmnist__cnn_scratch.json | 19764268891a48ac331ae275acc81c85b16828b02614ee8f32abda10afd4d909 |
| results/records/pneumoniamnist__cnn_scratch.json | e15163c863218cf6062262c8bfc0c3800e7a3b6995751ffb2802e5a204876209 |
| results/records/retinamnist__cnn_scratch.json | 702b5c2292a18b58c0d1deb42fee6ea2dc74f3c6afa7e10203958a5ec14b8b01 |
| results/records/tissuemnist__cnn_scratch.json | 792813023724858e90cb26cab9ff90c5a00925ee563ecf98bccb4e4c08ae4396 |
Declared by vendor
Five fields a vendor asserts about its own product — intended use, forbidden use, training cutoff, regulatory clearances, and who is personally attesting. NakedSignal never fills these in on a vendor's behalf.
No vendor declaration. This is an open reference baseline implemented by NakedSignal, not a commercial product: there is no vendor to assert an intended use or to sign an attestation, so the declared block is null rather than filled in on the vendor's behalf.
Observed in deployment
Production history and drift — how the model has actually behaved since it was deployed, on real traffic.
Not available: this requires continuous monitoring. Production history and drift are observed fields, and nothing is deployed, so there is nothing to observe. The object is present and null on purpose. The shape is the roadmap, not a claim.
Raw JSON — the whole signed document, 81,184 characters
{
"canonicalisation": {
"float_rounding": "every float is rounded once at emit to 6 decimal places (Python round(), banker's rounding); integers are left exact",
"form": "RFC 8785 (JCS)-compatible: UTF-8, object keys sorted by code point, no insignificant whitespace, ECMAScript number formatting",
"signed_over": "the whole document with the `signature` member removed, canonicalised as above and encoded UTF-8"
},
"computed": {
"audit": {
"audit_command": "python src/heldout/audit.py 1f66c7719ab3943c6fcc",
"bundles": 48,
"ledger_entries": 100,
"ledger_head": "6b7b371ff650a8edaf477490f2a5f9b043433f3ba9941989ee1962ed8f1b1d88",
"ledger_intact": true,
"rederived": 48
},
"cross_site": {
"code_fingerprint": "b444c02112cc",
"device": "mps",
"family": "organ_ct",
"matrix": {
"organamnist": {
"organamnist": {
"balanced_accuracy": 0.922699,
"chance_corrected": 0.914969,
"ece": 0.031029
},
"organcmnist": {
"balanced_accuracy": 0.739566,
"chance_corrected": 0.713523,
"ece": 0.144638,
"retention": 0.779832
},
"organsmnist": {
"balanced_accuracy": 0.402025,
"chance_corrected": 0.342228,
"ece": 0.399281,
"retention": 0.374032
}
},
"organcmnist": {
"organamnist": {
"balanced_accuracy": 0.670079,
"chance_corrected": 0.637087,
"ece": 0.180735,
"retention": 0.707762
},
"organcmnist": {
"balanced_accuracy": 0.90922,
"chance_corrected": 0.900142,
"ece": 0.039293
},
"organsmnist": {
"balanced_accuracy": 0.500357,
"chance_corrected": 0.450392,
"ece": 0.318423,
"retention": 0.500357
}
},
"organsmnist": {
"organamnist": {
"balanced_accuracy": 0.436419,
"chance_corrected": 0.38006,
"ece": 0.312101,
"retention": 0.5244
},
"organcmnist": {
"balanced_accuracy": 0.688418,
"chance_corrected": 0.65726,
"ece": 0.112176,
"retention": 0.906874
},
"organsmnist": {
"balanced_accuracy": 0.749776,
"chance_corrected": 0.724753,
"ece": 0.092168
}
}
},
"mean_retention": 0.63221,
"note": "retention is absolute retained skill (chance-corrected), per spec 2.2.1; the diagonal is same-site, every off-diagonal cell is train-on-row, test-on-column",
"score": 63.220953,
"site_labels": {
"organamnist": "axial",
"organcmnist": "coronal",
"organsmnist": "sagittal"
},
"sites": [
"organamnist",
"organcmnist",
"organsmnist"
],
"spec_version": "MedEval-1 v0.1",
"worst_retention": 0.374032
},
"evaluations": [
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 75.826128
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.964338,
"auc": 0.998087,
"balanced_accuracy": 0.962114,
"ci95": [
0.954715,
0.969113
],
"n_test": 3421,
"score": 95.670132
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.956948,
"mean_agreement": 0.675333,
"mean_kappa": 0.629668,
"mean_retention": 0.622626,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9783,
0.9475,
0.7467
],
"mean_kappa": 0.872276,
"mean_retention": 0.873256,
"retention_by_severity": [
0.9889,
0.9405,
0.6904
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8592,
0.8308,
0.6725
],
"mean_kappa": 0.749683,
"mean_retention": 0.729549,
"retention_by_severity": [
0.8312,
0.7962,
0.5613
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.7392,
0.1175,
0.0917
],
"mean_kappa": 0.244572,
"mean_retention": 0.250653,
"retention_by_severity": [
0.6868,
0.0538,
0.0114
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.8308,
0.7283,
0.6367
],
"mean_kappa": 0.689386,
"mean_retention": 0.696087,
"retention_by_severity": [
0.8,
0.6898,
0.5985
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.8775,
0.6892,
0.3842
],
"mean_kappa": 0.592421,
"mean_retention": 0.563584,
"retention_by_severity": [
0.8252,
0.5872,
0.2783
],
"scored": true
}
},
"score": 62.262611,
"worst_retention": 0.011446
},
"calibration": {
"accuracy": 0.964338,
"brier": 0.054458,
"ece": 0.014321,
"mean_confidence": 0.9765,
"overconfidence": 0.012162,
"score": 92.839737
},
"corruption": {
"clean_reference_cc": 0.956948,
"mean_agreement": 0.542698,
"mean_kappa": 0.477133,
"mean_retention": 0.482811,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.6908,
0.32,
0.2125
],
"mean_kappa": 0.285003,
"mean_retention": 0.227443,
"retention_by_severity": [
0.4935,
0.1427,
0.0461
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.3658,
0.2192,
0.2058
],
"mean_kappa": 0.160208,
"mean_retention": 0.183528,
"retention_by_severity": [
0.2769,
0.1476,
0.1261
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.8467,
0.6642,
0.4583
],
"mean_kappa": 0.605387,
"mean_retention": 0.621074,
"retention_by_severity": [
0.8174,
0.6348,
0.411
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9108,
0.6,
0.0875
],
"mean_kappa": 0.479048,
"mean_retention": 0.492202,
"retention_by_severity": [
0.9141,
0.552,
0.0105
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.845,
0.2517,
0.7017
],
"mean_kappa": 0.552308,
"mean_retention": 0.575394,
"retention_by_severity": [
0.817,
0.2444,
0.6648
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9892,
0.9742,
0.9175
],
"mean_kappa": 0.953413,
"mean_retention": 0.967025,
"retention_by_severity": [
0.9981,
0.9884,
0.9145
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.7067,
0.3483,
0.0808
],
"mean_kappa": 0.304562,
"mean_retention": 0.313014,
"retention_by_severity": [
0.6481,
0.291,
0
],
"scored": true
}
},
"score": 48.281147,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.956701,
"mean_retention": 0.361595,
"per_budget": {
"n100": {
"chance_corrected": 0.854905,
"n_labels": 800,
"retention": 0.893597
},
"n20": {
"chance_corrected": 0.18291,
"n_labels": 160,
"retention": 0.191188
},
"n5": {
"chance_corrected": 0,
"n_labels": 40,
"retention": 0
}
},
"score": 36.159496
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 91.549296,
"worst_class_recall": 0.926056
},
"uncertainty": {
"full_accuracy": 0.964338,
"risk_coverage_auc": 0.995969,
"score": 99.539304,
"selective_acc_at_50": 0.999415,
"selective_acc_at_80": 0.996346
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 11959,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Microscopy",
"n_classes": 8,
"n_test": 3421,
"runtime_s": 34.2,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "bloodmnist",
"task_name": "Peripheral blood cells (8-class)",
"timestamp": "2026-08-01T19:19:11+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 57.012677
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.858974,
"auc": 0.902673,
"balanced_accuracy": 0.798246,
"ci95": [
0.723641,
0.860586
],
"n_test": 156,
"score": 59.649123
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.596491,
"mean_agreement": 0.82265,
"mean_kappa": 0.549964,
"mean_retention": 0.734454,
"n_cases": 156,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9808,
0.9487,
0.7821
],
"mean_kappa": 0.633228,
"mean_retention": 0.636555,
"retention_by_severity": [
0.9349,
0.8697,
0.105
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8397,
0.7756,
0.7564
],
"mean_kappa": 0.535653,
"mean_retention": 0.892857,
"retention_by_severity": [
1,
0.8466,
0.8319
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9103,
0.859,
0.7821
],
"mean_kappa": 0.588286,
"mean_retention": 0.742297,
"retention_by_severity": [
0.9412,
0.8067,
0.479
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.7179,
0.7115,
0.5962
],
"mean_kappa": 0.38574,
"mean_retention": 0.820728,
"retention_by_severity": [
0.9538,
0.8887,
0.6197
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9679,
0.9167,
0.7949
],
"mean_kappa": 0.606914,
"mean_retention": 0.579832,
"retention_by_severity": [
0.9643,
0.645,
0.1303
],
"scored": true
}
},
"score": 73.445378,
"worst_retention": 0.105042
},
"calibration": {
"accuracy": 0.858974,
"brier": 0.209601,
"ece": 0.084756,
"mean_confidence": 0.924424,
"overconfidence": 0.06545,
"score": 57.62214
},
"corruption": {
"clean_reference_cc": 0.596491,
"mean_agreement": 0.815018,
"mean_kappa": 0.440312,
"mean_retention": 0.493297,
"n_cases": 156,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9359,
0.8462,
0.7692
],
"mean_kappa": 0.419008,
"mean_retention": 0.403361,
"retention_by_severity": [
0.7605,
0.4496,
0
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8974,
0.8205,
0.7308
],
"mean_kappa": 0.545439,
"mean_retention": 0.558824,
"retention_by_severity": [
0.8277,
0.5252,
0.3235
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.8077,
0.5769,
0.5064
],
"mean_kappa": 0.344081,
"mean_retention": 0.672969,
"retention_by_severity": [
0.9958,
0.6849,
0.3382
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.8782,
0.7692,
0.7692
],
"mean_kappa": 0.193072,
"mean_retention": 0.153361,
"retention_by_severity": [
0.4601,
0,
0
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.8654,
0.8974,
0.8333
],
"mean_kappa": 0.638861,
"mean_retention": 0.742997,
"retention_by_severity": [
0.9727,
0.8824,
0.3739
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9936,
0.9679,
0.8974
],
"mean_kappa": 0.851457,
"mean_retention": 0.864846,
"retention_by_severity": [
1,
0.9895,
0.605
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.8141,
0.7692,
0.7692
],
"mean_kappa": 0.090264,
"mean_retention": 0.056723,
"retention_by_severity": [
0.1702,
0,
0
],
"scored": true
}
},
"score": 49.329732,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.596491,
"mean_retention": 0.369048,
"per_budget": {
"n100": {
"chance_corrected": 0.359649,
"n_labels": 200,
"retention": 0.602941
},
"n20": {
"chance_corrected": 0.017544,
"n_labels": 40,
"retention": 0.029412
},
"n5": {
"chance_corrected": 0.283208,
"n_labels": 10,
"retention": 0.47479
}
},
"score": 36.904762
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 33.333333,
"worst_class_recall": 0.666667
},
"uncertainty": {
"full_accuracy": 0.858974,
"risk_coverage_auc": 0.945001,
"score": 89.000119,
"selective_acc_at_50": 0.935897,
"selective_acc_at_80": 0.92
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 546,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Ultrasound",
"n_classes": 2,
"n_test": 156,
"runtime_s": 2.5,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "breastmnist",
"task_name": "Breast ultrasound (malignancy)",
"timestamp": "2026-08-01T19:12:59+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 56.652235
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.771571,
"auc": 0.926125,
"balanced_accuracy": 0.522992,
"ci95": [
0.482709,
0.566408
],
"n_test": 2005,
"score": 44.349021
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.435485,
"mean_agreement": 0.778889,
"mean_kappa": 0.606071,
"mean_retention": 0.767174,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.89,
0.5975,
0.2533
],
"mean_kappa": 0.433673,
"mean_retention": 0.6537,
"retention_by_severity": [
0.9681,
0.7079,
0.2851
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.925,
0.8908,
0.8525
],
"mean_kappa": 0.754461,
"mean_retention": 0.840558,
"retention_by_severity": [
0.918,
0.8298,
0.7739
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8833,
0.7817,
0.6983
],
"mean_kappa": 0.559295,
"mean_retention": 0.609188,
"retention_by_severity": [
0.8306,
0.6287,
0.3683
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.8925,
0.86,
0.84
],
"mean_kappa": 0.690457,
"mean_retention": 0.847676,
"retention_by_severity": [
0.8655,
0.8422,
0.8354
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9058,
0.7892,
0.6233
],
"mean_kappa": 0.592466,
"mean_retention": 0.884745,
"retention_by_severity": [
1,
0.9052,
0.7491
],
"scored": true
}
},
"score": 76.717358,
"worst_retention": 0.285065
},
"calibration": {
"accuracy": 0.771571,
"brier": 0.320743,
"ece": 0.022903,
"mean_confidence": 0.794337,
"overconfidence": 0.022766,
"score": 88.548352
},
"corruption": {
"clean_reference_cc": 0.435485,
"mean_agreement": 0.666032,
"mean_kappa": 0.433985,
"mean_retention": 0.59039,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.82,
0.6758,
0.545
],
"mean_kappa": 0.382733,
"mean_retention": 0.614601,
"retention_by_severity": [
0.8398,
0.6934,
0.3106
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8367,
0.6925,
0.5975
],
"mean_kappa": 0.463335,
"mean_retention": 0.618717,
"retention_by_severity": [
0.9085,
0.6407,
0.3069
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9125,
0.84,
0.7867
],
"mean_kappa": 0.640404,
"mean_retention": 0.7437,
"retention_by_severity": [
0.8448,
0.7796,
0.6067
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.5717,
0.25,
0.34
],
"mean_kappa": 0.125043,
"mean_retention": 0.316395,
"retention_by_severity": [
0.7269,
0.2223,
0
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9375,
0.7183,
0.7742
],
"mean_kappa": 0.641592,
"mean_retention": 0.762315,
"retention_by_severity": [
1,
0.3425,
0.9444
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9775,
0.9117,
0.7133
],
"mean_kappa": 0.769279,
"mean_retention": 0.92525,
"retention_by_severity": [
0.9717,
0.9739,
0.8301
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.3242,
0.2983,
0.4633
],
"mean_kappa": 0.015512,
"mean_retention": 0.151756,
"retention_by_severity": [
0.3168,
0.1384,
0
],
"scored": true
}
},
"score": 59.039044,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.44349,
"mean_retention": 0.411532,
"per_budget": {
"n100": {
"chance_corrected": 0.510548,
"n_labels": 679,
"retention": 1
},
"n20": {
"chance_corrected": 0.089798,
"n_labels": 140,
"retention": 0.20248
},
"n5": {
"chance_corrected": 0.014243,
"n_labels": 35,
"retention": 0.032115
}
},
"score": 41.153195
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0.086957
},
"uncertainty": {
"full_accuracy": 0.771571,
"risk_coverage_auc": 0.923328,
"score": 91.054933,
"selective_acc_at_50": 0.95509,
"selective_acc_at_80": 0.850374
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 7007,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Dermatoscopy",
"n_classes": 7,
"n_test": 2005,
"runtime_s": 16.7,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "dermamnist",
"task_name": "Dermatoscopy (7-class skin lesion)",
"timestamp": "2026-08-01T19:17:31+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 36.642422
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.678,
"auc": 0.922577,
"balanced_accuracy": 0.678,
"ci95": [
0.657351,
0.70015
],
"n_test": 1000,
"score": 57.066667
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.570667,
"mean_agreement": 0.585067,
"mean_kappa": 0.357663,
"mean_retention": 0.523053,
"n_cases": 1000,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.622,
0.475,
0.334
],
"mean_kappa": 0.247795,
"mean_retention": 0.329439,
"retention_by_severity": [
0.5607,
0.3201,
0.1075
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.708,
0.592,
0.5
],
"mean_kappa": 0.33114,
"mean_retention": 0.301402,
"retention_by_severity": [
0.507,
0.2967,
0.1005
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.783,
0.651,
0.524
],
"mean_kappa": 0.522297,
"mean_retention": 0.913551,
"retention_by_severity": [
1,
1,
0.7407
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.533,
0.468,
0.457
],
"mean_kappa": 0.092338,
"mean_retention": 0.095016,
"retention_by_severity": [
0.1939,
0.0584,
0.0327
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.801,
0.694,
0.634
],
"mean_kappa": 0.594745,
"mean_retention": 0.975857,
"retention_by_severity": [
1,
1,
0.9276
],
"scored": true
}
},
"score": 52.305296,
"worst_retention": 0.03271
},
"calibration": {
"accuracy": 0.678,
"brier": 0.508364,
"ece": 0.206592,
"mean_confidence": 0.884592,
"overconfidence": 0.206592,
"score": 0
},
"corruption": {
"clean_reference_cc": 0.570667,
"mean_agreement": 0.53081,
"mean_kappa": 0.281624,
"mean_retention": 0.319648,
"n_cases": 1000,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.719,
0.595,
0.513
],
"mean_kappa": 0.340942,
"mean_retention": 0.313084,
"retention_by_severity": [
0.528,
0.2921,
0.1192
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.78,
0.635,
0.523
],
"mean_kappa": 0.435665,
"mean_retention": 0.46729,
"retention_by_severity": [
0.7079,
0.4509,
0.243
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.609,
0.453,
0.451
],
"mean_kappa": 0.129465,
"mean_retention": 0.130062,
"retention_by_severity": [
0.3364,
0.0257,
0.028
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.424,
0.307,
0.256
],
"mean_kappa": 0.08228,
"mean_retention": 0.140187,
"retention_by_severity": [
0.2967,
0.0818,
0.0421
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.678,
0.513,
0.491
],
"mean_kappa": 0.294225,
"mean_retention": 0.338785,
"retention_by_severity": [
0.486,
0.3715,
0.1589
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.894,
0.737,
0.57
],
"mean_kappa": 0.599565,
"mean_retention": 0.682243,
"retention_by_severity": [
0.9393,
0.6846,
0.4229
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.436,
0.3,
0.263
],
"mean_kappa": 0.089228,
"mean_retention": 0.165888,
"retention_by_severity": [
0.3598,
0.1075,
0.0304
],
"scored": true
}
},
"score": 31.964842,
"worst_retention": 0.025701
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.570667,
"mean_retention": 0.324766,
"per_budget": {
"n100": {
"chance_corrected": 0.556,
"n_labels": 400,
"retention": 0.974299
},
"n20": {
"chance_corrected": 0,
"n_labels": 80,
"retention": 0
},
"n5": {
"chance_corrected": 0,
"n_labels": 20,
"retention": 0
}
},
"score": 32.476636
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 1.866667,
"worst_class_recall": 0.264
},
"uncertainty": {
"full_accuracy": 0.678,
"risk_coverage_auc": 0.835797,
"score": 78.10626,
"selective_acc_at_50": 0.828,
"selective_acc_at_80": 0.73875
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 97477,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "OCT",
"n_classes": 4,
"n_test": 1000,
"runtime_s": 36.3,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "octmnist",
"task_name": "Retinal OCT (4-class)",
"timestamp": "2026-08-01T19:35:47+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 80.561432
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.931432,
"auc": 0.996829,
"balanced_accuracy": 0.922699,
"ci95": [
0.918021,
0.926978
],
"n_test": 17778,
"score": 91.496937
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.919705,
"mean_agreement": 0.787389,
"mean_kappa": 0.763876,
"mean_retention": 0.835561,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9875,
0.9167,
0.7175
],
"mean_kappa": 0.859811,
"mean_retention": 0.923762,
"retention_by_severity": [
0.9988,
0.978,
0.7945
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.86,
0.7383,
0.5892
],
"mean_kappa": 0.700363,
"mean_retention": 0.806403,
"retention_by_severity": [
0.9422,
0.8239,
0.6531
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9408,
0.7983,
0.6342
],
"mean_kappa": 0.766239,
"mean_retention": 0.775105,
"retention_by_severity": [
0.9542,
0.777,
0.5941
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.7733,
0.6675,
0.5458
],
"mean_kappa": 0.627291,
"mean_retention": 0.77018,
"retention_by_severity": [
0.8753,
0.7748,
0.6605
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9617,
0.8967,
0.7833
],
"mean_kappa": 0.865676,
"mean_retention": 0.902354,
"retention_by_severity": [
0.9872,
0.9251,
0.7947
],
"scored": true
}
},
"score": 83.556087,
"worst_retention": 0.594089
},
"calibration": {
"accuracy": 0.931432,
"brier": 0.104176,
"ece": 0.031029,
"mean_confidence": 0.962225,
"overconfidence": 0.030793,
"score": 84.485694
},
"corruption": {
"clean_reference_cc": 0.919705,
"mean_agreement": 0.710516,
"mean_kappa": 0.678368,
"mean_retention": 0.726603,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9192,
0.6717,
0.2392
],
"mean_kappa": 0.574978,
"mean_retention": 0.605434,
"retention_by_severity": [
0.9444,
0.6443,
0.2276
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.855,
0.3567,
0.1542
],
"mean_kappa": 0.40381,
"mean_retention": 0.456673,
"retention_by_severity": [
0.882,
0.3615,
0.1266
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.8317,
0.555,
0.3358
],
"mean_kappa": 0.531501,
"mean_retention": 0.658234,
"retention_by_severity": [
0.9247,
0.658,
0.392
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.98,
0.9233,
0.545
],
"mean_kappa": 0.787217,
"mean_retention": 0.794492,
"retention_by_severity": [
0.9903,
0.9159,
0.4773
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9108,
0.7583,
0.7033
],
"mean_kappa": 0.767267,
"mean_retention": 0.865359,
"retention_by_severity": [
0.9713,
0.832,
0.7928
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9942,
0.9892,
0.9783
],
"mean_kappa": 0.985669,
"mean_retention": 0.993537,
"retention_by_severity": [
0.9966,
0.9967,
0.9873
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9433,
0.8075,
0.4692
],
"mean_kappa": 0.698131,
"mean_retention": 0.712495,
"retention_by_severity": [
0.9556,
0.7984,
0.3835
],
"scored": true
}
},
"score": 72.660343,
"worst_retention": 0.126602
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.914969,
"mean_retention": 0.434471,
"per_budget": {
"n100": {
"chance_corrected": 0.831124,
"n_labels": 1100,
"retention": 0.908363
},
"n20": {
"chance_corrected": 0.296169,
"n_labels": 220,
"retention": 0.323693
},
"n5": {
"chance_corrected": 0.06529,
"n_labels": 55,
"retention": 0.071357
}
},
"score": 43.447091
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 77.776163,
"worst_class_recall": 0.797965
},
"uncertainty": {
"full_accuracy": 0.931432,
"risk_coverage_auc": 0.992153,
"score": 99.13683,
"selective_acc_at_50": 0.998425,
"selective_acc_at_80": 0.992828
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 34561,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 17778,
"runtime_s": 30.2,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organamnist",
"task_name": "Abdominal CT organs — axial (A)",
"timestamp": "2026-08-01T19:33:13+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 79.594617
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.919669,
"auc": 0.99405,
"balanced_accuracy": 0.90922,
"ci95": [
0.902614,
0.916165
],
"n_test": 8216,
"score": 90.014245
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.876982,
"mean_agreement": 0.784111,
"mean_kappa": 0.760243,
"mean_retention": 0.83541,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9875,
0.9292,
0.7083
],
"mean_kappa": 0.859823,
"mean_retention": 0.90103,
"retention_by_severity": [
0.9983,
0.9677,
0.7371
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8733,
0.7533,
0.5875
],
"mean_kappa": 0.710093,
"mean_retention": 0.829637,
"retention_by_severity": [
0.944,
0.8499,
0.6949
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9158,
0.8208,
0.7008
],
"mean_kappa": 0.788765,
"mean_retention": 0.834876,
"retention_by_severity": [
0.9438,
0.8436,
0.7173
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.6892,
0.5525,
0.4617
],
"mean_kappa": 0.524922,
"mean_retention": 0.650335,
"retention_by_severity": [
0.7579,
0.6474,
0.5457
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9758,
0.9342,
0.8717
],
"mean_kappa": 0.917614,
"mean_retention": 0.961174,
"retention_by_severity": [
1,
0.976,
0.9075
],
"scored": true
}
},
"score": 83.541043,
"worst_retention": 0.545704
},
"calibration": {
"accuracy": 0.919669,
"brier": 0.124899,
"ece": 0.039293,
"mean_confidence": 0.958782,
"overconfidence": 0.039113,
"score": 80.353307
},
"corruption": {
"clean_reference_cc": 0.876982,
"mean_agreement": 0.72869,
"mean_kappa": 0.699588,
"mean_retention": 0.751946,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9133,
0.7408,
0.2917
],
"mean_kappa": 0.614051,
"mean_retention": 0.677024,
"retention_by_severity": [
0.9742,
0.7672,
0.2897
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8792,
0.5408,
0.2008
],
"mean_kappa": 0.497414,
"mean_retention": 0.540865,
"retention_by_severity": [
0.9112,
0.5235,
0.1879
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.7825,
0.4667,
0.3267
],
"mean_kappa": 0.481739,
"mean_retention": 0.592335,
"retention_by_severity": [
0.8613,
0.5529,
0.3628
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9642,
0.8917,
0.7017
],
"mean_kappa": 0.832162,
"mean_retention": 0.862097,
"retention_by_severity": [
0.9855,
0.8977,
0.7031
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.92,
0.5933,
0.7433
],
"mean_kappa": 0.719758,
"mean_retention": 0.782324,
"retention_by_severity": [
0.9708,
0.5765,
0.7996
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9933,
0.9842,
0.965
],
"mean_kappa": 0.97834,
"mean_retention": 0.996824,
"retention_by_severity": [
0.9985,
1,
0.9919
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9233,
0.8192,
0.6608
],
"mean_kappa": 0.773653,
"mean_retention": 0.812152,
"retention_by_severity": [
0.9425,
0.8374,
0.6565
],
"scored": true
}
},
"score": 75.19459,
"worst_retention": 0.187869
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.900142,
"mean_retention": 0.41967,
"per_budget": {
"n100": {
"chance_corrected": 0.838214,
"n_labels": 1100,
"retention": 0.931202
},
"n20": {
"chance_corrected": 0.285778,
"n_labels": 220,
"retention": 0.317481
},
"n5": {
"chance_corrected": 0.009298,
"n_labels": 55,
"retention": 0.010329
}
},
"score": 41.967047
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 75.639615,
"worst_class_recall": 0.778542
},
"uncertainty": {
"full_accuracy": 0.919669,
"risk_coverage_auc": 0.989018,
"score": 98.791999,
"selective_acc_at_50": 0.997322,
"selective_acc_at_80": 0.986916
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 12975,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 8216,
"runtime_s": 38.1,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organcmnist",
"task_name": "Abdominal CT organs — coronal (C)",
"timestamp": "2026-08-01T19:28:20+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 64.100882
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.792228,
"auc": 0.978035,
"balanced_accuracy": 0.749776,
"ci95": [
0.7408,
0.75798
],
"n_test": 8827,
"score": 72.475324
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.722033,
"mean_agreement": 0.702833,
"mean_kappa": 0.664866,
"mean_retention": 0.781997,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.975,
0.835,
0.5808
],
"mean_kappa": 0.769526,
"mean_retention": 0.864886,
"retention_by_severity": [
1,
0.9617,
0.6329
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.785,
0.6467,
0.5025
],
"mean_kappa": 0.601249,
"mean_retention": 0.744786,
"retention_by_severity": [
0.9155,
0.7387,
0.5801
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8858,
0.7158,
0.56
],
"mean_kappa": 0.68117,
"mean_retention": 0.774015,
"retention_by_severity": [
0.948,
0.7799,
0.5942
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.6308,
0.5342,
0.4592
],
"mean_kappa": 0.490679,
"mean_retention": 0.629866,
"retention_by_severity": [
0.7242,
0.6355,
0.5299
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.93,
0.8383,
0.6633
],
"mean_kappa": 0.781708,
"mean_retention": 0.89643,
"retention_by_severity": [
0.9888,
0.9237,
0.7768
],
"scored": true
}
},
"score": 78.199663,
"worst_retention": 0.529926
},
"calibration": {
"accuracy": 0.792228,
"brier": 0.296927,
"ece": 0.092168,
"mean_confidence": 0.884397,
"overconfidence": 0.092168,
"score": 53.915875
},
"corruption": {
"clean_reference_cc": 0.722033,
"mean_agreement": 0.629008,
"mean_kappa": 0.574335,
"mean_retention": 0.651781,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.8208,
0.5192,
0.2575
],
"mean_kappa": 0.469462,
"mean_retention": 0.538917,
"retention_by_severity": [
0.8961,
0.5317,
0.189
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.6992,
0.3283,
0.1858
],
"mean_kappa": 0.34016,
"mean_retention": 0.419347,
"retention_by_severity": [
0.7862,
0.3269,
0.1449
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.7217,
0.4333,
0.3125
],
"mean_kappa": 0.439614,
"mean_retention": 0.559454,
"retention_by_severity": [
0.8261,
0.5094,
0.3428
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9417,
0.8183,
0.3975
],
"mean_kappa": 0.655566,
"mean_retention": 0.728886,
"retention_by_severity": [
0.9772,
0.887,
0.3225
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.85,
0.4333,
0.6625
],
"mean_kappa": 0.596312,
"mean_retention": 0.705502,
"retention_by_severity": [
0.9624,
0.4215,
0.7325
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9883,
0.98,
0.94
],
"mean_kappa": 0.96474,
"mean_retention": 0.989163,
"retention_by_severity": [
1,
0.9991,
0.9684
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.8683,
0.7117,
0.3392
],
"mean_kappa": 0.554494,
"mean_retention": 0.621202,
"retention_by_severity": [
0.9398,
0.7072,
0.2166
],
"scored": true
}
},
"score": 65.178139,
"worst_retention": 0.144947
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.724753,
"mean_retention": 0.406912,
"per_budget": {
"n100": {
"chance_corrected": 0.655458,
"n_labels": 1100,
"retention": 0.904388
},
"n20": {
"chance_corrected": 0.229275,
"n_labels": 220,
"retention": 0.316349
},
"n5": {
"chance_corrected": 0,
"n_labels": 55,
"retention": 0
}
},
"score": 40.691244
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 37.6082,
"worst_class_recall": 0.432802
},
"uncertainty": {
"full_accuracy": 0.792228,
"risk_coverage_auc": 0.938391,
"score": 93.222958,
"selective_acc_at_50": 0.973267,
"selective_acc_at_80": 0.876097
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 13932,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 8827,
"runtime_s": 25.4,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organsmnist",
"task_name": "Abdominal CT organs — sagittal (S)",
"timestamp": "2026-08-01T19:30:56+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 56.68336
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.848607,
"auc": 0.965322,
"balanced_accuracy": 0.811026,
"ci95": [
0.80097,
0.820898
],
"n_test": 7180,
"score": 78.7404
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.796619,
"mean_agreement": 0.484333,
"mean_kappa": 0.419403,
"mean_retention": 0.464125,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.8217,
0.2375,
0.125
],
"mean_kappa": 0.332934,
"mean_retention": 0.379424,
"retention_by_severity": [
0.8731,
0.2187,
0.0464
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.725,
0.675,
0.5867
],
"mean_kappa": 0.611504,
"mean_retention": 0.652849,
"retention_by_severity": [
0.7339,
0.664,
0.5607
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8717,
0.6458,
0.4692
],
"mean_kappa": 0.612889,
"mean_retention": 0.666107,
"retention_by_severity": [
0.9031,
0.6423,
0.4529
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.5217,
0.45,
0.3817
],
"mean_kappa": 0.370622,
"mean_retention": 0.407381,
"retention_by_severity": [
0.5001,
0.4109,
0.3112
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.3775,
0.2308,
0.1458
],
"mean_kappa": 0.169064,
"mean_retention": 0.214865,
"retention_by_severity": [
0.3973,
0.1836,
0.0638
],
"scored": true
}
},
"score": 46.412525,
"worst_retention": 0.04645
},
"calibration": {
"accuracy": 0.848607,
"brier": 0.24416,
"ece": 0.077413,
"mean_confidence": 0.92602,
"overconfidence": 0.077413,
"score": 61.2935
},
"corruption": {
"clean_reference_cc": 0.796619,
"mean_agreement": 0.413373,
"mean_kappa": 0.344685,
"mean_retention": 0.382924,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.4442,
0.2383,
0.2167
],
"mean_kappa": 0.219092,
"mean_retention": 0.258912,
"retention_by_severity": [
0.4543,
0.1678,
0.1547
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8142,
0.5933,
0.3567
],
"mean_kappa": 0.525403,
"mean_retention": 0.589015,
"retention_by_severity": [
0.867,
0.5949,
0.3052
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.63,
0.3717,
0.3275
],
"mean_kappa": 0.35723,
"mean_retention": 0.389592,
"retention_by_severity": [
0.6356,
0.2997,
0.2336
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.2575,
0.1092,
0.0825
],
"mean_kappa": 0.072155,
"mean_retention": 0.102473,
"retention_by_severity": [
0.2592,
0.0241,
0.0241
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.7758,
0.4992,
0.5567
],
"mean_kappa": 0.549219,
"mean_retention": 0.573188,
"retention_by_severity": [
0.785,
0.4159,
0.5186
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.955,
0.8067,
0.3692
],
"mean_kappa": 0.677348,
"mean_retention": 0.737653,
"retention_by_severity": [
0.9832,
0.8677,
0.3621
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.1158,
0.0858,
0.075
],
"mean_kappa": 0.012349,
"mean_retention": 0.029633,
"retention_by_severity": [
0.0414,
0.0146,
0.0329
],
"scored": true
}
},
"score": 38.292373,
"worst_retention": 0.014553
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.787404,
"mean_retention": 0.420779,
"per_budget": {
"n100": {
"chance_corrected": 0.604068,
"n_labels": 900,
"retention": 0.767164
},
"n20": {
"chance_corrected": 0.320459,
"n_labels": 180,
"retention": 0.406981
},
"n5": {
"chance_corrected": 0.069442,
"n_labels": 45,
"retention": 0.088191
}
},
"score": 42.077896
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 38.539192,
"worst_class_recall": 0.453682
},
"uncertainty": {
"full_accuracy": 0.848607,
"risk_coverage_auc": 0.944166,
"score": 93.718695,
"selective_acc_at_50": 0.953203,
"selective_acc_at_80": 0.913301
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 89996,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "Histopathology",
"n_classes": 9,
"n_test": 7180,
"runtime_s": 40.3,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "pathmnist",
"task_name": "Colorectal pathology (9-class tissue)",
"timestamp": "2026-08-01T19:42:24+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 61.913727
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.900641,
"auc": 0.968606,
"balanced_accuracy": 0.874359,
"ci95": [
0.849435,
0.902377
],
"n_test": 624,
"score": 74.871795
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.748718,
"mean_agreement": 0.87938,
"mean_kappa": 0.680321,
"mean_retention": 0.727093,
"n_cases": 624,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9696,
0.9279,
0.6699
],
"mean_kappa": 0.722389,
"mean_retention": 0.903349,
"retention_by_severity": [
1,
1,
0.71
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.859,
0.7468,
0.6987
],
"mean_kappa": 0.274432,
"mean_retention": 0.247336,
"retention_by_severity": [
0.5708,
0.1712,
0
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9744,
0.9439,
0.8974
],
"mean_kappa": 0.84745,
"mean_retention": 0.883942,
"retention_by_severity": [
0.9977,
0.9053,
0.7489
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9054,
0.8942,
0.8077
],
"mean_kappa": 0.638753,
"mean_retention": 0.600837,
"retention_by_severity": [
0.7272,
0.6872,
0.3881
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9824,
0.9647,
0.9487
],
"mean_kappa": 0.918579,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
}
},
"score": 72.709285,
"worst_retention": 0
},
"calibration": {
"accuracy": 0.900641,
"brier": 0.162961,
"ece": 0.070922,
"mean_confidence": 0.971563,
"overconfidence": 0.070922,
"score": 64.538792
},
"corruption": {
"clean_reference_cc": 0.748718,
"mean_agreement": 0.747634,
"mean_kappa": 0.487033,
"mean_retention": 0.553001,
"n_cases": 624,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9487,
0.8494,
0.7228
],
"mean_kappa": 0.528407,
"mean_retention": 0.531583,
"retention_by_severity": [
0.9155,
0.5685,
0.1107
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.9295,
0.8045,
0.6987
],
"mean_kappa": 0.417033,
"mean_retention": 0.396499,
"retention_by_severity": [
0.8128,
0.3767,
0
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9343,
0.8061,
0.7019
],
"mean_kappa": 0.428219,
"mean_retention": 0.407915,
"retention_by_severity": [
0.8299,
0.3824,
0.0114
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9119,
0.4615,
0.3013
],
"mean_kappa": 0.319598,
"mean_retention": 0.4414,
"retention_by_severity": [
1,
0.3242,
0
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9375,
0.8974,
0.8221
],
"mean_kappa": 0.752833,
"mean_retention": 0.950152,
"retention_by_severity": [
0.8505,
1,
1
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.984,
0.9744,
0.9006
],
"mean_kappa": 0.895122,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.5048,
0.3077,
0.3013
],
"mean_kappa": 0.068015,
"mean_retention": 0.143455,
"retention_by_severity": [
0.4167,
0.0137,
0
],
"scored": true
}
},
"score": 55.300065,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.748718,
"mean_retention": 0,
"per_budget": {
"n100": {
"chance_corrected": 0,
"n_labels": 200,
"retention": 0
},
"n20": {
"chance_corrected": 0,
"n_labels": 40,
"retention": 0
},
"n5": {
"chance_corrected": 0,
"n_labels": 10,
"retention": 0
}
},
"score": 0
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 53.846154,
"worst_class_recall": 0.769231
},
"uncertainty": {
"full_accuracy": 0.900641,
"risk_coverage_auc": 0.975418,
"score": 95.083525,
"selective_acc_at_50": 0.987179,
"selective_acc_at_80": 0.963928
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 4708,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Chest X-ray",
"n_classes": 2,
"n_test": 624,
"runtime_s": 9.1,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "pneumoniamnist",
"task_name": "Paediatric chest X-ray (pneumonia)",
"timestamp": "2026-08-01T19:16:26+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 54.747757
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.52,
"auc": 0.716715,
"balanced_accuracy": 0.33791,
"ci95": [
0.307025,
0.373363
],
"n_test": 400,
"score": 17.238807
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.172388,
"mean_agreement": 0.8015,
"mean_kappa": 0.706372,
"mean_retention": 0.893282,
"n_cases": 400,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.955,
0.9025,
0.8325
],
"mean_kappa": 0.84486,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.935,
0.8925,
0.805
],
"mean_kappa": 0.815123,
"mean_retention": 0.930762,
"retention_by_severity": [
0.9434,
0.9036,
0.9453
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.83,
0.6325,
0.5125
],
"mean_kappa": 0.485451,
"mean_retention": 0.856788,
"retention_by_severity": [
0.9731,
0.9127,
0.6846
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9625,
0.9475,
0.9325
],
"mean_kappa": 0.919558,
"mean_retention": 0.993513,
"retention_by_severity": [
0.9805,
1,
1
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.785,
0.6075,
0.49
],
"mean_kappa": 0.46687,
"mean_retention": 0.685347,
"retention_by_severity": [
0.8359,
0.847,
0.3732
],
"scored": true
}
},
"score": 89.328189,
"worst_retention": 0.373159
},
"calibration": {
"accuracy": 0.52,
"brier": 0.590948,
"ece": 0.039781,
"mean_confidence": 0.495429,
"overconfidence": -0.024571,
"score": 80.109489
},
"corruption": {
"clean_reference_cc": 0.172388,
"mean_agreement": 0.813333,
"mean_kappa": 0.72937,
"mean_retention": 0.943214,
"n_cases": 400,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.8025,
0.675,
0.3025
],
"mean_kappa": 0.443145,
"mean_retention": 0.71089,
"retention_by_severity": [
0.9899,
0.89,
0.2528
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8425,
0.6825,
0.62
],
"mean_kappa": 0.580462,
"mean_retention": 0.922481,
"retention_by_severity": [
0.949,
0.9314,
0.887
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9675,
0.9425,
0.9075
],
"mean_kappa": 0.906883,
"mean_retention": 0.969124,
"retention_by_severity": [
0.9648,
0.9954,
0.9472
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9125,
0.8,
0.67
],
"mean_kappa": 0.693887,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.96,
0.9525,
0.9225
],
"mean_kappa": 0.916338,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9725,
0.93,
0.865
],
"mean_kappa": 0.88401,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.8675,
0.7575,
0.7275
],
"mean_kappa": 0.680863,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
}
},
"score": 94.321353,
"worst_retention": 0.252787
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.172388,
"mean_retention": 0.361393,
"per_budget": {
"n100": {
"chance_corrected": 0.212155,
"n_labels": 466,
"retention": 1
},
"n20": {
"chance_corrected": 0.007184,
"n_labels": 100,
"retention": 0.041673
},
"n5": {
"chance_corrected": 0.007328,
"n_labels": 25,
"retention": 0.042506
}
},
"score": 36.139308
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0
},
"uncertainty": {
"full_accuracy": 0.52,
"risk_coverage_auc": 0.717889,
"score": 64.736128,
"selective_acc_at_50": 0.705,
"selective_acc_at_80": 0.5875
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 1080,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Fundus",
"n_classes": 5,
"n_test": 400,
"runtime_s": 4.3,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "retinamnist",
"task_name": "Fundus photography (DR grade)",
"timestamp": "2026-08-01T19:13:18+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 50.501494
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.592809,
"auc": 0.892541,
"balanced_accuracy": 0.489413,
"ci95": [
0.484492,
0.4951
],
"n_test": 47280,
"score": 41.647253
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.395455,
"mean_agreement": 0.4975,
"mean_kappa": 0.375259,
"mean_retention": 0.583909,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.5958,
0.3817,
0.2483
],
"mean_kappa": 0.306152,
"mean_retention": 0.51586,
"retention_by_severity": [
0.7317,
0.486,
0.3298
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.7767,
0.6917,
0.5567
],
"mean_kappa": 0.580545,
"mean_retention": 0.792985,
"retention_by_severity": [
0.9192,
0.824,
0.6358
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.5317,
0.38,
0.3442
],
"mean_kappa": 0.263092,
"mean_retention": 0.442412,
"retention_by_severity": [
0.6699,
0.4059,
0.2515
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.6908,
0.6067,
0.5575
],
"mean_kappa": 0.527708,
"mean_retention": 0.798538,
"retention_by_severity": [
0.8877,
0.7682,
0.7398
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.4092,
0.3642,
0.3275
],
"mean_kappa": 0.1988,
"mean_retention": 0.36975,
"retention_by_severity": [
0.5264,
0.341,
0.2419
],
"scored": true
}
},
"score": 58.390874,
"worst_retention": 0.241852
},
"calibration": {
"accuracy": 0.592809,
"brier": 0.540139,
"ece": 0.012568,
"mean_confidence": 0.605165,
"overconfidence": 0.012356,
"score": 93.716198
},
"corruption": {
"clean_reference_cc": 0.395455,
"mean_agreement": 0.493889,
"mean_kappa": 0.367043,
"mean_retention": 0.482268,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.5067,
0.3858,
0.3633
],
"mean_kappa": 0.235683,
"mean_retention": 0.361514,
"retention_by_severity": [
0.5582,
0.2758,
0.2506
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.6233,
0.4725,
0.4067
],
"mean_kappa": 0.329419,
"mean_retention": 0.328757,
"retention_by_severity": [
0.5515,
0.2867,
0.148
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.75,
0.5667,
0.4742
],
"mean_kappa": 0.501726,
"mean_retention": 0.744164,
"retention_by_severity": [
0.9487,
0.7357,
0.5481
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.385,
0.2558,
0.1733
],
"mean_kappa": 0.131399,
"mean_retention": 0.2462,
"retention_by_severity": [
0.4579,
0.274,
0.0067
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.8408,
0.6317,
0.6783
],
"mean_kappa": 0.634637,
"mean_retention": 0.682928,
"retention_by_severity": [
0.8728,
0.5901,
0.5859
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.8383,
0.7017,
0.3867
],
"mean_kappa": 0.545015,
"mean_retention": 0.656888,
"retention_by_severity": [
0.9174,
0.7146,
0.3387
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.4117,
0.3083,
0.2108
],
"mean_kappa": 0.191426,
"mean_retention": 0.355426,
"retention_by_severity": [
0.5618,
0.3994,
0.1051
],
"scored": true
}
},
"score": 48.226827,
"worst_retention": 0.006732
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.416473,
"mean_retention": 0.306291,
"per_budget": {
"n100": {
"chance_corrected": 0.325091,
"n_labels": 800,
"retention": 0.780583
},
"n20": {
"chance_corrected": 0.057595,
"n_labels": 160,
"retention": 0.138291
},
"n5": {
"chance_corrected": 0,
"n_labels": 40,
"retention": 0
}
},
"score": 30.629144
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 8.43836,
"worst_class_recall": 0.198836
},
"uncertainty": {
"full_accuracy": 0.592809,
"risk_coverage_auc": 0.75998,
"score": 72.569134,
"selective_acc_at_50": 0.75956,
"selective_acc_at_80": 0.656647
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 165466,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "Microscopy",
"n_classes": 8,
"n_test": 47280,
"runtime_s": 41.8,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "tissuemnist",
"task_name": "Kidney cortex cells (8-class)",
"timestamp": "2026-08-01T19:39:02+00:00"
}
],
"heldout": null,
"sources": [
{
"path": "results/cross_site/organ_ct__cnn_scratch.json",
"sha256": "2cb33bd19620e036d679332fd717024b5b74f1cae25c506645558f82ba8018ac"
},
{
"path": "results/heldout/audit_report.json",
"sha256": "df460b6d91a6512dfa1e7dc650637fce2b4dd3a1ec32e03446bf6e11f9c597d2"
},
{
"path": "results/heldout/cycle.json",
"sha256": "e012b8978457b51b5c31d18d14d2cc92f100731897523306acae19934f943502"
},
{
"path": "results/heldout/manifests/gen1.seal.json",
"sha256": "91a7adfa29870a0719125cdd48f6dcb149f1f6485ac1cfa104c6e00057ad9b81"
},
{
"path": "results/heldout/manifests/gen2.seal.json",
"sha256": "f61470918a09cf39e074ef14fb7ade4ac3a13d36d9385f18cf9b92e4b5329547"
},
{
"path": "results/heldout/manifests/public.seal.json",
"sha256": "1a4fd930d2e781ac7163483990f15d6694e28e507d1b3ca009ff09395e47497c"
},
{
"path": "results/records/bloodmnist__cnn_scratch.json",
"sha256": "658e3e5f7cfa4ad5fd6c7b39ba0aeafd63cd24b954494a2ea543b80339ff90ee"
},
{
"path": "results/records/breastmnist__cnn_scratch.json",
"sha256": "2212aec836e7f76f734e4219f2440104ddf80ee2db0595211786509d77fc9ad6"
},
{
"path": "results/records/dermamnist__cnn_scratch.json",
"sha256": "ecafb8abee712c7ae14cefcbc3e079ecdbbec6ca0eaf1915cd70d66b033443c4"
},
{
"path": "results/records/octmnist__cnn_scratch.json",
"sha256": "051221f9e35353e03d86cf3fc99a668fc38a9dde7a433eeee6c08430ff7577d7"
},
{
"path": "results/records/organamnist__cnn_scratch.json",
"sha256": "ac309b752caab9b09114ef594183239dc8b5fdb3732f0e694a9bf2aef0a9e46d"
},
{
"path": "results/records/organcmnist__cnn_scratch.json",
"sha256": "b00ac3f602f5687f42d5649af03dea831ae0f80f650542f3bbc20a3766d12225"
},
{
"path": "results/records/organsmnist__cnn_scratch.json",
"sha256": "83a9069311350c3760a04eb18c472285d1d26facd163ea735ef6448af4187a95"
},
{
"path": "results/records/pathmnist__cnn_scratch.json",
"sha256": "19764268891a48ac331ae275acc81c85b16828b02614ee8f32abda10afd4d909"
},
{
"path": "results/records/pneumoniamnist__cnn_scratch.json",
"sha256": "e15163c863218cf6062262c8bfc0c3800e7a3b6995751ffb2802e5a204876209"
},
{
"path": "results/records/retinamnist__cnn_scratch.json",
"sha256": "702b5c2292a18b58c0d1deb42fee6ea2dc74f3c6afa7e10203958a5ec14b8b01"
},
{
"path": "results/records/tissuemnist__cnn_scratch.json",
"sha256": "792813023724858e90cb26cab9ff90c5a00925ee563ecf98bccb4e4c08ae4396"
}
]
},
"declared": null,
"declared_note": "No vendor declaration. This is an open reference baseline implemented by NakedSignal, not a commercial product: there is no vendor to assert an intended use or to sign an attestation, so the declared block is null rather than filled in on the vendor's behalf.",
"expiry_basis": "the sealed held-out generation this evidence cycle is anchored to (gen2, set hash ad30d214daa285c5...) is published to expire 2026-09-20; per spec, a passport lapses when its generation retires or its spec version is superseded, whichever comes first",
"expiry_utc": "2026-09-20T00:00:00Z",
"issued_utc": "2026-08-03T16:03:42Z",
"issuer": {
"algo": "Ed25519",
"key_id": "ns-passport-2026-07",
"name": "NakedSignal",
"public_key_b64": "0PBCC6mjzcppR5QdPcK7vP/AamB6MP8l1T4ypMYbsMw="
},
"observed": null,
"observed_note": "Not available: this requires continuous monitoring. Production history and drift are observed fields, and nothing is deployed, so there is nothing to observe. The object is present and null on purpose. The shape is the roadmap, not a claim.",
"passport_version": "0.1",
"signature": "XcWDZfObYGpNaSmO3H0fx5rwHAJishZmoA52xWxcKeJB6L13mJrPoUPs/cdaja2oE03MAiOz883Cq6cwJJuDAw==",
"spec_version": "MedEval-1 v0.1",
"subject": {
"behaviour_version": "1.0.0",
"code_fingerprint": "b444c02112cc",
"code_fingerprints": [
"b444c02112cc"
],
"family": "trained",
"model_id": "cnn_scratch",
"model_name": "Small CNN, trained from scratch",
"params": "3 conv blocks, 20 epochs, adam 1e-3, early stop on val"
}
}Canonicalisation: RFC 8785 (JCS)-compatible: UTF-8, object keys sorted by code point, no insignificant whitespace, ECMAScript number formatting. every float is rounded once at emit to 6 decimal places (Python round(), banker's rounding); integers are left exact. Signed over the whole document with the `signature` member removed, canonicalised as above and encoded UTF-8. The public key for ns-passport-2026-07 is 0PBCC6mjzcppR5QdPcK7vP/AamB6MP8l1T4ypMYbsMw= — see Governance.