Back to the consoleDocumentation
ResNet-18 (ImageNet) fine-tuned
LIVEThe signed evidence record for resnet18_ft. Everything below was read out of the passport file; the signature was checked when this page was built, by the same code the Verify button runs.
SIGNATURE VALIDexit 0signature valid, issued by NakedSignal under key ns-passport-2026-07, not expired
SubjectResNet-18 (ImageNet) fine-tuned
resnet18_ftConfigurationImageNet init, all layers, 12 epochs, adam 3e-4, 32px
Code fingerprintb444c02112cc
Spec versionMedEval-1 v0.1
Issued2026-08-03T16:03:42Z by NakedSignal
Expires2026-09-20T00:00:00Z — the sealed held-out generation this evidence cycle is anchored to (gen2, set hash ad30d214daa285c5...) is published to expire 2026-09-20; per spec, a passport lapses when its generation retires or its spec version is superseded, whichever comes first
key ns-passport-2026-07 · Ed25519
The check runs against the public key published on the governance page, using your browser's own Ed25519 implementation. The exit code shown is the exit code verify_passport.py returns for the same document.
Measured by NakedSignal
Computed in this repo by the MedEval-1 harness on data the model had never seen. Every number below was copied out of a file the harness wrote — 17 of them, each listed with its SHA-256 at the foot of this panel — and is printed exactly as it was signed, not re-rounded for display.
| Task | Domain | n test | accuracy | calibration | uncertainty | subgroup | corruption | acquisition | limited data | cost | spec sensitivity | Index |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bloodmnist | medicine | 3421 | 95.698671 | 88.473647 | 99.463088 | 91.315075 | 57.584867 | 72.618391 | 61.950152 | n/a | n/a | 80.924678 |
| breastmnist | medicine | 156 | 65.66416 | 52.085287 | 85.59186 | 52.380952 | 50.754271 | 69.287532 | 61.959288 | n/a | n/a | 61.321474 |
| dermamnist | medicine | 2005 | 45.884991 | 24.552724 | 89.732718 | 13.768116 | 62.778149 | 78.059652 | 68.829476 | n/a | n/a | 52.489258 |
| octmnist | medicine | 1000 | 66.133333 | 0.372235 | 84.459686 | 14.666667 | 42.549923 | 53.965054 | 26.209677 | n/a | n/a | 42.151907 |
| organamnist | medicine | 17778 | 91.744011 | 79.38318 | 98.198688 | 73.299419 | 74.788629 | 86.290585 | 69.442539 | n/a | n/a | 82.515261 |
| organcmnist | medicine | 8216 | 91.82573 | 78.297436 | 99.063314 | 81.540578 | 79.217903 | 86.538199 | 67.673177 | n/a | n/a | 83.93641 |
| organsmnist | medicine | 8827 | 74.795833 | 35.806547 | 93.909408 | 42.61959 | 74.601834 | 86.208919 | 59.416501 | n/a | n/a | 67.326356 |
| pathmnist | medicine | 7180 | 86.983785 | 70.724941 | 97.827489 | 56.710214 | 49.921314 | 64.203061 | 73.489364 | n/a | n/a | 70.422162 |
| pneumoniamnist | medicine | 624 | 74.871795 | 62.353917 | 94.400981 | 51.282051 | 54.914112 | 70.182648 | 73.782344 | n/a | n/a | 67.356934 |
| retinamnist | medicine | 400 | 18.657326 | 0 | 60.75074 | 0 | 71.029093 | 77.092619 | 93.237916 | n/a | n/a | 42.304494 |
| tissuemnist | medicine | 47280 | 38.891855 | 0 | 69.326928 | 2.757341 | 51.577118 | 63.241067 | 41.914309 | n/a | n/a | 37.778889 |
Cross-site retention · organ_ct
retention is absolute retained skill (chance-corrected), per spec 2.2.1; the diagonal is same-site, every off-diagonal cell is train-on-row, test-on-column
Sitesorganamnist, organcmnist, organsmnist
Mean retention0.604513
Worst retention0.292273
Private held-out track
this subject was not submitted to a sealed generation.Known limitations
Subgroup metadataabsent on 11 of 11 corpora, so the subgroup score falls back to the worst diagnostic class. That absence is a disclosure, not a pass.
Training set cappedyes on at least one task — see the raw JSON for which
Observed in deploymentnothing — see the third panel
Audit chain
Bundles re-derived48 of 48
Ledger100 entries, head 6b7b371ff650a8ed…, chain intact
Re-run it yourself
python src/heldout/audit.py 1f66c7719ab3943c6fcc17 source files, with hashes
| results/cross_site/organ_ct__resnet18_ft.json | d27a1ee962818c0b7d1ba43d05d587c9ef3b38ae16c1dd2febb2f5f2a34c9332 |
| results/heldout/audit_report.json | df460b6d91a6512dfa1e7dc650637fce2b4dd3a1ec32e03446bf6e11f9c597d2 |
| results/heldout/cycle.json | e012b8978457b51b5c31d18d14d2cc92f100731897523306acae19934f943502 |
| results/heldout/manifests/gen1.seal.json | 91a7adfa29870a0719125cdd48f6dcb149f1f6485ac1cfa104c6e00057ad9b81 |
| results/heldout/manifests/gen2.seal.json | f61470918a09cf39e074ef14fb7ade4ac3a13d36d9385f18cf9b92e4b5329547 |
| results/heldout/manifests/public.seal.json | 1a4fd930d2e781ac7163483990f15d6694e28e507d1b3ca009ff09395e47497c |
| results/records/bloodmnist__resnet18_ft.json | cbf774ba932c41fcd50ac1eb2f3e807f5b564fd505b351efaa88012d6c4a2955 |
| results/records/breastmnist__resnet18_ft.json | 5852fd51f132615d49cfa9b0b363a02988fd415aedaf5561fb12ec8609154cce |
| results/records/dermamnist__resnet18_ft.json | a5c10c8831bf4f8da07c3daf21966725ba687e825116e1e9526ffac8c833c14e |
| results/records/octmnist__resnet18_ft.json | 4639d3534e8fe9353280dc9054bfd4eb3c01b4e553047e7dad33c20232566596 |
| results/records/organamnist__resnet18_ft.json | 8c1b3c1ba85b480ec44526fdc6dd9e14b13753bedf04a548914ca80e0c2e299a |
| results/records/organcmnist__resnet18_ft.json | f1c3e1518f67b216d2624b78f101799c09432e6fb4118a4dd9bd458d93c57e01 |
| results/records/organsmnist__resnet18_ft.json | a750e1c522188b4f44e3afab039814caee04b40c4343047f5d984ca3fe6d39ec |
| results/records/pathmnist__resnet18_ft.json | 18c573007d941b9b8bf244ba7229199f0e2e3254dfdaf7a6aa9911f6fcb06db8 |
| results/records/pneumoniamnist__resnet18_ft.json | 84d35e6465d3b422e594567e90c4fa1ae0cf7346f7dfdb988cfd2e7d692c8754 |
| results/records/retinamnist__resnet18_ft.json | 615ed85d61c7a6a2e19e56bdfc384f616c0abf343a081be9b531ad9df9cbc977 |
| results/records/tissuemnist__resnet18_ft.json | 27f3dc9a75b577b253954268459487951f0d92eba8c79f21762f69580e4d9328 |
Declared by vendor
Five fields a vendor asserts about its own product — intended use, forbidden use, training cutoff, regulatory clearances, and who is personally attesting. NakedSignal never fills these in on a vendor's behalf.
No vendor declaration. This is an open reference baseline implemented by NakedSignal, not a commercial product: there is no vendor to assert an intended use or to sign an attestation, so the declared block is null rather than filled in on the vendor's behalf.
Observed in deployment
Production history and drift — how the model has actually behaved since it was deployed, on real traffic.
Not available: this requires continuous monitoring. Production history and drift are observed fields, and nothing is deployed, so there is nothing to observe. The object is present and null on purpose. The shape is the roadmap, not a claim.
Raw JSON — the whole signed document, 81,478 characters
{
"canonicalisation": {
"float_rounding": "every float is rounded once at emit to 6 decimal places (Python round(), banker's rounding); integers are left exact",
"form": "RFC 8785 (JCS)-compatible: UTF-8, object keys sorted by code point, no insignificant whitespace, ECMAScript number formatting",
"signed_over": "the whole document with the `signature` member removed, canonicalised as above and encoded UTF-8"
},
"computed": {
"audit": {
"audit_command": "python src/heldout/audit.py 1f66c7719ab3943c6fcc",
"bundles": 48,
"ledger_entries": 100,
"ledger_head": "6b7b371ff650a8edaf477490f2a5f9b043433f3ba9941989ee1962ed8f1b1d88",
"ledger_intact": true,
"rederived": 48
},
"cross_site": {
"code_fingerprint": "b444c02112cc",
"device": "mps",
"family": "organ_ct",
"matrix": {
"organamnist": {
"organamnist": {
"balanced_accuracy": 0.924946,
"chance_corrected": 0.91744,
"ece": 0.041234
},
"organcmnist": {
"balanced_accuracy": 0.666938,
"chance_corrected": 0.633632,
"ece": 0.196864,
"retention": 0.690652
},
"organsmnist": {
"balanced_accuracy": 0.334675,
"chance_corrected": 0.268143,
"ece": 0.466509,
"retention": 0.292273
}
},
"organcmnist": {
"organamnist": {
"balanced_accuracy": 0.70388,
"chance_corrected": 0.674268,
"ece": 0.212567,
"retention": 0.73429
},
"organcmnist": {
"balanced_accuracy": 0.925688,
"chance_corrected": 0.918257,
"ece": 0.043405
},
"organsmnist": {
"balanced_accuracy": 0.516183,
"chance_corrected": 0.467801,
"ece": 0.344888,
"retention": 0.509445
}
},
"organsmnist": {
"organamnist": {
"balanced_accuracy": 0.419567,
"chance_corrected": 0.361524,
"ece": 0.416774,
"retention": 0.483348
},
"organcmnist": {
"balanced_accuracy": 0.714481,
"chance_corrected": 0.685929,
"ece": 0.150284,
"retention": 0.917069
},
"organsmnist": {
"balanced_accuracy": 0.770871,
"chance_corrected": 0.747958,
"ece": 0.128387
}
}
},
"mean_retention": 0.604513,
"note": "retention is absolute retained skill (chance-corrected), per spec 2.2.1; the diagonal is same-site, every off-diagonal cell is train-on-row, test-on-column",
"score": 60.451278,
"site_labels": {
"organamnist": "axial",
"organcmnist": "coronal",
"organsmnist": "sagittal"
},
"sites": [
"organamnist",
"organcmnist",
"organsmnist"
],
"spec_version": "MedEval-1 v0.1",
"worst_retention": 0.292273
},
"evaluations": [
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 80.924678
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.965799,
"auc": 0.997586,
"balanced_accuracy": 0.962363,
"ci95": [
0.955653,
0.968847
],
"n_test": 3421,
"score": 95.698671
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.957065,
"mean_agreement": 0.759611,
"mean_kappa": 0.721014,
"mean_retention": 0.726184,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9675,
0.8917,
0.6442
],
"mean_kappa": 0.804001,
"mean_retention": 0.789404,
"retention_by_severity": [
0.9707,
0.8423,
0.5551
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8725,
0.8117,
0.65
],
"mean_kappa": 0.739045,
"mean_retention": 0.719569,
"retention_by_severity": [
0.8542,
0.7684,
0.5361
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9092,
0.6475,
0.375
],
"mean_kappa": 0.59362,
"mean_retention": 0.604638,
"retention_by_severity": [
0.9022,
0.5822,
0.3296
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.81,
0.6892,
0.6408
],
"mean_kappa": 0.666465,
"mean_retention": 0.694871,
"retention_by_severity": [
0.797,
0.6727,
0.6149
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9517,
0.8717,
0.6617
],
"mean_kappa": 0.80194,
"mean_retention": 0.822437,
"retention_by_severity": [
0.9504,
0.8551,
0.6618
],
"scored": true
}
},
"score": 72.618391,
"worst_retention": 0.32957
},
"calibration": {
"accuracy": 0.965799,
"brier": 0.056266,
"ece": 0.023053,
"mean_confidence": 0.98676,
"overconfidence": 0.02096,
"score": 88.473647
},
"corruption": {
"clean_reference_cc": 0.957065,
"mean_agreement": 0.640714,
"mean_kappa": 0.584373,
"mean_retention": 0.575849,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.8525,
0.5675,
0.4717
],
"mean_kappa": 0.561246,
"mean_retention": 0.495227,
"retention_by_severity": [
0.7824,
0.4119,
0.2914
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.7167,
0.4042,
0.2375
],
"mean_kappa": 0.3635,
"mean_retention": 0.328655,
"retention_by_severity": [
0.6017,
0.2906,
0.0936
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.86,
0.6292,
0.5208
],
"mean_kappa": 0.615259,
"mean_retention": 0.636212,
"retention_by_severity": [
0.8458,
0.6233,
0.4395
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.8958,
0.6433,
0.2275
],
"mean_kappa": 0.533879,
"mean_retention": 0.531142,
"retention_by_severity": [
0.8513,
0.565,
0.1772
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.8975,
0.6258,
0.7217
],
"mean_kappa": 0.703941,
"mean_retention": 0.70831,
"retention_by_severity": [
0.9138,
0.4786,
0.7325
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9933,
0.975,
0.9125
],
"mean_kappa": 0.953396,
"mean_retention": 0.96309,
"retention_by_severity": [
0.9988,
0.9807,
0.9098
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.7208,
0.3917,
0.19
],
"mean_kappa": 0.359387,
"mean_retention": 0.368305,
"retention_by_severity": [
0.646,
0.3238,
0.1351
],
"scored": true
}
},
"score": 57.584867,
"worst_retention": 0.093633
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.956987,
"mean_retention": 0.619502,
"per_budget": {
"n100": {
"chance_corrected": 0.831674,
"n_labels": 800,
"retention": 0.869055
},
"n20": {
"chance_corrected": 0.612066,
"n_labels": 160,
"retention": 0.639576
},
"n5": {
"chance_corrected": 0.334824,
"n_labels": 40,
"retention": 0.349873
}
},
"score": 61.950152
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 91.315075,
"worst_class_recall": 0.924007
},
"uncertainty": {
"full_accuracy": 0.965799,
"risk_coverage_auc": 0.995302,
"score": 99.463088,
"selective_acc_at_50": 0.997076,
"selective_acc_at_80": 0.996346
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 11959,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Microscopy",
"n_classes": 8,
"n_test": 3421,
"runtime_s": 80.3,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "bloodmnist",
"task_name": "Peripheral blood cells (8-class)",
"timestamp": "2026-08-01T19:19:45+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 61.321474
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.858974,
"auc": 0.899749,
"balanced_accuracy": 0.828321,
"ci95": [
0.762419,
0.884729
],
"n_test": 156,
"score": 65.66416
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.656642,
"mean_agreement": 0.75641,
"mean_kappa": 0.504281,
"mean_retention": 0.692875,
"n_cases": 156,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.8974,
0.8782,
0.7756
],
"mean_kappa": 0.588856,
"mean_retention": 0.737913,
"retention_by_severity": [
1,
0.8531,
0.3607
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.6667,
0.5833,
0.5833
],
"mean_kappa": 0.317691,
"mean_retention": 0.613232,
"retention_by_severity": [
0.7519,
0.5553,
0.5324
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9167,
0.8718,
0.7756
],
"mean_kappa": 0.684894,
"mean_retention": 0.908397,
"retention_by_severity": [
0.9027,
0.958,
0.8645
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.5705,
0.5833,
0.6026
],
"mean_kappa": 0.273796,
"mean_retention": 0.48855,
"retention_by_severity": [
0.4027,
0.5363,
0.5267
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9551,
0.8846,
0.8013
],
"mean_kappa": 0.656169,
"mean_retention": 0.716285,
"retention_by_severity": [
0.9905,
0.7405,
0.4179
],
"scored": true
}
},
"score": 69.287532,
"worst_retention": 0.360687
},
"calibration": {
"accuracy": 0.858974,
"brier": 0.224907,
"ece": 0.095829,
"mean_confidence": 0.934436,
"overconfidence": 0.075462,
"score": 52.085287
},
"corruption": {
"clean_reference_cc": 0.656642,
"mean_agreement": 0.76862,
"mean_kappa": 0.408639,
"mean_retention": 0.507543,
"n_cases": 156,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9231,
0.8333,
0.7949
],
"mean_kappa": 0.559995,
"mean_retention": 0.515267,
"retention_by_severity": [
0.813,
0.4504,
0.2824
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8397,
0.6859,
0.5641
],
"mean_kappa": 0.404899,
"mean_retention": 0.552163,
"retention_by_severity": [
0.7156,
0.6698,
0.271
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.641,
0.6026,
0.6603
],
"mean_kappa": 0.298292,
"mean_retention": 0.417939,
"retention_by_severity": [
0.5496,
0.5267,
0.1775
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.8654,
0.7564,
0.7244
],
"mean_kappa": 0.285182,
"mean_retention": 0.302799,
"retention_by_severity": [
0.6317,
0.2405,
0.0363
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.8269,
0.7756,
0.7179
],
"mean_kappa": 0.417221,
"mean_retention": 0.728372,
"retention_by_severity": [
1,
0.8149,
0.3702
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9167,
0.8974,
0.8269
],
"mean_kappa": 0.679404,
"mean_retention": 0.783715,
"retention_by_severity": [
0.9599,
0.9046,
0.4866
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.7949,
0.7692,
0.7244
],
"mean_kappa": 0.215483,
"mean_retention": 0.252545,
"retention_by_severity": [
0.4313,
0.2901,
0.0363
],
"scored": true
}
},
"score": 50.754271,
"worst_retention": 0.03626
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.656642,
"mean_retention": 0.619593,
"per_budget": {
"n100": {
"chance_corrected": 0.607769,
"n_labels": 200,
"retention": 0.925573
},
"n20": {
"chance_corrected": 0.281955,
"n_labels": 40,
"retention": 0.429389
},
"n5": {
"chance_corrected": 0.330827,
"n_labels": 10,
"retention": 0.503817
}
},
"score": 61.959288
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 52.380952,
"worst_class_recall": 0.761905
},
"uncertainty": {
"full_accuracy": 0.858974,
"risk_coverage_auc": 0.927959,
"score": 85.59186,
"selective_acc_at_50": 0.935897,
"selective_acc_at_80": 0.904
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 546,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Ultrasound",
"n_classes": 2,
"n_test": 156,
"runtime_s": 5.9,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "breastmnist",
"task_name": "Breast ultrasound (malignancy)",
"timestamp": "2026-08-01T19:13:02+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 52.489258
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.774065,
"auc": 0.916365,
"balanced_accuracy": 0.536157,
"ci95": [
0.496176,
0.580822
],
"n_test": 2005,
"score": 45.884991
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.461465,
"mean_agreement": 0.828833,
"mean_kappa": 0.614523,
"mean_retention": 0.780597,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.8858,
0.8042,
0.7108
],
"mean_kappa": 0.549391,
"mean_retention": 0.760856,
"retention_by_severity": [
0.9602,
0.7386,
0.5837
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.9008,
0.8517,
0.8217
],
"mean_kappa": 0.672393,
"mean_retention": 0.90288,
"retention_by_severity": [
0.9563,
0.8965,
0.8558
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8858,
0.8017,
0.7467
],
"mean_kappa": 0.581166,
"mean_retention": 0.568579,
"retention_by_severity": [
0.8435,
0.5451,
0.3171
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.87,
0.8392,
0.8317
],
"mean_kappa": 0.631003,
"mean_retention": 0.776453,
"retention_by_severity": [
0.8911,
0.7512,
0.6871
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.8983,
0.8292,
0.755
],
"mean_kappa": 0.638662,
"mean_retention": 0.894214,
"retention_by_severity": [
1,
0.9266,
0.756
],
"scored": true
}
},
"score": 78.059652,
"worst_retention": 0.317126
},
"calibration": {
"accuracy": 0.774065,
"brier": 0.362398,
"ece": 0.150895,
"mean_confidence": 0.924843,
"overconfidence": 0.150778,
"score": 24.552724
},
"corruption": {
"clean_reference_cc": 0.461465,
"mean_agreement": 0.732421,
"mean_kappa": 0.455349,
"mean_retention": 0.627781,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.865,
0.7558,
0.7125
],
"mean_kappa": 0.508637,
"mean_retention": 0.723646,
"retention_by_severity": [
0.9391,
0.682,
0.5498
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8708,
0.775,
0.6767
],
"mean_kappa": 0.553698,
"mean_retention": 0.729678,
"retention_by_severity": [
0.9589,
0.8042,
0.426
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.8942,
0.8142,
0.7533
],
"mean_kappa": 0.513243,
"mean_retention": 0.64831,
"retention_by_severity": [
0.9966,
0.6413,
0.3071
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.81,
0.6908,
0.335
],
"mean_kappa": 0.30811,
"mean_retention": 0.503467,
"retention_by_severity": [
0.8125,
0.5232,
0.1747
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.8508,
0.7317,
0.675
],
"mean_kappa": 0.425565,
"mean_retention": 0.548799,
"retention_by_severity": [
0.9489,
0.12,
0.5775
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9383,
0.8642,
0.7042
],
"mean_kappa": 0.676945,
"mean_retention": 0.872895,
"retention_by_severity": [
1,
0.9221,
0.6966
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.725,
0.6342,
0.3042
],
"mean_kappa": 0.201244,
"mean_retention": 0.367675,
"retention_by_severity": [
0.5675,
0.3928,
0.1428
],
"scored": true
}
},
"score": 62.778149,
"worst_retention": 0.120019
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.45885,
"mean_retention": 0.688295,
"per_budget": {
"n100": {
"chance_corrected": 0.509039,
"n_labels": 679,
"retention": 1
},
"n20": {
"chance_corrected": 0.287882,
"n_labels": 140,
"retention": 0.627399
},
"n5": {
"chance_corrected": 0.20074,
"n_labels": 35,
"retention": 0.437485
}
},
"score": 68.829476
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 13.768116,
"worst_class_recall": 0.26087
},
"uncertainty": {
"full_accuracy": 0.774065,
"risk_coverage_auc": 0.911995,
"score": 89.732718,
"selective_acc_at_50": 0.941118,
"selective_acc_at_80": 0.858479
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 7007,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Dermatoscopy",
"n_classes": 7,
"n_test": 2005,
"runtime_s": 43.3,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "dermamnist",
"task_name": "Dermatoscopy (7-class skin lesion)",
"timestamp": "2026-08-01T19:17:48+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 42.151907
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.746,
"auc": 0.940383,
"balanced_accuracy": 0.746,
"ci95": [
0.724459,
0.765025
],
"n_test": 1000,
"score": 66.133333
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.661333,
"mean_agreement": 0.606467,
"mean_kappa": 0.416729,
"mean_retention": 0.539651,
"n_cases": 1000,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.646,
0.478,
0.318
],
"mean_kappa": 0.293405,
"mean_retention": 0.393145,
"retention_by_severity": [
0.7137,
0.2944,
0.1714
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.735,
0.614,
0.477
],
"mean_kappa": 0.39153,
"mean_retention": 0.417339,
"retention_by_severity": [
0.6573,
0.4294,
0.1653
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.829,
0.738,
0.658
],
"mean_kappa": 0.638524,
"mean_retention": 0.895833,
"retention_by_severity": [
1,
0.9234,
0.7641
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.519,
0.426,
0.423
],
"mean_kappa": 0.121143,
"mean_retention": 0.122312,
"retention_by_severity": [
0.2419,
0.0685,
0.0565
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.844,
0.736,
0.656
],
"mean_kappa": 0.639041,
"mean_retention": 0.869624,
"retention_by_severity": [
1,
0.9234,
0.6855
],
"scored": true
}
},
"score": 53.965054,
"worst_retention": 0.056452
},
"calibration": {
"accuracy": 0.746,
"brier": 0.445167,
"ece": 0.199256,
"mean_confidence": 0.94232,
"overconfidence": 0.19632,
"score": 0.372235
},
"corruption": {
"clean_reference_cc": 0.661333,
"mean_agreement": 0.535667,
"mean_kappa": 0.361867,
"mean_retention": 0.425499,
"n_cases": 1000,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.792,
0.695,
0.636
],
"mean_kappa": 0.553522,
"mean_retention": 0.566532,
"retention_by_severity": [
0.7097,
0.5242,
0.4657
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.835,
0.733,
0.602
],
"mean_kappa": 0.603025,
"mean_retention": 0.672715,
"retention_by_severity": [
0.8105,
0.6714,
0.5363
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.622,
0.423,
0.439
],
"mean_kappa": 0.187026,
"mean_retention": 0.192204,
"retention_by_severity": [
0.4395,
0.0524,
0.0847
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.471,
0.251,
0.154
],
"mean_kappa": 0.136847,
"mean_retention": 0.221102,
"retention_by_severity": [
0.4617,
0.1774,
0.0242
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.694,
0.243,
0.442
],
"mean_kappa": 0.257774,
"mean_retention": 0.325941,
"retention_by_severity": [
0.6875,
0.0726,
0.2177
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.889,
0.77,
0.58
],
"mean_kappa": 0.633787,
"mean_retention": 0.747312,
"retention_by_severity": [
0.9798,
0.8125,
0.4496
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.466,
0.306,
0.206
],
"mean_kappa": 0.16109,
"mean_retention": 0.252688,
"retention_by_severity": [
0.4456,
0.2339,
0.0786
],
"scored": true
}
},
"score": 42.549923,
"worst_retention": 0.024194
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.661333,
"mean_retention": 0.262097,
"per_budget": {
"n100": {
"chance_corrected": 0.406667,
"n_labels": 400,
"retention": 0.614919
},
"n20": {
"chance_corrected": 0.113333,
"n_labels": 80,
"retention": 0.171371
},
"n5": {
"chance_corrected": 0,
"n_labels": 20,
"retention": 0
}
},
"score": 26.209677
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 14.666667,
"worst_class_recall": 0.36
},
"uncertainty": {
"full_accuracy": 0.746,
"risk_coverage_auc": 0.883448,
"score": 84.459686,
"selective_acc_at_50": 0.896,
"selective_acc_at_80": 0.81
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 97477,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "OCT",
"n_classes": 4,
"n_test": 1000,
"runtime_s": 89.3,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "octmnist",
"task_name": "Retinal OCT (4-class)",
"timestamp": "2026-08-01T19:36:23+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 82.515261
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.933401,
"auc": 0.992954,
"balanced_accuracy": 0.924946,
"ci95": [
0.920286,
0.929167
],
"n_test": 17778,
"score": 91.744011
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.926804,
"mean_agreement": 0.832,
"mean_kappa": 0.814366,
"mean_retention": 0.862906,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.985,
0.9075,
0.6842
],
"mean_kappa": 0.843275,
"mean_retention": 0.879557,
"retention_by_severity": [
0.9925,
0.9362,
0.7099
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8692,
0.7492,
0.5667
],
"mean_kappa": 0.699847,
"mean_retention": 0.779116,
"retention_by_severity": [
0.9104,
0.8175,
0.6095
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9742,
0.9525,
0.9108
],
"mean_kappa": 0.939456,
"mean_retention": 0.972823,
"retention_by_severity": [
1,
0.9841,
0.9343
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.8067,
0.7033,
0.5233
],
"mean_kappa": 0.646243,
"mean_retention": 0.729861,
"retention_by_severity": [
0.8562,
0.7594,
0.5739
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9808,
0.9592,
0.9075
],
"mean_kappa": 0.943007,
"mean_retention": 0.953173,
"retention_by_severity": [
0.9872,
0.963,
0.9094
],
"scored": true
}
},
"score": 86.290585,
"worst_retention": 0.573945
},
"calibration": {
"accuracy": 0.933401,
"brier": 0.110082,
"ece": 0.041234,
"mean_confidence": 0.974316,
"overconfidence": 0.040915,
"score": 79.38318
},
"corruption": {
"clean_reference_cc": 0.926804,
"mean_agreement": 0.749484,
"mean_kappa": 0.725127,
"mean_retention": 0.747886,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.96,
0.88,
0.7225
],
"mean_kappa": 0.837575,
"mean_retention": 0.841518,
"retention_by_severity": [
0.9657,
0.8678,
0.6911
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.9117,
0.7375,
0.38
],
"mean_kappa": 0.644419,
"mean_retention": 0.692852,
"retention_by_severity": [
0.9304,
0.7687,
0.3794
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.835,
0.5833,
0.1967
],
"mean_kappa": 0.502835,
"mean_retention": 0.580102,
"retention_by_severity": [
0.8947,
0.6416,
0.204
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9625,
0.9067,
0.6533
],
"mean_kappa": 0.820209,
"mean_retention": 0.810775,
"retention_by_severity": [
0.9708,
0.8893,
0.5721
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.8983,
0.5025,
0.2442
],
"mean_kappa": 0.510017,
"mean_retention": 0.565911,
"retention_by_severity": [
0.9292,
0.5189,
0.2496
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9942,
0.9908,
0.9642
],
"mean_kappa": 0.98103,
"mean_retention": 0.98877,
"retention_by_severity": [
0.9968,
0.9984,
0.9711
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9308,
0.845,
0.64
],
"mean_kappa": 0.779806,
"mean_retention": 0.755275,
"retention_by_severity": [
0.9254,
0.8058,
0.5346
],
"scored": true
}
},
"score": 74.788629,
"worst_retention": 0.204019
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.91744,
"mean_retention": 0.694425,
"per_budget": {
"n100": {
"chance_corrected": 0.854444,
"n_labels": 1100,
"retention": 0.931335
},
"n20": {
"chance_corrected": 0.651795,
"n_labels": 220,
"retention": 0.71045
},
"n5": {
"chance_corrected": 0.405042,
"n_labels": 55,
"retention": 0.441492
}
},
"score": 69.442539
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 73.299419,
"worst_class_recall": 0.757267
},
"uncertainty": {
"full_accuracy": 0.933401,
"risk_coverage_auc": 0.983624,
"score": 98.198688,
"selective_acc_at_50": 0.988975,
"selective_acc_at_80": 0.98439
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 34561,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 17778,
"runtime_s": 100.8,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organamnist",
"task_name": "Abdominal CT organs — axial (A)",
"timestamp": "2026-08-01T19:33:43+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 83.93641
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.933909,
"auc": 0.995663,
"balanced_accuracy": 0.925688,
"ci95": [
0.919396,
0.932425
],
"n_test": 8216,
"score": 91.82573
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.910513,
"mean_agreement": 0.828389,
"mean_kappa": 0.809728,
"mean_retention": 0.865382,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9808,
0.9192,
0.6408
],
"mean_kappa": 0.828897,
"mean_retention": 0.860824,
"retention_by_severity": [
0.9976,
0.9489,
0.636
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.865,
0.755,
0.5608
],
"mean_kappa": 0.698549,
"mean_retention": 0.800407,
"retention_by_severity": [
0.917,
0.8365,
0.6477
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9767,
0.9433,
0.8958
],
"mean_kappa": 0.930706,
"mean_retention": 0.956748,
"retention_by_severity": [
0.9922,
0.9641,
0.914
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.78,
0.6933,
0.5225
],
"mean_kappa": 0.630983,
"mean_retention": 0.723976,
"retention_by_severity": [
0.8446,
0.7419,
0.5854
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9858,
0.97,
0.9367
],
"mean_kappa": 0.959507,
"mean_retention": 0.984955,
"retention_by_severity": [
1,
0.9922,
0.9627
],
"scored": true
}
},
"score": 86.538199,
"worst_retention": 0.585422
},
"calibration": {
"accuracy": 0.933909,
"brier": 0.106419,
"ece": 0.043405,
"mean_confidence": 0.977304,
"overconfidence": 0.043395,
"score": 78.297436
},
"corruption": {
"clean_reference_cc": 0.910513,
"mean_agreement": 0.780159,
"mean_kappa": 0.754863,
"mean_retention": 0.792179,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9533,
0.8758,
0.675
],
"mean_kappa": 0.815398,
"mean_retention": 0.863787,
"retention_by_severity": [
0.9842,
0.9056,
0.7016
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.96,
0.8558,
0.5217
],
"mean_kappa": 0.756742,
"mean_retention": 0.814502,
"retention_by_severity": [
0.9788,
0.8896,
0.575
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.8458,
0.5242,
0.25
],
"mean_kappa": 0.501422,
"mean_retention": 0.582668,
"retention_by_severity": [
0.8913,
0.5869,
0.2698
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9592,
0.9117,
0.6508
],
"mean_kappa": 0.819489,
"mean_retention": 0.834843,
"retention_by_severity": [
0.9795,
0.928,
0.5971
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.905,
0.4883,
0.6408
],
"mean_kappa": 0.630353,
"mean_retention": 0.664585,
"retention_by_severity": [
0.9481,
0.3801,
0.6656
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9942,
0.9858,
0.9517
],
"mean_kappa": 0.974258,
"mean_retention": 0.992713,
"retention_by_severity": [
0.9973,
1,
0.9808
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9317,
0.8475,
0.655
],
"mean_kappa": 0.786381,
"mean_retention": 0.792156,
"retention_by_severity": [
0.9396,
0.8395,
0.5974
],
"scored": true
}
},
"score": 79.217903,
"worst_retention": 0.269816
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.918257,
"mean_retention": 0.676732,
"per_budget": {
"n100": {
"chance_corrected": 0.849849,
"n_labels": 1100,
"retention": 0.925502
},
"n20": {
"chance_corrected": 0.653302,
"n_labels": 220,
"retention": 0.711459
},
"n5": {
"chance_corrected": 0.36109,
"n_labels": 55,
"retention": 0.393234
}
},
"score": 67.673177
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 81.540578,
"worst_class_recall": 0.832187
},
"uncertainty": {
"full_accuracy": 0.933909,
"risk_coverage_auc": 0.991485,
"score": 99.063314,
"selective_acc_at_50": 0.995862,
"selective_acc_at_80": 0.990872
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 12975,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 8216,
"runtime_s": 80,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organcmnist",
"task_name": "Abdominal CT organs — coronal (C)",
"timestamp": "2026-08-01T19:28:58+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 67.326356
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.815792,
"auc": 0.971994,
"balanced_accuracy": 0.770871,
"ci95": [
0.762711,
0.779135
],
"n_test": 8827,
"score": 74.795833
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.761428,
"mean_agreement": 0.785944,
"mean_kappa": 0.757387,
"mean_retention": 0.862089,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9633,
0.8383,
0.5417
],
"mean_kappa": 0.752208,
"mean_retention": 0.862874,
"retention_by_severity": [
0.995,
0.9503,
0.6433
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8158,
0.7233,
0.5233
],
"mean_kappa": 0.646416,
"mean_retention": 0.767802,
"retention_by_severity": [
0.9187,
0.8021,
0.5826
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9533,
0.8933,
0.8308
],
"mean_kappa": 0.876723,
"mean_retention": 0.955521,
"retention_by_severity": [
1,
0.9593,
0.9073
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.7792,
0.6725,
0.5808
],
"mean_kappa": 0.637108,
"mean_retention": 0.76847,
"retention_by_severity": [
0.888,
0.7722,
0.6453
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9583,
0.9,
0.815
],
"mean_kappa": 0.874477,
"mean_retention": 0.955778,
"retention_by_severity": [
1,
0.9645,
0.9028
],
"scored": true
}
},
"score": 86.208919,
"worst_retention": 0.582636
},
"calibration": {
"accuracy": 0.815792,
"brier": 0.303411,
"ece": 0.128387,
"mean_confidence": 0.944179,
"overconfidence": 0.128387,
"score": 35.806547
},
"corruption": {
"clean_reference_cc": 0.761428,
"mean_agreement": 0.721389,
"mean_kappa": 0.682228,
"mean_retention": 0.746018,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.91,
0.7875,
0.5617
],
"mean_kappa": 0.718201,
"mean_retention": 0.804016,
"retention_by_severity": [
0.969,
0.8495,
0.5936
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.9042,
0.7242,
0.355
],
"mean_kappa": 0.622332,
"mean_retention": 0.696776,
"retention_by_severity": [
0.9621,
0.7905,
0.3377
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.82,
0.5417,
0.2642
],
"mean_kappa": 0.493581,
"mean_retention": 0.619971,
"retention_by_severity": [
0.9582,
0.6418,
0.2599
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9275,
0.8333,
0.6
],
"mean_kappa": 0.748995,
"mean_retention": 0.791234,
"retention_by_severity": [
0.9753,
0.863,
0.5354
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.8783,
0.4742,
0.4717
],
"mean_kappa": 0.548995,
"mean_retention": 0.600198,
"retention_by_severity": [
0.9699,
0.4468,
0.3839
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.985,
0.9692,
0.9108
],
"mean_kappa": 0.948108,
"mean_retention": 0.982898,
"retention_by_severity": [
1,
0.9931,
0.9556
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.8733,
0.7892,
0.5683
],
"mean_kappa": 0.695385,
"mean_retention": 0.727035,
"retention_by_severity": [
0.903,
0.8017,
0.4765
],
"scored": true
}
},
"score": 74.601834,
"worst_retention": 0.259869
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.747958,
"mean_retention": 0.594165,
"per_budget": {
"n100": {
"chance_corrected": 0.668587,
"n_labels": 1100,
"retention": 0.893882
},
"n20": {
"chance_corrected": 0.440921,
"n_labels": 220,
"retention": 0.5895
},
"n5": {
"chance_corrected": 0.223724,
"n_labels": 55,
"retention": 0.299113
}
},
"score": 59.416501
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 42.61959,
"worst_class_recall": 0.47836
},
"uncertainty": {
"full_accuracy": 0.815792,
"risk_coverage_auc": 0.944631,
"score": 93.909408,
"selective_acc_at_50": 0.96715,
"selective_acc_at_80": 0.903568
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 13932,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 8827,
"runtime_s": 71,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organsmnist",
"task_name": "Abdominal CT organs — sagittal (S)",
"timestamp": "2026-08-01T19:31:22+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 70.422162
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.909192,
"auc": 0.989016,
"balanced_accuracy": 0.8843,
"ci95": [
0.876799,
0.892099
],
"n_test": 7180,
"score": 86.983785
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.851374,
"mean_agreement": 0.641556,
"mean_kappa": 0.596822,
"mean_retention": 0.642031,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.8242,
0.5383,
0.245
],
"mean_kappa": 0.485186,
"mean_retention": 0.542473,
"retention_by_severity": [
0.8642,
0.5523,
0.2109
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.7933,
0.7683,
0.7183
],
"mean_kappa": 0.723728,
"mean_retention": 0.749497,
"retention_by_severity": [
0.7876,
0.756,
0.7049
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9358,
0.8558,
0.7317
],
"mean_kappa": 0.819311,
"mean_retention": 0.89015,
"retention_by_severity": [
0.992,
0.9085,
0.77
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.6683,
0.59,
0.495
],
"mean_kappa": 0.521465,
"mean_retention": 0.539033,
"retention_by_severity": [
0.6297,
0.5385,
0.4489
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.7733,
0.4125,
0.2733
],
"mean_kappa": 0.43442,
"mean_retention": 0.489001,
"retention_by_severity": [
0.8221,
0.4001,
0.2448
],
"scored": true
}
},
"score": 64.203061,
"worst_retention": 0.21093
},
"calibration": {
"accuracy": 0.909192,
"brier": 0.150432,
"ece": 0.05855,
"mean_confidence": 0.966577,
"overconfidence": 0.057384,
"score": 70.724941
},
"corruption": {
"clean_reference_cc": 0.851374,
"mean_agreement": 0.509087,
"mean_kappa": 0.450777,
"mean_retention": 0.499213,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.8358,
0.5483,
0.2933
],
"mean_kappa": 0.513696,
"mean_retention": 0.577258,
"retention_by_severity": [
0.8966,
0.58,
0.2552
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.9067,
0.7592,
0.4758
],
"mean_kappa": 0.675424,
"mean_retention": 0.737573,
"retention_by_severity": [
0.9556,
0.7882,
0.469
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.7533,
0.49,
0.2233
],
"mean_kappa": 0.413331,
"mean_retention": 0.448709,
"retention_by_severity": [
0.7472,
0.4461,
0.1528
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.5583,
0.3117,
0.135
],
"mean_kappa": 0.266766,
"mean_retention": 0.308919,
"retention_by_severity": [
0.5767,
0.3019,
0.0481
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.81,
0.1742,
0.3983
],
"mean_kappa": 0.385294,
"mean_retention": 0.428616,
"retention_by_severity": [
0.8392,
0.0893,
0.3573
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9525,
0.8275,
0.5467
],
"mean_kappa": 0.748574,
"mean_retention": 0.807486,
"retention_by_severity": [
0.9753,
0.8845,
0.5627
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.3383,
0.2192,
0.1333
],
"mean_kappa": 0.152356,
"mean_retention": 0.185932,
"retention_by_severity": [
0.3262,
0.1736,
0.058
],
"scored": true
}
},
"score": 49.921314,
"worst_retention": 0.048101
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.869838,
"mean_retention": 0.734894,
"per_budget": {
"n100": {
"chance_corrected": 0.785745,
"n_labels": 900,
"retention": 0.903324
},
"n20": {
"chance_corrected": 0.684688,
"n_labels": 180,
"retention": 0.787144
},
"n5": {
"chance_corrected": 0.447282,
"n_labels": 45,
"retention": 0.514213
}
},
"score": 73.489364
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 56.710214,
"worst_class_recall": 0.615202
},
"uncertainty": {
"full_accuracy": 0.909192,
"risk_coverage_auc": 0.980689,
"score": 97.827489,
"selective_acc_at_50": 0.989972,
"selective_acc_at_80": 0.970752
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 89996,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "Histopathology",
"n_classes": 9,
"n_test": 7180,
"runtime_s": 96.6,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "pathmnist",
"task_name": "Colorectal pathology (9-class tissue)",
"timestamp": "2026-08-01T19:43:04+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 67.356934
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.903846,
"auc": 0.971844,
"balanced_accuracy": 0.874359,
"ci95": [
0.847321,
0.900801
],
"n_test": 624,
"score": 74.871795
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.748718,
"mean_agreement": 0.888782,
"mean_kappa": 0.675389,
"mean_retention": 0.701826,
"n_cases": 624,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9567,
0.9087,
0.8301
],
"mean_kappa": 0.7237,
"mean_retention": 0.775875,
"retention_by_severity": [
1,
0.8687,
0.4589
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8558,
0.7548,
0.7131
],
"mean_kappa": 0.265304,
"mean_retention": 0.224505,
"retention_by_severity": [
0.5137,
0.1541,
0.0057
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9824,
0.9615,
0.9423
],
"mean_kappa": 0.906256,
"mean_retention": 0.964612,
"retention_by_severity": [
0.9966,
0.9795,
0.9178
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.8798,
0.867,
0.8365
],
"mean_kappa": 0.605339,
"mean_retention": 0.54414,
"retention_by_severity": [
0.6107,
0.5651,
0.4566
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9615,
0.9439,
0.9375
],
"mean_kappa": 0.876348,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
}
},
"score": 70.182648,
"worst_retention": 0.005708
},
"calibration": {
"accuracy": 0.903846,
"brier": 0.168444,
"ece": 0.075292,
"mean_confidence": 0.969127,
"overconfidence": 0.06528,
"score": 62.353917
},
"corruption": {
"clean_reference_cc": 0.748718,
"mean_agreement": 0.852335,
"mean_kappa": 0.551051,
"mean_retention": 0.549141,
"n_cases": 624,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9599,
0.9423,
0.8397
],
"mean_kappa": 0.768791,
"mean_retention": 0.822679,
"retention_by_severity": [
1,
0.9384,
0.5297
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8894,
0.7901,
0.7147
],
"mean_kappa": 0.353035,
"mean_retention": 0.308219,
"retention_by_severity": [
0.6336,
0.2797,
0.0114
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9151,
0.7869,
0.7212
],
"mean_kappa": 0.384879,
"mean_retention": 0.342466,
"retention_by_severity": [
0.7249,
0.2683,
0.0342
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9375,
0.875,
0.7772
],
"mean_kappa": 0.600539,
"mean_retention": 0.615297,
"retention_by_severity": [
1,
0.621,
0.2249
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.8702,
0.774,
0.7196
],
"mean_kappa": 0.323351,
"mean_retention": 0.281202,
"retention_by_severity": [
0.5559,
0.2477,
0.04
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9856,
0.9615,
0.9119
],
"mean_kappa": 0.880801,
"mean_retention": 0.921994,
"retention_by_severity": [
1,
0.9566,
0.8094
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9071,
0.8349,
0.7853
],
"mean_kappa": 0.545961,
"mean_retention": 0.552131,
"retention_by_severity": [
0.8402,
0.5126,
0.3037
],
"scored": true
}
},
"score": 54.914112,
"worst_retention": 0.011416
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.748718,
"mean_retention": 0.737823,
"per_budget": {
"n100": {
"chance_corrected": 0.637607,
"n_labels": 200,
"retention": 0.851598
},
"n20": {
"chance_corrected": 0.632479,
"n_labels": 40,
"retention": 0.844749
},
"n5": {
"chance_corrected": 0.387179,
"n_labels": 10,
"retention": 0.517123
}
},
"score": 73.782344
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 51.282051,
"worst_class_recall": 0.75641
},
"uncertainty": {
"full_accuracy": 0.903846,
"risk_coverage_auc": 0.972005,
"score": 94.400981,
"selective_acc_at_50": 0.980769,
"selective_acc_at_80": 0.955912
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 4708,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Chest X-ray",
"n_classes": 2,
"n_test": 624,
"runtime_s": 25.7,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "pneumoniamnist",
"task_name": "Paediatric chest X-ray (pneumonia)",
"timestamp": "2026-08-01T19:16:35+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 42.304494
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.5,
"auc": 0.723223,
"balanced_accuracy": 0.349259,
"ci95": [
0.295929,
0.40954
],
"n_test": 400,
"score": 18.657326
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.186573,
"mean_agreement": 0.631167,
"mean_kappa": 0.452271,
"mean_retention": 0.770926,
"n_cases": 400,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.6975,
0.5175,
0.3475
],
"mean_kappa": 0.344416,
"mean_retention": 0.739427,
"retention_by_severity": [
0.9468,
0.7872,
0.4842
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.7525,
0.6225,
0.46
],
"mean_kappa": 0.437795,
"mean_retention": 0.863651,
"retention_by_severity": [
0.6956,
0.8954,
1
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.795,
0.65,
0.57
],
"mean_kappa": 0.495995,
"mean_retention": 0.653676,
"retention_by_severity": [
0.6577,
0.8788,
0.4246
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.7025,
0.68,
0.625
],
"mean_kappa": 0.477982,
"mean_retention": 0.819965,
"retention_by_severity": [
0.6842,
0.9328,
0.8429
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.8,
0.65,
0.5975
],
"mean_kappa": 0.505168,
"mean_retention": 0.777911,
"retention_by_severity": [
1,
0.6372,
0.6965
],
"scored": true
}
},
"score": 77.092619,
"worst_retention": 0.424583
},
"calibration": {
"accuracy": 0.5,
"brier": 0.727707,
"ece": 0.263934,
"mean_confidence": 0.763934,
"overconfidence": 0.263934,
"score": 0
},
"corruption": {
"clean_reference_cc": 0.186573,
"mean_agreement": 0.538333,
"mean_kappa": 0.343836,
"mean_retention": 0.710291,
"n_cases": 400,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.7225,
0.625,
0.5525
],
"mean_kappa": 0.460057,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.745,
0.55,
0.4575
],
"mean_kappa": 0.368521,
"mean_retention": 0.718452,
"retention_by_severity": [
0.7723,
0.7313,
0.6517
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.755,
0.6475,
0.5325
],
"mean_kappa": 0.454669,
"mean_retention": 0.828861,
"retention_by_severity": [
0.8352,
0.9885,
0.6629
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.51,
0.2975,
0.215
],
"mean_kappa": 0.14784,
"mean_retention": 0.431862,
"retention_by_severity": [
1,
0.2956,
0
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.5275,
0.48,
0.4475
],
"mean_kappa": 0.237428,
"mean_retention": 0.727442,
"retention_by_severity": [
0.5744,
0.9253,
0.6826
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.87,
0.7525,
0.59
],
"mean_kappa": 0.588069,
"mean_retention": 0.835698,
"retention_by_severity": [
0.8808,
1,
0.6263
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.485,
0.325,
0.2175
],
"mean_kappa": 0.150265,
"mean_retention": 0.429721,
"retention_by_severity": [
0.8542,
0.2971,
0.1378
],
"scored": true
}
},
"score": 71.029093,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.186573,
"mean_retention": 0.932379,
"per_budget": {
"n100": {
"chance_corrected": 0.191054,
"n_labels": 466,
"retention": 1
},
"n20": {
"chance_corrected": 0.188652,
"n_labels": 100,
"retention": 1
},
"n5": {
"chance_corrected": 0.148725,
"n_labels": 25,
"retention": 0.797137
}
},
"score": 93.237916
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0.086957
},
"uncertainty": {
"full_accuracy": 0.5,
"risk_coverage_auc": 0.686006,
"score": 60.75074,
"selective_acc_at_50": 0.685,
"selective_acc_at_80": 0.540625
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 1080,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Fundus",
"n_classes": 5,
"n_test": 400,
"runtime_s": 11,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "retinamnist",
"task_name": "Fundus photography (DR grade)",
"timestamp": "2026-08-01T19:13:22+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 37.778889
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.575825,
"auc": 0.851561,
"balanced_accuracy": 0.465304,
"ci95": [
0.460775,
0.469939
],
"n_test": 47280,
"score": 38.891855
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.387207,
"mean_agreement": 0.501222,
"mean_kappa": 0.382826,
"mean_retention": 0.632411,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.6217,
0.4517,
0.3817
],
"mean_kappa": 0.349907,
"mean_retention": 0.566586,
"retention_by_severity": [
0.768,
0.5435,
0.3883
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.7292,
0.63,
0.5308
],
"mean_kappa": 0.534456,
"mean_retention": 0.840818,
"retention_by_severity": [
0.9767,
0.8809,
0.6649
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.5717,
0.3983,
0.3208
],
"mean_kappa": 0.297922,
"mean_retention": 0.477747,
"retention_by_severity": [
0.692,
0.4664,
0.2749
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.6592,
0.54,
0.49
],
"mean_kappa": 0.471653,
"mean_retention": 0.795702,
"retention_by_severity": [
0.9185,
0.7577,
0.7109
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.4708,
0.3883,
0.3342
],
"mean_kappa": 0.260189,
"mean_retention": 0.4812,
"retention_by_severity": [
0.6237,
0.4597,
0.3602
],
"scored": true
}
},
"score": 63.241067,
"worst_retention": 0.274855
},
"calibration": {
"accuracy": 0.575825,
"brier": 0.68395,
"ece": 0.284214,
"mean_confidence": 0.860039,
"overconfidence": 0.284214,
"score": 0
},
"corruption": {
"clean_reference_cc": 0.387207,
"mean_agreement": 0.467579,
"mean_kappa": 0.339422,
"mean_retention": 0.515771,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.4283,
0.315,
0.2458
],
"mean_kappa": 0.197593,
"mean_retention": 0.305269,
"retention_by_severity": [
0.458,
0.2773,
0.1805
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.6617,
0.4892,
0.3317
],
"mean_kappa": 0.366012,
"mean_retention": 0.526429,
"retention_by_severity": [
0.7341,
0.501,
0.3442
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.7292,
0.49,
0.3192
],
"mean_kappa": 0.423452,
"mean_retention": 0.677451,
"retention_by_severity": [
0.9095,
0.7059,
0.417
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.4575,
0.41,
0.3358
],
"mean_kappa": 0.224457,
"mean_retention": 0.358235,
"retention_by_severity": [
0.5385,
0.3641,
0.1721
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.6975,
0.435,
0.4608
],
"mean_kappa": 0.411829,
"mean_retention": 0.593546,
"retention_by_severity": [
0.8307,
0.4804,
0.4695
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.7517,
0.6067,
0.3558
],
"mean_kappa": 0.475212,
"mean_retention": 0.700291,
"retention_by_severity": [
0.8711,
0.8065,
0.4233
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.5017,
0.4233,
0.3733
],
"mean_kappa": 0.277396,
"mean_retention": 0.449177,
"retention_by_severity": [
0.5788,
0.4738,
0.295
],
"scored": true
}
},
"score": 51.577118,
"worst_retention": 0.17208
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.388919,
"mean_retention": 0.419143,
"per_budget": {
"n100": {
"chance_corrected": 0.285461,
"n_labels": 800,
"retention": 0.733986
},
"n20": {
"chance_corrected": 0.144413,
"n_labels": 160,
"retention": 0.37132
},
"n5": {
"chance_corrected": 0.059163,
"n_labels": 40,
"retention": 0.152123
}
},
"score": 41.914309
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 2.757341,
"worst_class_recall": 0.149127
},
"uncertainty": {
"full_accuracy": 0.575825,
"risk_coverage_auc": 0.731611,
"score": 69.326928,
"selective_acc_at_50": 0.734772,
"selective_acc_at_80": 0.634412
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 165466,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "Microscopy",
"n_classes": 8,
"n_test": 47280,
"runtime_s": 111.7,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "tissuemnist",
"task_name": "Kidney cortex cells (8-class)",
"timestamp": "2026-08-01T19:39:43+00:00"
}
],
"heldout": null,
"sources": [
{
"path": "results/cross_site/organ_ct__resnet18_ft.json",
"sha256": "d27a1ee962818c0b7d1ba43d05d587c9ef3b38ae16c1dd2febb2f5f2a34c9332"
},
{
"path": "results/heldout/audit_report.json",
"sha256": "df460b6d91a6512dfa1e7dc650637fce2b4dd3a1ec32e03446bf6e11f9c597d2"
},
{
"path": "results/heldout/cycle.json",
"sha256": "e012b8978457b51b5c31d18d14d2cc92f100731897523306acae19934f943502"
},
{
"path": "results/heldout/manifests/gen1.seal.json",
"sha256": "91a7adfa29870a0719125cdd48f6dcb149f1f6485ac1cfa104c6e00057ad9b81"
},
{
"path": "results/heldout/manifests/gen2.seal.json",
"sha256": "f61470918a09cf39e074ef14fb7ade4ac3a13d36d9385f18cf9b92e4b5329547"
},
{
"path": "results/heldout/manifests/public.seal.json",
"sha256": "1a4fd930d2e781ac7163483990f15d6694e28e507d1b3ca009ff09395e47497c"
},
{
"path": "results/records/bloodmnist__resnet18_ft.json",
"sha256": "cbf774ba932c41fcd50ac1eb2f3e807f5b564fd505b351efaa88012d6c4a2955"
},
{
"path": "results/records/breastmnist__resnet18_ft.json",
"sha256": "5852fd51f132615d49cfa9b0b363a02988fd415aedaf5561fb12ec8609154cce"
},
{
"path": "results/records/dermamnist__resnet18_ft.json",
"sha256": "a5c10c8831bf4f8da07c3daf21966725ba687e825116e1e9526ffac8c833c14e"
},
{
"path": "results/records/octmnist__resnet18_ft.json",
"sha256": "4639d3534e8fe9353280dc9054bfd4eb3c01b4e553047e7dad33c20232566596"
},
{
"path": "results/records/organamnist__resnet18_ft.json",
"sha256": "8c1b3c1ba85b480ec44526fdc6dd9e14b13753bedf04a548914ca80e0c2e299a"
},
{
"path": "results/records/organcmnist__resnet18_ft.json",
"sha256": "f1c3e1518f67b216d2624b78f101799c09432e6fb4118a4dd9bd458d93c57e01"
},
{
"path": "results/records/organsmnist__resnet18_ft.json",
"sha256": "a750e1c522188b4f44e3afab039814caee04b40c4343047f5d984ca3fe6d39ec"
},
{
"path": "results/records/pathmnist__resnet18_ft.json",
"sha256": "18c573007d941b9b8bf244ba7229199f0e2e3254dfdaf7a6aa9911f6fcb06db8"
},
{
"path": "results/records/pneumoniamnist__resnet18_ft.json",
"sha256": "84d35e6465d3b422e594567e90c4fa1ae0cf7346f7dfdb988cfd2e7d692c8754"
},
{
"path": "results/records/retinamnist__resnet18_ft.json",
"sha256": "615ed85d61c7a6a2e19e56bdfc384f616c0abf343a081be9b531ad9df9cbc977"
},
{
"path": "results/records/tissuemnist__resnet18_ft.json",
"sha256": "27f3dc9a75b577b253954268459487951f0d92eba8c79f21762f69580e4d9328"
}
]
},
"declared": null,
"declared_note": "No vendor declaration. This is an open reference baseline implemented by NakedSignal, not a commercial product: there is no vendor to assert an intended use or to sign an attestation, so the declared block is null rather than filled in on the vendor's behalf.",
"expiry_basis": "the sealed held-out generation this evidence cycle is anchored to (gen2, set hash ad30d214daa285c5...) is published to expire 2026-09-20; per spec, a passport lapses when its generation retires or its spec version is superseded, whichever comes first",
"expiry_utc": "2026-09-20T00:00:00Z",
"issued_utc": "2026-08-03T16:03:42Z",
"issuer": {
"algo": "Ed25519",
"key_id": "ns-passport-2026-07",
"name": "NakedSignal",
"public_key_b64": "0PBCC6mjzcppR5QdPcK7vP/AamB6MP8l1T4ypMYbsMw="
},
"observed": null,
"observed_note": "Not available: this requires continuous monitoring. Production history and drift are observed fields, and nothing is deployed, so there is nothing to observe. The object is present and null on purpose. The shape is the roadmap, not a claim.",
"passport_version": "0.1",
"signature": "o504dfbx9eJqvcE+HhMrXYfqo1nXru4mzAcdIf6gNZ06s5UuW2MYQ38giplo5VhuKoWoqOpVLI5PSIP9nsrhDA==",
"spec_version": "MedEval-1 v0.1",
"subject": {
"behaviour_version": "1.0.0",
"code_fingerprint": "b444c02112cc",
"code_fingerprints": [
"b444c02112cc"
],
"family": "trained",
"model_id": "resnet18_ft",
"model_name": "ResNet-18 (ImageNet) fine-tuned",
"params": "ImageNet init, all layers, 12 epochs, adam 3e-4, 32px"
}
}Canonicalisation: RFC 8785 (JCS)-compatible: UTF-8, object keys sorted by code point, no insignificant whitespace, ECMAScript number formatting. every float is rounded once at emit to 6 decimal places (Python round(), banker's rounding); integers are left exact. Signed over the whole document with the `signature` member removed, canonicalised as above and encoded UTF-8. The public key for ns-passport-2026-07 is 0PBCC6mjzcppR5QdPcK7vP/AamB6MP8l1T4ypMYbsMw= — see Governance.