Back to the consoleDocumentation
PCA-64 + logistic regression
LIVEThe signed evidence record for pca_logreg. Everything below was read out of the passport file; the signature was checked when this page was built, by the same code the Verify button runs.
SIGNATURE VALIDexit 0signature valid, issued by NakedSignal under key ns-passport-2026-07, not expired
SubjectPCA-64 + logistic regression
pca_logregConfigurationup to 64 components, L2, lbfgs, C=1
Code fingerprintb444c02112cc
Spec versionMedEval-1 v0.1
Issued2026-08-03T16:03:42Z by NakedSignal
Expires2026-09-20T00:00:00Z — the sealed held-out generation this evidence cycle is anchored to (gen2, set hash ad30d214daa285c5...) is published to expire 2026-09-20; per spec, a passport lapses when its generation retires or its spec version is superseded, whichever comes first
key ns-passport-2026-07 · Ed25519
The check runs against the public key published on the governance page, using your browser's own Ed25519 implementation. The exit code shown is the exit code verify_passport.py returns for the same document.
Measured by NakedSignal
Computed in this repo by the MedEval-1 harness on data the model had never seen. Every number below was copied out of a file the harness wrote — 17 of them, each listed with its SHA-256 at the foot of this panel — and is printed exactly as it was signed, not re-rounded for display.
| Task | Domain | n test | accuracy | calibration | uncertainty | subgroup | corruption | acquisition | limited data | cost | spec sensitivity | Index |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bloodmnist | medicine | 3421 | 68.153251 | 85.977292 | 90.076798 | 32.394366 | 73.465143 | 80.018395 | 63.821275 | n/a | n/a | 70.683134 |
| breastmnist | medicine | 156 | 32.581454 | 51.62641 | 77.601988 | 0 | 77.655678 | 94.025641 | 75.128205 | n/a | n/a | 56.882992 |
| dermamnist | medicine | 2005 | 24.144058 | 84.486736 | 85.005692 | 0 | 86.736253 | 86.619186 | 88.121287 | n/a | n/a | 61.194711 |
| octmnist | medicine | 1000 | 15.733333 | 0 | 24.105241 | 0 | 77.48184 | 71.073446 | 28.248588 | n/a | n/a | 33.546546 |
| organamnist | medicine | 17778 | 55.700851 | 71.263858 | 83.411665 | 23.764479 | 87.984401 | 92.088761 | 70.838241 | n/a | n/a | 69.332005 |
| organcmnist | medicine | 8216 | 61.709198 | 90.665484 | 86.870293 | 38.236659 | 89.315972 | 93.284876 | 65.343896 | n/a | n/a | 75.415611 |
| organsmnist | medicine | 8827 | 36.555468 | 76.793707 | 58.326703 | 16.696629 | 86.474591 | 91.652986 | 64.900078 | n/a | n/a | 62.349258 |
| pathmnist | medicine | 7180 | 47.266068 | 57.765351 | 69.151655 | 0 | 81.23686 | 74.170252 | 59.994663 | n/a | n/a | 56.345269 |
| pneumoniamnist | medicine | 624 | 59.401709 | 52.064128 | 83.758137 | 21.367521 | 77.910243 | 82.215827 | 92.517986 | n/a | n/a | 65.670714 |
| retinamnist | medicine | 400 | 15.032374 | 47.761785 | 63.379869 | 0 | 94.674899 | 93.208241 | 75.092625 | n/a | n/a | 53.699376 |
| tissuemnist | medicine | 47280 | 16.783286 | 84.852063 | 53.048767 | 0 | 77.316863 | 76.526514 | 63.976023 | n/a | n/a | 52.053348 |
Cross-site retention · organ_ct
retention is absolute retained skill (chance-corrected), per spec 2.2.1; the diagonal is same-site, every off-diagonal cell is train-on-row, test-on-column
Sitesorganamnist, organcmnist, organsmnist
Mean retention0.405116
Worst retention0.118962
Private held-out track
PUBLIC-DATA DEMONSTRATION OF THE PRIVATE PROTOCOL
| Generation | Public | Private (equated) | Gap | z | Verdict |
|---|---|---|---|---|---|
| gen1 | 69.368163 | 65.53647 | 3.831693 | 1.628733 | clear |
| gen2 | 68.991661 | 62.123559 | 6.868102 | 1.502294 | clear |
Known limitations
Subgroup metadataabsent on 11 of 11 corpora, so the subgroup score falls back to the worst diagnostic class. That absence is a disclosure, not a pass.
Training set cappedyes on at least one task — see the raw JSON for which
Observed in deploymentnothing — see the third panel
Audit chain
Bundles re-derived48 of 48
Ledger100 entries, head 6b7b371ff650a8ed…, chain intact
Re-run it yourself
python src/heldout/audit.py 1f66c7719ab3943c6fcc17 source files, with hashes
| results/cross_site/organ_ct__pca_logreg.json | 17a80d5e8afe2ac5601b3b76ad575ad648bff0063f20c5e052e5c6dcb38e1ef1 |
| results/heldout/audit_report.json | df460b6d91a6512dfa1e7dc650637fce2b4dd3a1ec32e03446bf6e11f9c597d2 |
| results/heldout/cycle.json | e012b8978457b51b5c31d18d14d2cc92f100731897523306acae19934f943502 |
| results/heldout/manifests/gen1.seal.json | 91a7adfa29870a0719125cdd48f6dcb149f1f6485ac1cfa104c6e00057ad9b81 |
| results/heldout/manifests/gen2.seal.json | f61470918a09cf39e074ef14fb7ade4ac3a13d36d9385f18cf9b92e4b5329547 |
| results/heldout/manifests/public.seal.json | 1a4fd930d2e781ac7163483990f15d6694e28e507d1b3ca009ff09395e47497c |
| results/records/bloodmnist__pca_logreg.json | 5092aa652607bc98f4a2d98316aadfa0cfa9ec0545d3f448dd35f9d4660306cf |
| results/records/breastmnist__pca_logreg.json | f869c03ae4ac7519290106cd1070a1569ad8264a833c2a7e33e8ca3233575b62 |
| results/records/dermamnist__pca_logreg.json | 52dae364cf5789ca7d2240146a5e5d3db08b31224638f3b130cfbf822ab9e016 |
| results/records/octmnist__pca_logreg.json | 6e204cf19896c2a73b74e3659ceb4bc8ebaf715d064a8642146d3cd540df501f |
| results/records/organamnist__pca_logreg.json | bfc86f4705a0666d05500a5e3f89ca3735d58fc50bdde1f6bb136953da953390 |
| results/records/organcmnist__pca_logreg.json | 2cfd9d683d499a69fa6129eabf99e43228b827baa4d181d4a695c060e498c766 |
| results/records/organsmnist__pca_logreg.json | 4755a3c534ea3464e9f38479db5c6e1c2858bbd9048df338f28a85babd82931c |
| results/records/pathmnist__pca_logreg.json | 1b6e554ea95a6e966c40eba60b899e97972b4b25ead7fb1ec685402c6549dd90 |
| results/records/pneumoniamnist__pca_logreg.json | 12f6156a7d6377f782eac7d8e71f9dde2f6bae40982c4ffcbbde8b5d9b25bcfa |
| results/records/retinamnist__pca_logreg.json | 3bb482d1c1fd7fc1d8f87471af555941534d97bc12267c02cb1d6f928487e600 |
| results/records/tissuemnist__pca_logreg.json | 0f9ebf50fa49b406c7679cf1d48f7ae547926223257e0fbcceb27c6bdc5dca8d |
Declared by vendor
Five fields a vendor asserts about its own product — intended use, forbidden use, training cutoff, regulatory clearances, and who is personally attesting. NakedSignal never fills these in on a vendor's behalf.
No vendor declaration. This is an open reference baseline implemented by NakedSignal, not a commercial product: there is no vendor to assert an intended use or to sign an attestation, so the declared block is null rather than filled in on the vendor's behalf.
Observed in deployment
Production history and drift — how the model has actually behaved since it was deployed, on real traffic.
Not available: this requires continuous monitoring. Production history and drift are observed fields, and nothing is deployed, so there is nothing to observe. The object is present and null on purpose. The shape is the roadmap, not a claim.
Raw JSON — the whole signed document, 82,313 characters
{
"canonicalisation": {
"float_rounding": "every float is rounded once at emit to 6 decimal places (Python round(), banker's rounding); integers are left exact",
"form": "RFC 8785 (JCS)-compatible: UTF-8, object keys sorted by code point, no insignificant whitespace, ECMAScript number formatting",
"signed_over": "the whole document with the `signature` member removed, canonicalised as above and encoded UTF-8"
},
"computed": {
"audit": {
"audit_command": "python src/heldout/audit.py 1f66c7719ab3943c6fcc",
"bundles": 48,
"ledger_entries": 100,
"ledger_head": "6b7b371ff650a8edaf477490f2a5f9b043433f3ba9941989ee1962ed8f1b1d88",
"ledger_intact": true,
"rederived": 48
},
"cross_site": {
"code_fingerprint": "b444c02112cc",
"device": "mps",
"family": "organ_ct",
"matrix": {
"organamnist": {
"organamnist": {
"balanced_accuracy": 0.59728,
"chance_corrected": 0.557009,
"ece": 0.057472
},
"organcmnist": {
"balanced_accuracy": 0.332428,
"chance_corrected": 0.265671,
"ece": 0.232122,
"retention": 0.47696
},
"organsmnist": {
"balanced_accuracy": 0.151148,
"chance_corrected": 0.066263,
"ece": 0.398572,
"retention": 0.118962
}
},
"organcmnist": {
"organamnist": {
"balanced_accuracy": 0.37054,
"chance_corrected": 0.307594,
"ece": 0.229956,
"retention": 0.498458
},
"organcmnist": {
"balanced_accuracy": 0.651902,
"chance_corrected": 0.617092,
"ece": 0.018669
},
"organsmnist": {
"balanced_accuracy": 0.282283,
"chance_corrected": 0.210512,
"ece": 0.268042,
"retention": 0.341135
}
},
"organsmnist": {
"organamnist": {
"balanced_accuracy": 0.19006,
"chance_corrected": 0.109066,
"ece": 0.289716,
"retention": 0.298356
},
"organcmnist": {
"balanced_accuracy": 0.32248,
"chance_corrected": 0.254728,
"ece": 0.114531,
"retention": 0.696827
},
"organsmnist": {
"balanced_accuracy": 0.423232,
"chance_corrected": 0.365555,
"ece": 0.046413
}
}
},
"mean_retention": 0.405116,
"note": "retention is absolute retained skill (chance-corrected), per spec 2.2.1; the diagonal is same-site, every off-diagonal cell is train-on-row, test-on-column",
"score": 40.511635,
"site_labels": {
"organamnist": "axial",
"organcmnist": "coronal",
"organsmnist": "sagittal"
},
"sites": [
"organamnist",
"organcmnist",
"organsmnist"
],
"spec_version": "MedEval-1 v0.1",
"worst_retention": 0.118962
},
"evaluations": [
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 70.683134
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.755919,
"auc": 0.95308,
"balanced_accuracy": 0.721341,
"ci95": [
0.70525,
0.737754
],
"n_test": 3421,
"score": 68.153251
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.70195,
"mean_agreement": 0.735333,
"mean_kappa": 0.695012,
"mean_retention": 0.800184,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9817,
0.9592,
0.9242
],
"mean_kappa": 0.946494,
"mean_retention": 0.988568,
"retention_by_severity": [
0.9894,
0.9942,
0.9821
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.81,
0.6108,
0.4133
],
"mean_kappa": 0.548494,
"mean_retention": 0.689206,
"retention_by_severity": [
0.9129,
0.7001,
0.4547
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.5608,
0.3567,
0.2317
],
"mean_kappa": 0.301181,
"mean_retention": 0.463161,
"retention_by_severity": [
0.7061,
0.4367,
0.2467
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9558,
0.9183,
0.8742
],
"mean_kappa": 0.900022,
"mean_retention": 0.954994,
"retention_by_severity": [
0.9998,
0.9607,
0.9044
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9142,
0.8242,
0.695
],
"mean_kappa": 0.778872,
"mean_retention": 0.904991,
"retention_by_severity": [
0.9604,
0.9276,
0.827
],
"scored": true
}
},
"score": 80.018395,
"worst_retention": 0.246692
},
"calibration": {
"accuracy": 0.755919,
"brier": 0.338515,
"ece": 0.028045,
"mean_confidence": 0.729197,
"overconfidence": -0.026722,
"score": 85.977292
},
"corruption": {
"clean_reference_cc": 0.70195,
"mean_agreement": 0.740317,
"mean_kappa": 0.685892,
"mean_retention": 0.734651,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.46,
0.3242,
0.135
],
"mean_kappa": 0.170152,
"mean_retention": 0.21361,
"retention_by_severity": [
0.4338,
0.2009,
0.0061
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.55,
0.415,
0.3433
],
"mean_kappa": 0.308435,
"mean_retention": 0.335502,
"retention_by_severity": [
0.5039,
0.3046,
0.198
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9658,
0.9125,
0.8017
],
"mean_kappa": 0.872146,
"mean_retention": 0.909734,
"retention_by_severity": [
0.9993,
0.9684,
0.7616
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9683,
0.9442,
0.8708
],
"mean_kappa": 0.914307,
"mean_retention": 0.98029,
"retention_by_severity": [
0.988,
0.9851,
0.9678
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9442,
0.4317,
0.8558
],
"mean_kappa": 0.684992,
"mean_retention": 0.744481,
"retention_by_severity": [
0.9905,
0.3249,
0.9181
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.985,
0.9908,
0.9575
],
"mean_kappa": 0.973643,
"mean_retention": 0.991063,
"retention_by_severity": [
0.9922,
0.9936,
0.9874
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9358,
0.9083,
0.8467
],
"mean_kappa": 0.877571,
"mean_retention": 0.96788,
"retention_by_severity": [
0.9861,
0.9818,
0.9357
],
"scored": true
}
},
"score": 73.465143,
"worst_retention": 0.006115
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.681533,
"mean_retention": 0.638213,
"per_budget": {
"n100": {
"chance_corrected": 0.585306,
"n_labels": 800,
"retention": 0.858808
},
"n20": {
"chance_corrected": 0.393906,
"n_labels": 160,
"retention": 0.577971
},
"n5": {
"chance_corrected": 0.325677,
"n_labels": 40,
"retention": 0.477859
}
},
"score": 63.821275
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 32.394366,
"worst_class_recall": 0.408451
},
"uncertainty": {
"full_accuracy": 0.755919,
"risk_coverage_auc": 0.913172,
"score": 90.076798,
"selective_acc_at_50": 0.950292,
"selective_acc_at_80": 0.831567
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 11959,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Microscopy",
"n_classes": 8,
"n_test": 3421,
"runtime_s": 2.7,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "bloodmnist",
"task_name": "Peripheral blood cells (8-class)",
"timestamp": "2026-08-01T19:18:31+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 56.882992
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.782051,
"auc": 0.805764,
"balanced_accuracy": 0.662907,
"ci95": [
0.59406,
0.737709
],
"n_test": 156,
"score": 32.581454
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.325815,
"mean_agreement": 0.938034,
"mean_kappa": 0.803467,
"mean_retention": 0.940256,
"n_cases": 156,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
1,
0.9808,
0.9808
],
"mean_kappa": 0.954545,
"mean_retention": 0.957692,
"retention_by_severity": [
1,
1,
0.8731
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.9231,
0.8846,
0.8141
],
"mean_kappa": 0.578624,
"mean_retention": 0.767949,
"retention_by_severity": [
1,
0.6923,
0.6115
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9808,
0.9359,
0.9103
],
"mean_kappa": 0.81787,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
1,
0.9808,
0.9679
],
"mean_kappa": 0.9375,
"mean_retention": 0.975641,
"retention_by_severity": [
1,
1,
0.9269
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9744,
0.9038,
0.8333
],
"mean_kappa": 0.728793,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
}
},
"score": 94.025641,
"worst_retention": 0.611538
},
"calibration": {
"accuracy": 0.782051,
"brier": 0.304484,
"ece": 0.096747,
"mean_confidence": 0.8336,
"overconfidence": 0.051549,
"score": 51.62641
},
"corruption": {
"clean_reference_cc": 0.325815,
"mean_agreement": 0.931929,
"mean_kappa": 0.703367,
"mean_retention": 0.776557,
"n_cases": 156,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9487,
0.8654,
0.8526
],
"mean_kappa": 0.417379,
"mean_retention": 0.4,
"retention_by_severity": [
0.8154,
0.2654,
0.1192
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.9167,
0.859,
0.8333
],
"mean_kappa": 0.285853,
"mean_retention": 0.280769,
"retention_by_severity": [
0.65,
0.1923,
0
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
1,
0.9744,
0.9423
],
"mean_kappa": 0.898918,
"mean_retention": 0.942308,
"retention_by_severity": [
1,
1,
0.8269
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9936,
0.9487,
0.9038
],
"mean_kappa": 0.815059,
"mean_retention": 0.884615,
"retention_by_severity": [
0.9269,
0.8538,
0.8731
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9679,
0.8333,
0.9231
],
"mean_kappa": 0.732308,
"mean_retention": 0.988462,
"retention_by_severity": [
0.9731,
1,
0.9923
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9936,
0.9872,
0.9872
],
"mean_kappa": 0.961878,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9744,
0.9808,
0.8846
],
"mean_kappa": 0.812175,
"mean_retention": 0.939744,
"retention_by_severity": [
1,
0.9731,
0.8462
],
"scored": true
}
},
"score": 77.655678,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.325815,
"mean_retention": 0.751282,
"per_budget": {
"n100": {
"chance_corrected": 0.370927,
"n_labels": 200,
"retention": 1
},
"n20": {
"chance_corrected": 0.082707,
"n_labels": 40,
"retention": 0.253846
},
"n5": {
"chance_corrected": 0.369674,
"n_labels": 10,
"retention": 1
}
},
"score": 75.128205
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0.404762
},
"uncertainty": {
"full_accuracy": 0.782051,
"risk_coverage_auc": 0.88801,
"score": 77.601988,
"selective_acc_at_50": 0.897436,
"selective_acc_at_80": 0.848
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 546,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Ultrasound",
"n_classes": 2,
"n_test": 156,
"runtime_s": 0.3,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "breastmnist",
"task_name": "Breast ultrasound (malignancy)",
"timestamp": "2026-08-01T19:12:55+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 61.194711
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.700748,
"auc": 0.864319,
"balanced_accuracy": 0.349806,
"ci95": [
0.314821,
0.387933
],
"n_test": 2005,
"score": 24.144058
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.258723,
"mean_agreement": 0.932444,
"mean_kappa": 0.778148,
"mean_retention": 0.866192,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9925,
0.9775,
0.9567
],
"mean_kappa": 0.919593,
"mean_retention": 0.973343,
"retention_by_severity": [
1,
0.9623,
0.9577
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.9708,
0.9308,
0.8975
],
"mean_kappa": 0.771463,
"mean_retention": 0.864303,
"retention_by_severity": [
0.9973,
0.853,
0.7426
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9333,
0.8875,
0.8242
],
"mean_kappa": 0.609966,
"mean_retention": 0.571094,
"retention_by_severity": [
0.7962,
0.5275,
0.3896
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9858,
0.9708,
0.9625
],
"mean_kappa": 0.910795,
"mean_retention": 0.927661,
"retention_by_severity": [
0.9444,
0.9275,
0.9111
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9567,
0.8975,
0.8425
],
"mean_kappa": 0.678924,
"mean_retention": 0.994559,
"retention_by_severity": [
1,
0.9837,
1
],
"scored": true
}
},
"score": 86.619186,
"worst_retention": 0.389568
},
"calibration": {
"accuracy": 0.700748,
"brier": 0.412664,
"ece": 0.031027,
"mean_confidence": 0.717206,
"overconfidence": 0.016458,
"score": 84.486736
},
"corruption": {
"clean_reference_cc": 0.258723,
"mean_agreement": 0.870516,
"mean_kappa": 0.720606,
"mean_retention": 0.867363,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.885,
0.6058,
0.1708
],
"mean_kappa": 0.321526,
"mean_retention": 0.672721,
"retention_by_severity": [
1,
0.8439,
0.1743
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.9433,
0.8708,
0.7417
],
"mean_kappa": 0.607122,
"mean_retention": 0.657024,
"retention_by_severity": [
0.8033,
0.6148,
0.553
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.99,
0.9742,
0.9433
],
"mean_kappa": 0.897498,
"mean_retention": 0.969066,
"retention_by_severity": [
0.9984,
0.9423,
0.9665
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9708,
0.9567,
0.8958
],
"mean_kappa": 0.81543,
"mean_retention": 0.888716,
"retention_by_severity": [
0.8885,
0.9216,
0.856
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9758,
0.7283,
0.9608
],
"mean_kappa": 0.734476,
"mean_retention": 0.978333,
"retention_by_severity": [
0.935,
1,
1
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.995,
0.99,
0.9275
],
"mean_kappa": 0.906902,
"mean_retention": 0.981525,
"retention_by_severity": [
0.9984,
0.9848,
0.9614
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9608,
0.9342,
0.86
],
"mean_kappa": 0.761287,
"mean_retention": 0.924152,
"retention_by_severity": [
1,
0.8688,
0.9036
],
"scored": true
}
},
"score": 86.736253,
"worst_retention": 0.174268
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.241441,
"mean_retention": 0.881213,
"per_budget": {
"n100": {
"chance_corrected": 0.352675,
"n_labels": 679,
"retention": 1
},
"n20": {
"chance_corrected": 0.227204,
"n_labels": 140,
"retention": 0.941036
},
"n5": {
"chance_corrected": 0.169637,
"n_labels": 35,
"retention": 0.702603
}
},
"score": 88.121287
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0.075758
},
"uncertainty": {
"full_accuracy": 0.700748,
"risk_coverage_auc": 0.871477,
"score": 85.005692,
"selective_acc_at_50": 0.883234,
"selective_acc_at_80": 0.776808
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 7007,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Dermatoscopy",
"n_classes": 7,
"n_test": 2005,
"runtime_s": 2.6,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "dermamnist",
"task_name": "Dermatoscopy (7-class skin lesion)",
"timestamp": "2026-08-01T19:17:00+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 33.546546
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.368,
"auc": 0.612868,
"balanced_accuracy": 0.368,
"ci95": [
0.347415,
0.388076
],
"n_test": 1000,
"score": 15.733333
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.157333,
"mean_agreement": 0.8374,
"mean_kappa": 0.578069,
"mean_retention": 0.710734,
"n_cases": 1000,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.973,
0.917,
0.868
],
"mean_kappa": 0.812844,
"mean_retention": 0.915254,
"retention_by_severity": [
0.9492,
0.9576,
0.839
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.855,
0.81,
0.666
],
"mean_kappa": 0.548388,
"mean_retention": 0.69774,
"retention_by_severity": [
0.7797,
0.7034,
0.6102
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.811,
0.76,
0.73
],
"mean_kappa": 0.284418,
"mean_retention": 0.392655,
"retention_by_severity": [
0.5847,
0.3898,
0.2034
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.943,
0.921,
0.863
],
"mean_kappa": 0.786434,
"mean_retention": 0.937853,
"retention_by_severity": [
0.9915,
0.9407,
0.8814
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.874,
0.803,
0.767
],
"mean_kappa": 0.458263,
"mean_retention": 0.610169,
"retention_by_severity": [
0.8051,
0.5763,
0.4492
],
"scored": true
}
},
"score": 71.073446,
"worst_retention": 0.20339
},
"calibration": {
"accuracy": 0.368,
"brier": 0.831806,
"ece": 0.206414,
"mean_confidence": 0.573045,
"overconfidence": 0.205045,
"score": 0
},
"corruption": {
"clean_reference_cc": 0.157333,
"mean_agreement": 0.779524,
"mean_kappa": 0.59831,
"mean_retention": 0.774818,
"n_cases": 1000,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.507,
0.306,
0.298
],
"mean_kappa": 0.071023,
"mean_retention": 0.129944,
"retention_by_severity": [
0.3898,
0,
0
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.878,
0.771,
0.634
],
"mean_kappa": 0.546048,
"mean_retention": 0.912429,
"retention_by_severity": [
1,
0.8898,
0.8475
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.968,
0.931,
0.876
],
"mean_kappa": 0.826034,
"mean_retention": 0.977401,
"retention_by_severity": [
0.9915,
0.9831,
0.9576
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.943,
0.85,
0.741
],
"mean_kappa": 0.65916,
"mean_retention": 0.822034,
"retention_by_severity": [
0.9746,
0.822,
0.6695
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.912,
0.56,
0.728
],
"mean_kappa": 0.497209,
"mean_retention": 0.776836,
"retention_by_severity": [
0.9407,
0.6864,
0.7034
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.981,
0.964,
0.898
],
"mean_kappa": 0.875945,
"mean_retention": 0.935028,
"retention_by_severity": [
1,
0.9831,
0.822
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.95,
0.875,
0.799
],
"mean_kappa": 0.712754,
"mean_retention": 0.870056,
"retention_by_severity": [
1,
0.8729,
0.7373
],
"scored": true
}
},
"score": 77.48184,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.157333,
"mean_retention": 0.282486,
"per_budget": {
"n100": {
"chance_corrected": 0.076,
"n_labels": 400,
"retention": 0.483051
},
"n20": {
"chance_corrected": 0.057333,
"n_labels": 80,
"retention": 0.364407
},
"n5": {
"chance_corrected": 0,
"n_labels": 20,
"retention": 0
}
},
"score": 28.248588
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0
},
"uncertainty": {
"full_accuracy": 0.368,
"risk_coverage_auc": 0.430789,
"score": 24.105241,
"selective_acc_at_50": 0.412,
"selective_acc_at_80": 0.39125
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 97477,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "OCT",
"n_classes": 4,
"n_test": 1000,
"runtime_s": 1.1,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "octmnist",
"task_name": "Retinal OCT (4-class)",
"timestamp": "2026-08-01T19:35:24+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 69.332005
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.622061,
"auc": 0.915369,
"balanced_accuracy": 0.59728,
"ci95": [
0.590006,
0.603566
],
"n_test": 17778,
"score": 55.700851
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.527333,
"mean_agreement": 0.821444,
"mean_kappa": 0.799742,
"mean_retention": 0.920888,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9842,
0.9583,
0.9208
],
"mean_kappa": 0.948588,
"mean_retention": 0.992781,
"retention_by_severity": [
0.9974,
0.989,
0.9919
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.7608,
0.5717,
0.3808
],
"mean_kappa": 0.520801,
"mean_retention": 0.711014,
"retention_by_severity": [
0.878,
0.7442,
0.5108
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8917,
0.7925,
0.6967
],
"mean_kappa": 0.767969,
"mean_retention": 0.931779,
"retention_by_severity": [
0.9712,
0.9373,
0.8868
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.94,
0.9133,
0.8667
],
"mean_kappa": 0.894707,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9483,
0.8783,
0.8175
],
"mean_kappa": 0.866643,
"mean_retention": 0.968864,
"retention_by_severity": [
0.9953,
0.9748,
0.9364
],
"scored": true
}
},
"score": 92.088761,
"worst_retention": 0.510823
},
"calibration": {
"accuracy": 0.622061,
"brier": 0.487636,
"ece": 0.057472,
"mean_confidence": 0.679506,
"overconfidence": 0.057445,
"score": 71.263858
},
"corruption": {
"clean_reference_cc": 0.527333,
"mean_agreement": 0.818651,
"mean_kappa": 0.797073,
"mean_retention": 0.879844,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.8058,
0.59,
0.3775
],
"mean_kappa": 0.549021,
"mean_retention": 0.725966,
"retention_by_severity": [
0.9213,
0.7571,
0.4995
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8675,
0.7108,
0.4117
],
"mean_kappa": 0.625262,
"mean_retention": 0.690617,
"retention_by_severity": [
0.921,
0.7604,
0.3905
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9808,
0.9417,
0.8575
],
"mean_kappa": 0.917286,
"mean_retention": 0.991035,
"retention_by_severity": [
1,
0.9938,
0.9793
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.965,
0.9358,
0.8625
],
"mean_kappa": 0.910989,
"mean_retention": 0.974121,
"retention_by_severity": [
0.9948,
0.9953,
0.9323
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9567,
0.4567,
0.8125
],
"mean_kappa": 0.705168,
"mean_retention": 0.81053,
"retention_by_severity": [
1,
0.4541,
0.9774
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9942,
0.9842,
0.9642
],
"mean_kappa": 0.978376,
"mean_retention": 0.996327,
"retention_by_severity": [
0.9946,
1,
0.9944
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9425,
0.9108,
0.8633
],
"mean_kappa": 0.893409,
"mean_retention": 0.970312,
"retention_by_severity": [
0.9743,
0.9807,
0.9559
],
"scored": true
}
},
"score": 87.984401,
"worst_retention": 0.390493
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.557009,
"mean_retention": 0.708382,
"per_budget": {
"n100": {
"chance_corrected": 0.517746,
"n_labels": 1100,
"retention": 0.929511
},
"n20": {
"chance_corrected": 0.37252,
"n_labels": 220,
"retention": 0.668786
},
"n5": {
"chance_corrected": 0.29346,
"n_labels": 55,
"retention": 0.52685
}
},
"score": 70.838241
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 23.764479,
"worst_class_recall": 0.30695
},
"uncertainty": {
"full_accuracy": 0.622061,
"risk_coverage_auc": 0.849197,
"score": 83.411665,
"selective_acc_at_50": 0.879514,
"selective_acc_at_80": 0.711292
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 34561,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 17778,
"runtime_s": 1.4,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organamnist",
"task_name": "Abdominal CT organs — axial (A)",
"timestamp": "2026-08-01T19:32:33+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 75.415611
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.680136,
"auc": 0.919484,
"balanced_accuracy": 0.651902,
"ci95": [
0.64103,
0.66214
],
"n_test": 8216,
"score": 61.709198
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.606721,
"mean_agreement": 0.859222,
"mean_kappa": 0.840309,
"mean_retention": 0.932849,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.98,
0.965,
0.935
],
"mean_kappa": 0.95429,
"mean_retention": 0.989442,
"retention_by_severity": [
0.994,
0.9801,
0.9943
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.7875,
0.6183,
0.4542
],
"mean_kappa": 0.571709,
"mean_retention": 0.762258,
"retention_by_severity": [
0.9131,
0.7758,
0.5978
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9208,
0.8633,
0.7967
],
"mean_kappa": 0.840069,
"mean_retention": 0.936658,
"retention_by_severity": [
0.9648,
0.9349,
0.9103
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9633,
0.935,
0.9067
],
"mean_kappa": 0.925732,
"mean_retention": 0.998885,
"retention_by_severity": [
1,
0.9967,
1
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.965,
0.9275,
0.87
],
"mean_kappa": 0.909745,
"mean_retention": 0.977001,
"retention_by_severity": [
0.9855,
0.9765,
0.969
],
"scored": true
}
},
"score": 93.284876,
"worst_retention": 0.597813
},
"calibration": {
"accuracy": 0.680136,
"brier": 0.434168,
"ece": 0.018669,
"mean_confidence": 0.694461,
"overconfidence": 0.014325,
"score": 90.665484
},
"corruption": {
"clean_reference_cc": 0.606721,
"mean_agreement": 0.846429,
"mean_kappa": 0.824433,
"mean_retention": 0.89316,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.8792,
0.7417,
0.5392
],
"mean_kappa": 0.684042,
"mean_retention": 0.79525,
"retention_by_severity": [
0.957,
0.8417,
0.5871
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8867,
0.7058,
0.465
],
"mean_kappa": 0.643751,
"mean_retention": 0.717542,
"retention_by_severity": [
0.958,
0.7369,
0.4577
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.98,
0.9467,
0.8833
],
"mean_kappa": 0.927682,
"mean_retention": 0.998335,
"retention_by_severity": [
1,
1,
0.995
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9733,
0.9475,
0.885
],
"mean_kappa": 0.92602,
"mean_retention": 0.972075,
"retention_by_severity": [
0.9954,
0.9784,
0.9424
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9575,
0.4025,
0.8558
],
"mean_kappa": 0.694002,
"mean_retention": 0.788738,
"retention_by_severity": [
0.9982,
0.3748,
0.9932
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9958,
0.9808,
0.9683
],
"mean_kappa": 0.97905,
"mean_retention": 0.995171,
"retention_by_severity": [
0.9959,
0.9974,
0.9922
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.965,
0.9392,
0.8767
],
"mean_kappa": 0.916488,
"mean_retention": 0.985007,
"retention_by_severity": [
0.9886,
0.9874,
0.9791
],
"scored": true
}
},
"score": 89.315972,
"worst_retention": 0.374779
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.617092,
"mean_retention": 0.653439,
"per_budget": {
"n100": {
"chance_corrected": 0.565403,
"n_labels": 1100,
"retention": 0.916239
},
"n20": {
"chance_corrected": 0.396266,
"n_labels": 220,
"retention": 0.642151
},
"n5": {
"chance_corrected": 0.248026,
"n_labels": 55,
"retention": 0.401927
}
},
"score": 65.343896
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 38.236659,
"worst_class_recall": 0.438515
},
"uncertainty": {
"full_accuracy": 0.680136,
"risk_coverage_auc": 0.880639,
"score": 86.870293,
"selective_acc_at_50": 0.915044,
"selective_acc_at_80": 0.77651
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 12975,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 8216,
"runtime_s": 2.6,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organcmnist",
"task_name": "Abdominal CT organs — coronal (C)",
"timestamp": "2026-08-01T19:27:33+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 62.349258
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.461652,
"auc": 0.854527,
"balanced_accuracy": 0.423232,
"ci95": [
0.413252,
0.435102
],
"n_test": 8827,
"score": 36.555468
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.388254,
"mean_agreement": 0.799556,
"mean_kappa": 0.763662,
"mean_retention": 0.91653,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.98,
0.9475,
0.8967
],
"mean_kappa": 0.930346,
"mean_retention": 0.983284,
"retention_by_severity": [
0.9849,
0.9889,
0.976
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.7575,
0.5642,
0.4233
],
"mean_kappa": 0.502442,
"mean_retention": 0.716079,
"retention_by_severity": [
0.8956,
0.7201,
0.5325
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8475,
0.7367,
0.6425
],
"mean_kappa": 0.699895,
"mean_retention": 0.907588,
"retention_by_severity": [
0.964,
0.9274,
0.8313
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9275,
0.8975,
0.8342
],
"mean_kappa": 0.864919,
"mean_retention": 0.989367,
"retention_by_severity": [
1,
1,
0.9681
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9283,
0.85,
0.76
],
"mean_kappa": 0.820706,
"mean_retention": 0.986331,
"retention_by_severity": [
0.9957,
0.9927,
0.9706
],
"scored": true
}
},
"score": 91.652986,
"worst_retention": 0.532521
},
"calibration": {
"accuracy": 0.461652,
"brier": 0.667443,
"ece": 0.046413,
"mean_confidence": 0.508064,
"overconfidence": 0.046413,
"score": 76.793707
},
"corruption": {
"clean_reference_cc": 0.388254,
"mean_agreement": 0.804365,
"mean_kappa": 0.7661,
"mean_retention": 0.864746,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.82,
0.6575,
0.48
],
"mean_kappa": 0.589129,
"mean_retention": 0.790973,
"retention_by_severity": [
0.929,
0.821,
0.6229
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8175,
0.6483,
0.4425
],
"mean_kappa": 0.554936,
"mean_retention": 0.600818,
"retention_by_severity": [
0.8528,
0.6631,
0.2865
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9633,
0.8983,
0.8233
],
"mean_kappa": 0.874433,
"mean_retention": 0.996695,
"retention_by_severity": [
1,
1,
0.9901
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9525,
0.9033,
0.8167
],
"mean_kappa": 0.870178,
"mean_retention": 0.959112,
"retention_by_severity": [
0.9761,
0.983,
0.9183
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9308,
0.395,
0.7642
],
"mean_kappa": 0.640753,
"mean_retention": 0.756245,
"retention_by_severity": [
0.9795,
0.3832,
0.9061
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9883,
0.9817,
0.9483
],
"mean_kappa": 0.96774,
"mean_retention": 0.989798,
"retention_by_severity": [
0.9879,
0.9956,
0.9858
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9383,
0.8975,
0.8242
],
"mean_kappa": 0.865529,
"mean_retention": 0.959581,
"retention_by_severity": [
0.9801,
0.9609,
0.9378
],
"scored": true
}
},
"score": 86.474591,
"worst_retention": 0.286532
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.365555,
"mean_retention": 0.649001,
"per_budget": {
"n100": {
"chance_corrected": 0.324023,
"n_labels": 1100,
"retention": 0.886386
},
"n20": {
"chance_corrected": 0.240722,
"n_labels": 220,
"retention": 0.65851
},
"n5": {
"chance_corrected": 0.146992,
"n_labels": 55,
"retention": 0.402106
}
},
"score": 64.900078
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 16.696629,
"worst_class_recall": 0.242697
},
"uncertainty": {
"full_accuracy": 0.461652,
"risk_coverage_auc": 0.621152,
"score": 58.326703,
"selective_acc_at_50": 0.608065,
"selective_acc_at_80": 0.517134
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 13932,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 8827,
"runtime_s": 1.4,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organsmnist",
"task_name": "Abdominal CT organs — sagittal (S)",
"timestamp": "2026-08-01T19:30:18+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 56.345269
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.615181,
"auc": 0.894197,
"balanced_accuracy": 0.531254,
"ci95": [
0.520684,
0.540598
],
"n_test": 7180,
"score": 47.266068
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.476487,
"mean_agreement": 0.708722,
"mean_kappa": 0.666775,
"mean_retention": 0.741703,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9625,
0.9483,
0.9167
],
"mean_kappa": 0.932457,
"mean_retention": 0.991285,
"retention_by_severity": [
0.9808,
0.9931,
1
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.935,
0.8933,
0.8475
],
"mean_kappa": 0.872937,
"mean_retention": 0.9742,
"retention_by_severity": [
0.9827,
0.984,
0.9559
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.7,
0.5125,
0.3833
],
"mean_kappa": 0.448856,
"mean_retention": 0.509481,
"retention_by_severity": [
0.6873,
0.4843,
0.3568
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9608,
0.955,
0.9433
],
"mean_kappa": 0.944819,
"mean_retention": 0.990881,
"retention_by_severity": [
0.9804,
0.9922,
1
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.3042,
0.2108,
0.1575
],
"mean_kappa": 0.134807,
"mean_retention": 0.242666,
"retention_by_severity": [
0.4504,
0.2005,
0.0771
],
"scored": true
}
},
"score": 74.170252,
"worst_retention": 0.07711
},
"calibration": {
"accuracy": 0.615181,
"brier": 0.530412,
"ece": 0.084469,
"mean_confidence": 0.588216,
"overconfidence": -0.026965,
"score": 57.765351
},
"corruption": {
"clean_reference_cc": 0.476487,
"mean_agreement": 0.774048,
"mean_kappa": 0.740166,
"mean_retention": 0.812369,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.3075,
0.245,
0.2525
],
"mean_kappa": 0.180956,
"mean_retention": 0.296905,
"retention_by_severity": [
0.4337,
0.2247,
0.2323
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.9442,
0.9217,
0.9075
],
"mean_kappa": 0.910961,
"mean_retention": 0.986508,
"retention_by_severity": [
0.9895,
0.9806,
0.9894
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9692,
0.96,
0.9367
],
"mean_kappa": 0.947391,
"mean_retention": 0.990644,
"retention_by_severity": [
0.9897,
0.9926,
0.9896
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9408,
0.9,
0.7758
],
"mean_kappa": 0.849918,
"mean_retention": 0.944022,
"retention_by_severity": [
0.9851,
1,
0.847
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.965,
0.2492,
0.915
],
"mean_kappa": 0.658346,
"mean_retention": 0.697347,
"retention_by_severity": [
0.9881,
0.1276,
0.9764
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9625,
0.915,
0.8217
],
"mean_kappa": 0.882251,
"mean_retention": 0.912427,
"retention_by_severity": [
0.9785,
0.9303,
0.8285
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.905,
0.8442,
0.6167
],
"mean_kappa": 0.751342,
"mean_retention": 0.858728,
"retention_by_severity": [
1,
0.9446,
0.6316
],
"scored": true
}
},
"score": 81.23686,
"worst_retention": 0.127558
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.472661,
"mean_retention": 0.599947,
"per_budget": {
"n100": {
"chance_corrected": 0.376546,
"n_labels": 900,
"retention": 0.796651
},
"n20": {
"chance_corrected": 0.222605,
"n_labels": 180,
"retention": 0.470961
},
"n5": {
"chance_corrected": 0.251563,
"n_labels": 45,
"retention": 0.532227
}
},
"score": 59.994663
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0.099762
},
"uncertainty": {
"full_accuracy": 0.615181,
"risk_coverage_auc": 0.725792,
"score": 69.151655,
"selective_acc_at_50": 0.755989,
"selective_acc_at_80": 0.661212
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 89996,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "Histopathology",
"n_classes": 9,
"n_test": 7180,
"runtime_s": 3.5,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "pathmnist",
"task_name": "Colorectal pathology (9-class tissue)",
"timestamp": "2026-08-01T19:41:36+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 65.670714
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.844551,
"auc": 0.918847,
"balanced_accuracy": 0.797009,
"ci95": [
0.767287,
0.830566
],
"n_test": 624,
"score": 59.401709
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.594017,
"mean_agreement": 0.933868,
"mean_kappa": 0.778251,
"mean_retention": 0.822158,
"n_cases": 624,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9904,
0.9872,
0.9551
],
"mean_kappa": 0.936401,
"mean_retention": 0.953957,
"retention_by_severity": [
0.9856,
0.9856,
0.8906
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8846,
0.8013,
0.766
],
"mean_kappa": 0.288373,
"mean_retention": 0.261391,
"retention_by_severity": [
0.5683,
0.2086,
0.0072
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9744,
0.9455,
0.9311
],
"mean_kappa": 0.87252,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9808,
0.9696,
0.9567
],
"mean_kappa": 0.91008,
"mean_retention": 0.895444,
"retention_by_severity": [
0.9396,
0.8892,
0.8576
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9744,
0.9535,
0.9375
],
"mean_kappa": 0.883883,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
}
},
"score": 82.215827,
"worst_retention": 0.007194
},
"calibration": {
"accuracy": 0.844551,
"brier": 0.254701,
"ece": 0.095872,
"mean_confidence": 0.936794,
"overconfidence": 0.092243,
"score": 52.064128
},
"corruption": {
"clean_reference_cc": 0.594017,
"mean_agreement": 0.932692,
"mean_kappa": 0.772333,
"mean_retention": 0.779102,
"n_cases": 624,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9856,
0.9455,
0.8798
],
"mean_kappa": 0.796469,
"mean_retention": 0.743885,
"retention_by_severity": [
0.9353,
0.7784,
0.518
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.9135,
0.8221,
0.7644
],
"mean_kappa": 0.352076,
"mean_retention": 0.309353,
"retention_by_severity": [
0.6691,
0.259,
0
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9872,
0.9631,
0.9038
],
"mean_kappa": 0.848963,
"mean_retention": 0.814388,
"retention_by_severity": [
0.9424,
0.8604,
0.6403
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9824,
0.9663,
0.9375
],
"mean_kappa": 0.894472,
"mean_retention": 0.965468,
"retention_by_severity": [
0.9928,
1,
0.9036
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9487,
0.8269,
0.9327
],
"mean_kappa": 0.675721,
"mean_retention": 0.694964,
"retention_by_severity": [
0.8043,
0.2806,
1
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9952,
0.9904,
0.976
],
"mean_kappa": 0.964128,
"mean_retention": 0.97554,
"retention_by_severity": [
1,
0.9741,
0.9525
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9808,
0.9583,
0.9263
],
"mean_kappa": 0.8745,
"mean_retention": 0.95012,
"retention_by_severity": [
1,
0.9741,
0.8763
],
"scored": true
}
},
"score": 77.910243,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.594017,
"mean_retention": 0.92518,
"per_budget": {
"n100": {
"chance_corrected": 0.623077,
"n_labels": 200,
"retention": 1
},
"n20": {
"chance_corrected": 0.623077,
"n_labels": 40,
"retention": 1
},
"n5": {
"chance_corrected": 0.460684,
"n_labels": 10,
"retention": 0.77554
}
},
"score": 92.517986
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 21.367521,
"worst_class_recall": 0.606838
},
"uncertainty": {
"full_accuracy": 0.844551,
"risk_coverage_auc": 0.918791,
"score": 83.758137,
"selective_acc_at_50": 0.942308,
"selective_acc_at_80": 0.8998
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 4708,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Chest X-ray",
"n_classes": 2,
"n_test": 624,
"runtime_s": 0.7,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "pneumoniamnist",
"task_name": "Paediatric chest X-ray (pneumonia)",
"timestamp": "2026-08-01T19:16:13+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 53.699376
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.4975,
"auc": 0.730988,
"balanced_accuracy": 0.320259,
"ci95": [
0.285737,
0.358014
],
"n_test": 400,
"score": 15.032374
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.150324,
"mean_agreement": 0.787333,
"mean_kappa": 0.681825,
"mean_retention": 0.932082,
"n_cases": 400,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.95,
0.885,
0.8125
],
"mean_kappa": 0.821129,
"mean_retention": 0.979138,
"retention_by_severity": [
1,
0.9374,
1
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.7,
0.6775,
0.6225
],
"mean_kappa": 0.465532,
"mean_retention": 0.810539,
"retention_by_severity": [
0.6959,
0.8357,
0.9
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9,
0.8,
0.7
],
"mean_kappa": 0.71067,
"mean_retention": 0.999816,
"retention_by_severity": [
1,
1,
0.9994
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.915,
0.865,
0.81
],
"mean_kappa": 0.786931,
"mean_retention": 0.870919,
"retention_by_severity": [
0.9234,
0.8543,
0.8351
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.88,
0.7425,
0.55
],
"mean_kappa": 0.624865,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
}
},
"score": 93.208241,
"worst_retention": 0.695899
},
"calibration": {
"accuracy": 0.4975,
"brier": 0.604105,
"ece": 0.104476,
"mean_confidence": 0.593769,
"overconfidence": 0.096269,
"score": 47.761785
},
"corruption": {
"clean_reference_cc": 0.150324,
"mean_agreement": 0.804167,
"mean_kappa": 0.696974,
"mean_retention": 0.946749,
"n_cases": 400,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.765,
0.695,
0.6575
],
"mean_kappa": 0.504595,
"mean_retention": 0.814745,
"retention_by_severity": [
0.9711,
0.7935,
0.6796
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.87,
0.7625,
0.67
],
"mean_kappa": 0.649497,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.925,
0.855,
0.7925
],
"mean_kappa": 0.779839,
"mean_retention": 0.929124,
"retention_by_severity": [
0.9532,
0.9212,
0.913
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.8875,
0.825,
0.6775
],
"mean_kappa": 0.700299,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.89,
0.76,
0.7025
],
"mean_kappa": 0.667854,
"mean_retention": 0.990772,
"retention_by_severity": [
0.9723,
1,
1
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.975,
0.94,
0.8375
],
"mean_kappa": 0.875792,
"mean_retention": 0.991848,
"retention_by_severity": [
1,
0.9755,
1
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.8575,
0.82,
0.7225
],
"mean_kappa": 0.70094,
"mean_retention": 0.900754,
"retention_by_severity": [
0.9872,
0.9053,
0.8097
],
"scored": true
}
},
"score": 94.674899,
"worst_retention": 0.679626
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.150324,
"mean_retention": 0.750926,
"per_budget": {
"n100": {
"chance_corrected": 0.155867,
"n_labels": 466,
"retention": 1
},
"n20": {
"chance_corrected": 0.129645,
"n_labels": 100,
"retention": 0.862442
},
"n5": {
"chance_corrected": 0.058677,
"n_labels": 25,
"retention": 0.390337
}
},
"score": 75.092625
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0
},
"uncertainty": {
"full_accuracy": 0.4975,
"risk_coverage_auc": 0.707039,
"score": 63.379869,
"selective_acc_at_50": 0.695,
"selective_acc_at_80": 0.578125
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 1080,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Fundus",
"n_classes": 5,
"n_test": 400,
"runtime_s": 0.9,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "retinamnist",
"task_name": "Fundus photography (DR grade)",
"timestamp": "2026-08-01T19:13:08+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 52.053348
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.476248,
"auc": 0.758633,
"balanced_accuracy": 0.271854,
"ci95": [
0.269391,
0.27523
],
"n_test": 47280,
"score": 16.783286
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.167407,
"mean_agreement": 0.749167,
"mean_kappa": 0.637555,
"mean_retention": 0.765265,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9683,
0.9308,
0.8192
],
"mean_kappa": 0.847675,
"mean_retention": 0.949708,
"retention_by_severity": [
0.9838,
0.9604,
0.9049
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.895,
0.805,
0.7033
],
"mean_kappa": 0.681426,
"mean_retention": 0.815343,
"retention_by_severity": [
0.9211,
0.8228,
0.7022
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.7042,
0.4992,
0.3967
],
"mean_kappa": 0.343486,
"mean_retention": 0.484752,
"retention_by_severity": [
0.8161,
0.4368,
0.2014
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.9608,
0.9317,
0.9033
],
"mean_kappa": 0.892156,
"mean_retention": 0.92431,
"retention_by_severity": [
0.9574,
0.906,
0.9095
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.6908,
0.555,
0.4742
],
"mean_kappa": 0.423033,
"mean_retention": 0.652212,
"retention_by_severity": [
0.8728,
0.6359,
0.4479
],
"scored": true
}
},
"score": 76.526514,
"worst_retention": 0.201376
},
"calibration": {
"accuracy": 0.476248,
"brier": 0.679199,
"ece": 0.030296,
"mean_confidence": 0.460418,
"overconfidence": -0.01583,
"score": 84.852063
},
"corruption": {
"clean_reference_cc": 0.167407,
"mean_agreement": 0.795675,
"mean_kappa": 0.652875,
"mean_retention": 0.773169,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.595,
0.5325,
0.505
],
"mean_kappa": 0.178008,
"mean_retention": 0.434911,
"retention_by_severity": [
0.5053,
0.3369,
0.4625
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8283,
0.7092,
0.6125
],
"mean_kappa": 0.482805,
"mean_retention": 0.445187,
"retention_by_severity": [
0.7131,
0.4382,
0.1842
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9717,
0.9225,
0.8417
],
"mean_kappa": 0.857357,
"mean_retention": 0.88111,
"retention_by_severity": [
0.9675,
0.8958,
0.78
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.9367,
0.8133,
0.6125
],
"mean_kappa": 0.633993,
"mean_retention": 0.812392,
"retention_by_severity": [
0.9871,
0.8741,
0.576
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9533,
0.5942,
0.8833
],
"mean_kappa": 0.720951,
"mean_retention": 0.853956,
"retention_by_severity": [
0.9466,
0.6925,
0.9228
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9783,
0.9367,
0.7692
],
"mean_kappa": 0.845181,
"mean_retention": 0.986616,
"retention_by_severity": [
1,
1,
0.9598
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.9383,
0.9175,
0.8575
],
"mean_kappa": 0.851827,
"mean_retention": 0.998009,
"retention_by_severity": [
0.994,
1,
1
],
"scored": true
}
},
"score": 77.316863,
"worst_retention": 0.184215
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.167833,
"mean_retention": 0.63976,
"per_budget": {
"n100": {
"chance_corrected": 0.158844,
"n_labels": 800,
"retention": 0.946442
},
"n20": {
"chance_corrected": 0.094691,
"n_labels": 160,
"retention": 0.5642
},
"n5": {
"chance_corrected": 0.068583,
"n_labels": 40,
"retention": 0.408639
}
},
"score": 63.976023
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0
},
"uncertainty": {
"full_accuracy": 0.476248,
"risk_coverage_auc": 0.589177,
"score": 53.048767,
"selective_acc_at_50": 0.592259,
"selective_acc_at_80": 0.524165
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 165466,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "Microscopy",
"n_classes": 8,
"n_test": 47280,
"runtime_s": 1.8,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "tissuemnist",
"task_name": "Kidney cortex cells (8-class)",
"timestamp": "2026-08-01T19:37:53+00:00"
}
],
"heldout": {
"entries": [
{
"direction": "no gap beyond sampling error",
"equating_offset": 0,
"gap": 3.831693,
"gap_se": 2.35256,
"generation": "gen1",
"merkle_root": "61c508cbafed177bb1553f05aa2cc80d4157adc1bc1ff611d88581aab9ca61c5",
"model": "pca_logreg",
"score_private_equated": 65.53647,
"score_private_raw": 65.53647,
"score_public": 69.368163,
"set_hash": "641ab45baeaf445a1057c060eb9b35205bd55fcde4e629b2b88cfa38fba30047",
"submitter": "Submitter A",
"verdict": "clear",
"version_tag": "v1",
"z": 1.628733,
"z_crit": 3
},
{
"direction": "no gap beyond sampling error",
"equating_offset": 13.753223,
"gap": 6.868102,
"gap_se": 4.571743,
"generation": "gen2",
"merkle_root": "4adc9bbb0bc17564ad2effaa4d6fc902a0f319bb3e26735ac67d1b988cbe6e2d",
"model": "pca_logreg",
"score_private_equated": 62.123559,
"score_private_raw": 48.370336,
"score_public": 68.991661,
"set_hash": "ad30d214daa285c5123df991dd0eae58038f99f3490b840299cb620279ff2e73",
"submitter": "Submitter A",
"verdict": "clear",
"version_tag": "v2",
"z": 1.502294,
"z_crit": 3
}
],
"labelling": "PUBLIC-DATA DEMONSTRATION OF THE PRIVATE PROTOCOL"
},
"sources": [
{
"path": "results/cross_site/organ_ct__pca_logreg.json",
"sha256": "17a80d5e8afe2ac5601b3b76ad575ad648bff0063f20c5e052e5c6dcb38e1ef1"
},
{
"path": "results/heldout/audit_report.json",
"sha256": "df460b6d91a6512dfa1e7dc650637fce2b4dd3a1ec32e03446bf6e11f9c597d2"
},
{
"path": "results/heldout/cycle.json",
"sha256": "e012b8978457b51b5c31d18d14d2cc92f100731897523306acae19934f943502"
},
{
"path": "results/heldout/manifests/gen1.seal.json",
"sha256": "91a7adfa29870a0719125cdd48f6dcb149f1f6485ac1cfa104c6e00057ad9b81"
},
{
"path": "results/heldout/manifests/gen2.seal.json",
"sha256": "f61470918a09cf39e074ef14fb7ade4ac3a13d36d9385f18cf9b92e4b5329547"
},
{
"path": "results/heldout/manifests/public.seal.json",
"sha256": "1a4fd930d2e781ac7163483990f15d6694e28e507d1b3ca009ff09395e47497c"
},
{
"path": "results/records/bloodmnist__pca_logreg.json",
"sha256": "5092aa652607bc98f4a2d98316aadfa0cfa9ec0545d3f448dd35f9d4660306cf"
},
{
"path": "results/records/breastmnist__pca_logreg.json",
"sha256": "f869c03ae4ac7519290106cd1070a1569ad8264a833c2a7e33e8ca3233575b62"
},
{
"path": "results/records/dermamnist__pca_logreg.json",
"sha256": "52dae364cf5789ca7d2240146a5e5d3db08b31224638f3b130cfbf822ab9e016"
},
{
"path": "results/records/octmnist__pca_logreg.json",
"sha256": "6e204cf19896c2a73b74e3659ceb4bc8ebaf715d064a8642146d3cd540df501f"
},
{
"path": "results/records/organamnist__pca_logreg.json",
"sha256": "bfc86f4705a0666d05500a5e3f89ca3735d58fc50bdde1f6bb136953da953390"
},
{
"path": "results/records/organcmnist__pca_logreg.json",
"sha256": "2cfd9d683d499a69fa6129eabf99e43228b827baa4d181d4a695c060e498c766"
},
{
"path": "results/records/organsmnist__pca_logreg.json",
"sha256": "4755a3c534ea3464e9f38479db5c6e1c2858bbd9048df338f28a85babd82931c"
},
{
"path": "results/records/pathmnist__pca_logreg.json",
"sha256": "1b6e554ea95a6e966c40eba60b899e97972b4b25ead7fb1ec685402c6549dd90"
},
{
"path": "results/records/pneumoniamnist__pca_logreg.json",
"sha256": "12f6156a7d6377f782eac7d8e71f9dde2f6bae40982c4ffcbbde8b5d9b25bcfa"
},
{
"path": "results/records/retinamnist__pca_logreg.json",
"sha256": "3bb482d1c1fd7fc1d8f87471af555941534d97bc12267c02cb1d6f928487e600"
},
{
"path": "results/records/tissuemnist__pca_logreg.json",
"sha256": "0f9ebf50fa49b406c7679cf1d48f7ae547926223257e0fbcceb27c6bdc5dca8d"
}
]
},
"declared": null,
"declared_note": "No vendor declaration. This is an open reference baseline implemented by NakedSignal, not a commercial product: there is no vendor to assert an intended use or to sign an attestation, so the declared block is null rather than filled in on the vendor's behalf.",
"expiry_basis": "the sealed held-out generation this evidence cycle is anchored to (gen2, set hash ad30d214daa285c5...) is published to expire 2026-09-20; per spec, a passport lapses when its generation retires or its spec version is superseded, whichever comes first",
"expiry_utc": "2026-09-20T00:00:00Z",
"issued_utc": "2026-08-03T16:03:42Z",
"issuer": {
"algo": "Ed25519",
"key_id": "ns-passport-2026-07",
"name": "NakedSignal",
"public_key_b64": "0PBCC6mjzcppR5QdPcK7vP/AamB6MP8l1T4ypMYbsMw="
},
"observed": null,
"observed_note": "Not available: this requires continuous monitoring. Production history and drift are observed fields, and nothing is deployed, so there is nothing to observe. The object is present and null on purpose. The shape is the roadmap, not a claim.",
"passport_version": "0.1",
"signature": "N+FNMZ/oLyO+6BHYMRVS0fLcSDUQkRaoalVKFL1kSJwG4UkDYykuEixsXccEZ66pPEQx7oJgkqqExmJBtOYCBQ==",
"spec_version": "MedEval-1 v0.1",
"subject": {
"behaviour_version": "1.0.0",
"code_fingerprint": "b444c02112cc",
"code_fingerprints": [
"b444c02112cc"
],
"family": "classical",
"model_id": "pca_logreg",
"model_name": "PCA-64 + logistic regression",
"params": "up to 64 components, L2, lbfgs, C=1"
}
}Canonicalisation: RFC 8785 (JCS)-compatible: UTF-8, object keys sorted by code point, no insignificant whitespace, ECMAScript number formatting. every float is rounded once at emit to 6 decimal places (Python round(), banker's rounding); integers are left exact. Signed over the whole document with the `signature` member removed, canonicalised as above and encoded UTF-8. The public key for ns-passport-2026-07 is 0PBCC6mjzcppR5QdPcK7vP/AamB6MP8l1T4ypMYbsMw= — see Governance.