Back to the consoleDocumentation
HOG + logistic regression
LIVEThe signed evidence record for hog_logreg. Everything below was read out of the passport file; the signature was checked when this page was built, by the same code the Verify button runs.
SIGNATURE VALIDexit 0signature valid, issued by NakedSignal under key ns-passport-2026-07, not expired
SubjectHOG + logistic regression
hog_logregConfiguration9 orientations, 7x7 cells, 2x2 blocks
Code fingerprintb444c02112cc
Spec versionMedEval-1 v0.1
Issued2026-08-03T16:03:42Z by NakedSignal
Expires2026-09-20T00:00:00Z — the sealed held-out generation this evidence cycle is anchored to (gen2, set hash ad30d214daa285c5...) is published to expire 2026-09-20; per spec, a passport lapses when its generation retires or its spec version is superseded, whichever comes first
key ns-passport-2026-07 · Ed25519
The check runs against the public key published on the governance page, using your browser's own Ed25519 implementation. The exit code shown is the exit code verify_passport.py returns for the same document.
Measured by NakedSignal
Computed in this repo by the MedEval-1 harness on data the model had never seen. Every number below was copied out of a file the harness wrote — 17 of them, each listed with its SHA-256 at the foot of this panel — and is printed exactly as it was signed, not re-rounded for display.
| Task | Domain | n test | accuracy | calibration | uncertainty | subgroup | corruption | acquisition | limited data | cost | spec sensitivity | Index |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bloodmnist | medicine | 3421 | 55.811884 | 76.403917 | 80.195542 | 27.162978 | 71.741488 | 81.584001 | 58.397083 | n/a | n/a | 64.631521 |
| breastmnist | medicine | 156 | 32.080201 | 24.667196 | 64.458295 | 0 | 66.034226 | 79.010417 | 82.161458 | n/a | n/a | 48.018847 |
| dermamnist | medicine | 2005 | 10.285797 | 74.070753 | 78.789671 | 0 | 45.517303 | 70.569094 | 89.536484 | n/a | n/a | 46.476005 |
| octmnist | medicine | 1000 | 34.666667 | 0 | 53.557877 | 0 | 54.542125 | 69.74359 | 65.897436 | n/a | n/a | 39.402992 |
| organamnist | medicine | 17778 | 66.137654 | 55.604065 | 89.559015 | 42.053571 | 75.849148 | 83.649363 | 68.964388 | n/a | n/a | 68.335442 |
| organcmnist | medicine | 8216 | 71.196888 | 52.434091 | 91.490629 | 53.039443 | 83.30109 | 89.849529 | 79.972192 | n/a | n/a | 73.811877 |
| organsmnist | medicine | 8827 | 48.254782 | 33.116598 | 72.128816 | 27.619048 | 72.614422 | 80.475974 | 58.34461 | n/a | n/a | 55.959617 |
| pathmnist | medicine | 7180 | 25.798203 | 50.897211 | 54.028993 | 1.315789 | 56.365997 | 75.101552 | 44.740221 | n/a | n/a | 44.228081 |
| pneumoniamnist | medicine | 624 | 56.752137 | 44.456178 | 89.213252 | 17.094017 | 55.335628 | 68.654618 | 96.134538 | n/a | n/a | 57.914465 |
| retinamnist | medicine | 400 | 8.165292 | 0 | 35.46052 | 0 | 68.151168 | 73.489284 | 100 | n/a | n/a | 37.757002 |
| tissuemnist | medicine | 47280 | 17.231657 | 75.699094 | 54.199143 | 0 | 53.15076 | 68.534744 | 50.538977 | n/a | n/a | 44.356107 |
Cross-site retention · organ_ct
retention is absolute retained skill (chance-corrected), per spec 2.2.1; the diagonal is same-site, every off-diagonal cell is train-on-row, test-on-column
Sitesorganamnist, organcmnist, organsmnist
Mean retention0.452197
Worst retention0.171669
Private held-out track
PUBLIC-DATA DEMONSTRATION OF THE PRIVATE PROTOCOL
| Generation | Public | Private (equated) | Gap | z | Verdict |
|---|---|---|---|---|---|
| gen1 | 50.824189 | 52.592004 | -1.767815 | -0.675305 | clear |
| gen2 | 53.33675 | 59.599916 | -6.263166 | -1.348959 | clear |
Known limitations
Subgroup metadataabsent on 11 of 11 corpora, so the subgroup score falls back to the worst diagnostic class. That absence is a disclosure, not a pass.
Training set cappedyes on at least one task — see the raw JSON for which
Observed in deploymentnothing — see the third panel
Audit chain
Bundles re-derived48 of 48
Ledger100 entries, head 6b7b371ff650a8ed…, chain intact
Re-run it yourself
python src/heldout/audit.py 1f66c7719ab3943c6fcc17 source files, with hashes
| results/cross_site/organ_ct__hog_logreg.json | c49a4fcc7dccbbd685afd6f064c5760cc7ea427c7598537fea7be72e00ab0825 |
| results/heldout/audit_report.json | df460b6d91a6512dfa1e7dc650637fce2b4dd3a1ec32e03446bf6e11f9c597d2 |
| results/heldout/cycle.json | e012b8978457b51b5c31d18d14d2cc92f100731897523306acae19934f943502 |
| results/heldout/manifests/gen1.seal.json | 91a7adfa29870a0719125cdd48f6dcb149f1f6485ac1cfa104c6e00057ad9b81 |
| results/heldout/manifests/gen2.seal.json | f61470918a09cf39e074ef14fb7ade4ac3a13d36d9385f18cf9b92e4b5329547 |
| results/heldout/manifests/public.seal.json | 1a4fd930d2e781ac7163483990f15d6694e28e507d1b3ca009ff09395e47497c |
| results/records/bloodmnist__hog_logreg.json | d2febf8b19a5988009a8aff67ab2bdde80de417107b392684b462e165d08c7e0 |
| results/records/breastmnist__hog_logreg.json | d75604762557b0864ca0a403c2996cf3aa0728b2e787008abc4bc8b4da219b9a |
| results/records/dermamnist__hog_logreg.json | ccf178af0ba9e8959491356308727a750a072c95e35ab031df057fa06d2d7b21 |
| results/records/octmnist__hog_logreg.json | c5155cc76e8b49d696181591400e7a3c6b0c7cc911b127a466120759321db0dd |
| results/records/organamnist__hog_logreg.json | 63cc89e81e87e9886976a9e7057fc8954074065c329c23b96bcc6999b49d2b09 |
| results/records/organcmnist__hog_logreg.json | 9008f98f8a75ca6168b557ad1e99faa67b1d0cffaf9d6c09da7e2b40ad4691f5 |
| results/records/organsmnist__hog_logreg.json | 1d531b0ba53255fc44411bf6b4447e30dc608e05f708ff6a12d9bc6831eb3aff |
| results/records/pathmnist__hog_logreg.json | e1c02f91a99057416458ae6eb2e93b312cfc707ffefb25e89e86856f6e054fb7 |
| results/records/pneumoniamnist__hog_logreg.json | d1aa310d240f5f0d78abe0ebaf9a3b8c4d1348af60811b9042df8f693f077bf4 |
| results/records/retinamnist__hog_logreg.json | 6d3fa677d879d539e75751fafec225977c5b801431bfece78876547c544a4339 |
| results/records/tissuemnist__hog_logreg.json | 4e6d90310286e491c2ab3fdc28e036f08175c1c4fa5ed8835b209a57f405afa2 |
Declared by vendor
Five fields a vendor asserts about its own product — intended use, forbidden use, training cutoff, regulatory clearances, and who is personally attesting. NakedSignal never fills these in on a vendor's behalf.
No vendor declaration. This is an open reference baseline implemented by NakedSignal, not a commercial product: there is no vendor to assert an intended use or to sign an attestation, so the declared block is null rather than filled in on the vendor's behalf.
Observed in deployment
Production history and drift — how the model has actually behaved since it was deployed, on real traffic.
Not available: this requires continuous monitoring. Production history and drift are observed fields, and nothing is deployed, so there is nothing to observe. The object is present and null on purpose. The shape is the roadmap, not a claim.
Raw JSON — the whole signed document, 82,442 characters
{
"canonicalisation": {
"float_rounding": "every float is rounded once at emit to 6 decimal places (Python round(), banker's rounding); integers are left exact",
"form": "RFC 8785 (JCS)-compatible: UTF-8, object keys sorted by code point, no insignificant whitespace, ECMAScript number formatting",
"signed_over": "the whole document with the `signature` member removed, canonicalised as above and encoded UTF-8"
},
"computed": {
"audit": {
"audit_command": "python src/heldout/audit.py 1f66c7719ab3943c6fcc",
"bundles": 48,
"ledger_entries": 100,
"ledger_head": "6b7b371ff650a8edaf477490f2a5f9b043433f3ba9941989ee1962ed8f1b1d88",
"ledger_intact": true,
"rederived": 48
},
"cross_site": {
"code_fingerprint": "b444c02112cc",
"device": "mps",
"family": "organ_ct",
"matrix": {
"organamnist": {
"organamnist": {
"balanced_accuracy": 0.69216,
"chance_corrected": 0.661377,
"ece": 0.088792
},
"organcmnist": {
"balanced_accuracy": 0.446522,
"chance_corrected": 0.391174,
"ece": 0.302569,
"retention": 0.591454
},
"organsmnist": {
"balanced_accuracy": 0.194126,
"chance_corrected": 0.113538,
"ece": 0.526602,
"retention": 0.171669
}
},
"organcmnist": {
"organamnist": {
"balanced_accuracy": 0.396293,
"chance_corrected": 0.335922,
"ece": 0.321512,
"retention": 0.471822
},
"organcmnist": {
"balanced_accuracy": 0.738154,
"chance_corrected": 0.711969,
"ece": 0.095132
},
"organsmnist": {
"balanced_accuracy": 0.316302,
"chance_corrected": 0.247932,
"ece": 0.396609,
"retention": 0.348235
}
},
"organsmnist": {
"organamnist": {
"balanced_accuracy": 0.214056,
"chance_corrected": 0.135462,
"ece": 0.415348,
"retention": 0.280723
},
"organcmnist": {
"balanced_accuracy": 0.463472,
"chance_corrected": 0.409819,
"ece": 0.172471,
"retention": 0.849281
},
"organsmnist": {
"balanced_accuracy": 0.529589,
"chance_corrected": 0.482548,
"ece": 0.133767
}
}
},
"mean_retention": 0.452197,
"note": "retention is absolute retained skill (chance-corrected), per spec 2.2.1; the diagonal is same-site, every off-diagonal cell is train-on-row, test-on-column",
"score": 45.219732,
"site_labels": {
"organamnist": "axial",
"organcmnist": "coronal",
"organsmnist": "sagittal"
},
"sites": [
"organamnist",
"organcmnist",
"organsmnist"
],
"spec_version": "MedEval-1 v0.1",
"worst_retention": 0.171669
},
"evaluations": [
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 64.631521
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.665887,
"auc": 0.923684,
"balanced_accuracy": 0.613354,
"ci95": [
0.598473,
0.630663
],
"n_test": 3421,
"score": 55.811884
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.564735,
"mean_agreement": 0.6775,
"mean_kappa": 0.61702,
"mean_retention": 0.81584,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.8225,
0.6942,
0.5258
],
"mean_kappa": 0.619523,
"mean_retention": 0.838667,
"retention_by_severity": [
0.9756,
0.9095,
0.631
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.7042,
0.535,
0.345
],
"mean_kappa": 0.43574,
"mean_retention": 0.606425,
"retention_by_severity": [
0.8876,
0.6628,
0.2688
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8592,
0.7167,
0.5967
],
"mean_kappa": 0.674281,
"mean_retention": 0.862558,
"retention_by_severity": [
0.9683,
0.8856,
0.7338
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.6433,
0.59,
0.5942
],
"mean_kappa": 0.539116,
"mean_retention": 0.815705,
"retention_by_severity": [
0.8363,
0.804,
0.8068
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9358,
0.8642,
0.7358
],
"mean_kappa": 0.81644,
"mean_retention": 0.955846,
"retention_by_severity": [
1,
0.9617,
0.9058
],
"scored": true
}
},
"score": 81.584001,
"worst_retention": 0.268791
},
"calibration": {
"accuracy": 0.665887,
"brier": 0.461648,
"ece": 0.047192,
"mean_confidence": 0.706034,
"overconfidence": 0.040147,
"score": 76.403917
},
"corruption": {
"clean_reference_cc": 0.564735,
"mean_agreement": 0.624603,
"mean_kappa": 0.559408,
"mean_retention": 0.717415,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9242,
0.735,
0.4183
],
"mean_kappa": 0.636921,
"mean_retention": 0.757012,
"retention_by_severity": [
1,
0.8435,
0.4276
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.9567,
0.9517,
0.9175
],
"mean_kappa": 0.930881,
"mean_retention": 0.983544,
"retention_by_severity": [
0.9942,
0.9766,
0.9798
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.715,
0.5358,
0.405
],
"mean_kappa": 0.473102,
"mean_retention": 0.712507,
"retention_by_severity": [
0.8608,
0.7064,
0.5703
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.675,
0.5258,
0.38
],
"mean_kappa": 0.427417,
"mean_retention": 0.584875,
"retention_by_severity": [
0.8268,
0.5873,
0.3405
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.6608,
0.2467,
0.2092
],
"mean_kappa": 0.286927,
"mean_retention": 0.478736,
"retention_by_severity": [
0.926,
0.3281,
0.1821
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.8875,
0.8158,
0.6767
],
"mean_kappa": 0.752377,
"mean_retention": 0.881505,
"retention_by_severity": [
0.974,
0.9125,
0.758
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.6008,
0.4975,
0.3817
],
"mean_kappa": 0.408231,
"mean_retention": 0.623724,
"retention_by_severity": [
0.7643,
0.6296,
0.4773
],
"scored": true
}
},
"score": 71.741488,
"worst_retention": 0.18211
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.558119,
"mean_retention": 0.583971,
"per_budget": {
"n100": {
"chance_corrected": 0.35594,
"n_labels": 800,
"retention": 0.637749
},
"n20": {
"chance_corrected": 0.332573,
"n_labels": 160,
"retention": 0.595882
},
"n5": {
"chance_corrected": 0.289262,
"n_labels": 40,
"retention": 0.518281
}
},
"score": 58.397083
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 27.162978,
"worst_class_recall": 0.362676
},
"uncertainty": {
"full_accuracy": 0.665887,
"risk_coverage_auc": 0.826711,
"score": 80.195542,
"selective_acc_at_50": 0.833333,
"selective_acc_at_80": 0.730362
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 11959,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Microscopy",
"n_classes": 8,
"n_test": 3421,
"runtime_s": 10.1,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "bloodmnist",
"task_name": "Peripheral blood cells (8-class)",
"timestamp": "2026-08-01T19:18:41+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 48.018847
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.75641,
"auc": 0.709691,
"balanced_accuracy": 0.660401,
"ci95": [
0.583711,
0.747874
],
"n_test": 156,
"score": 32.080201
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.320802,
"mean_agreement": 0.804274,
"mean_kappa": 0.478238,
"mean_retention": 0.790104,
"n_cases": 156,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.7821,
0.7115,
0.6346
],
"mean_kappa": 0.209513,
"mean_retention": 0.411458,
"retention_by_severity": [
0.4375,
0.2617,
0.5352
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.7885,
0.7756,
0.6859
],
"mean_kappa": 0.379427,
"mean_retention": 0.925781,
"retention_by_severity": [
0.9102,
0.9961,
0.8711
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9167,
0.8846,
0.8846
],
"mean_kappa": 0.714619,
"mean_retention": 0.994792,
"retention_by_severity": [
1,
0.9844,
1
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.7628,
0.7821,
0.7372
],
"mean_kappa": 0.349261,
"mean_retention": 0.697917,
"retention_by_severity": [
0.6602,
0.9375,
0.4961
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9808,
0.891,
0.8462
],
"mean_kappa": 0.738371,
"mean_retention": 0.920573,
"retention_by_severity": [
1,
0.7617,
1
],
"scored": true
}
},
"score": 79.010417,
"worst_retention": 0.261719
},
"calibration": {
"accuracy": 0.75641,
"brier": 0.381831,
"ece": 0.150666,
"mean_confidence": 0.907076,
"overconfidence": 0.150666,
"score": 24.667196
},
"corruption": {
"clean_reference_cc": 0.320802,
"mean_agreement": 0.779609,
"mean_kappa": 0.426671,
"mean_retention": 0.660342,
"n_cases": 156,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9936,
0.9936,
0.9744
],
"mean_kappa": 0.962523,
"mean_retention": 0.981771,
"retention_by_severity": [
0.9727,
0.9727,
1
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.9615,
0.9359,
0.9231
],
"mean_kappa": 0.825415,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.7949,
0.7244,
0.7244
],
"mean_kappa": 0.324682,
"mean_retention": 0.695313,
"retention_by_severity": [
0.9922,
0.6523,
0.4414
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.7179,
0.6154,
0.5064
],
"mean_kappa": 0.04851,
"mean_retention": 0.467448,
"retention_by_severity": [
0.5781,
0.3047,
0.5195
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.7885,
0.8013,
0.6859
],
"mean_kappa": 0.232886,
"mean_retention": 0.397135,
"retention_by_severity": [
0.6289,
0.3086,
0.2539
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.8333,
0.8141,
0.7051
],
"mean_kappa": 0.419951,
"mean_retention": 0.77474,
"retention_by_severity": [
0.9531,
0.7227,
0.6484
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.6859,
0.5962,
0.5962
],
"mean_kappa": 0.17273,
"mean_retention": 0.30599,
"retention_by_severity": [
0.4258,
0.0977,
0.3945
],
"scored": true
}
},
"score": 66.034226,
"worst_retention": 0.097656
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.320802,
"mean_retention": 0.821615,
"per_budget": {
"n100": {
"chance_corrected": 0.360902,
"n_labels": 200,
"retention": 1
},
"n20": {
"chance_corrected": 0.239348,
"n_labels": 40,
"retention": 0.746094
},
"n5": {
"chance_corrected": 0.230576,
"n_labels": 10,
"retention": 0.71875
}
},
"score": 82.161458
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0.452381
},
"uncertainty": {
"full_accuracy": 0.75641,
"risk_coverage_auc": 0.822291,
"score": 64.458295,
"selective_acc_at_50": 0.858974,
"selective_acc_at_80": 0.8
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 546,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Ultrasound",
"n_classes": 2,
"n_test": 156,
"runtime_s": 0.8,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "breastmnist",
"task_name": "Breast ultrasound (malignancy)",
"timestamp": "2026-08-01T19:12:56+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 46.476005
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.666334,
"auc": 0.720351,
"balanced_accuracy": 0.231021,
"ci95": [
0.211513,
0.253505
],
"n_test": 2005,
"score": 10.285797
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.106795,
"mean_agreement": 0.802556,
"mean_kappa": 0.486764,
"mean_retention": 0.705691,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.7158,
0.4342,
0.1233
],
"mean_kappa": 0.157704,
"mean_retention": 0.651255,
"retention_by_severity": [
0.7773,
0.8311,
0.3453
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8725,
0.8558,
0.835
],
"mean_kappa": 0.394235,
"mean_retention": 0.61217,
"retention_by_severity": [
0.7248,
0.4466,
0.6651
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9492,
0.9275,
0.9075
],
"mean_kappa": 0.756549,
"mean_retention": 0.916977,
"retention_by_severity": [
0.9439,
0.8927,
0.9143
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.875,
0.8608,
0.8508
],
"mean_kappa": 0.329567,
"mean_retention": 0.443781,
"retention_by_severity": [
0.5008,
0.4412,
0.3894
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9533,
0.9458,
0.9317
],
"mean_kappa": 0.795764,
"mean_retention": 0.904271,
"retention_by_severity": [
0.9888,
0.8656,
0.8585
],
"scored": true
}
},
"score": 70.569094,
"worst_retention": 0.345312
},
"calibration": {
"accuracy": 0.666334,
"brier": 0.480517,
"ece": 0.051858,
"mean_confidence": 0.710496,
"overconfidence": 0.044162,
"score": 74.070753
},
"corruption": {
"clean_reference_cc": 0.106795,
"mean_agreement": 0.67123,
"mean_kappa": 0.343857,
"mean_retention": 0.455173,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9892,
0.9492,
0.9092
],
"mean_kappa": 0.807337,
"mean_retention": 0.859634,
"retention_by_severity": [
0.9889,
0.9206,
0.6695
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.9383,
0.9333,
0.9133
],
"mean_kappa": 0.752428,
"mean_retention": 0.863985,
"retention_by_severity": [
0.8851,
0.8877,
0.8191
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.8892,
0.8558,
0.8408
],
"mean_kappa": 0.293163,
"mean_retention": 0.330396,
"retention_by_severity": [
0.6711,
0.2473,
0.0728
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.3842,
0.0992,
0.05
],
"mean_kappa": 0.042878,
"mean_retention": 0.222832,
"retention_by_severity": [
0.5121,
0.0007,
0.1558
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.8533,
0.82,
0.8417
],
"mean_kappa": 0.229072,
"mean_retention": 0.34396,
"retention_by_severity": [
0.6754,
0.2824,
0.0741
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.8683,
0.8458,
0.8375
],
"mean_kappa": 0.271394,
"mean_retention": 0.323755,
"retention_by_severity": [
0.7763,
0.195,
0
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.1608,
0.0683,
0.0483
],
"mean_kappa": 0.010729,
"mean_retention": 0.241651,
"retention_by_severity": [
0.4234,
0,
0.3016
],
"scored": true
}
},
"score": 45.517303,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.102858,
"mean_retention": 0.895365,
"per_budget": {
"n100": {
"chance_corrected": 0.080315,
"n_labels": 679,
"retention": 0.780833
},
"n20": {
"chance_corrected": 0.111387,
"n_labels": 140,
"retention": 1
},
"n5": {
"chance_corrected": 0.093113,
"n_labels": 35,
"retention": 0.905261
}
},
"score": 89.536484
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0
},
"uncertainty": {
"full_accuracy": 0.666334,
"risk_coverage_auc": 0.818197,
"score": 78.789671,
"selective_acc_at_50": 0.842315,
"selective_acc_at_80": 0.742519
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 7007,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Dermatoscopy",
"n_classes": 7,
"n_test": 2005,
"runtime_s": 8.8,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "dermamnist",
"task_name": "Dermatoscopy (7-class skin lesion)",
"timestamp": "2026-08-01T19:17:09+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 39.402992
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.51,
"auc": 0.820744,
"balanced_accuracy": 0.51,
"ci95": [
0.491697,
0.526785
],
"n_test": 1000,
"score": 34.666667
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.346667,
"mean_agreement": 0.711733,
"mean_kappa": 0.527885,
"mean_retention": 0.697436,
"n_cases": 1000,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.436,
0.386,
0.305
],
"mean_kappa": 0.042493,
"mean_retention": 0.025641,
"retention_by_severity": [
0.0346,
0.0154,
0.0269
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.834,
0.814,
0.796
],
"mean_kappa": 0.680545,
"mean_retention": 0.939744,
"retention_by_severity": [
0.9231,
0.9808,
0.9154
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.925,
0.873,
0.786
],
"mean_kappa": 0.765886,
"mean_retention": 0.932051,
"retention_by_severity": [
0.9615,
0.9308,
0.9038
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.721,
0.68,
0.67
],
"mean_kappa": 0.465934,
"mean_retention": 0.758974,
"retention_by_severity": [
0.7731,
0.7269,
0.7769
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.933,
0.813,
0.704
],
"mean_kappa": 0.684566,
"mean_retention": 0.830769,
"retention_by_severity": [
0.9654,
0.7846,
0.7423
],
"scored": true
}
},
"score": 69.74359,
"worst_retention": 0.015385
},
"calibration": {
"accuracy": 0.51,
"brier": 0.681143,
"ece": 0.228733,
"mean_confidence": 0.738733,
"overconfidence": 0.228733,
"score": 0
},
"corruption": {
"clean_reference_cc": 0.346667,
"mean_agreement": 0.624952,
"mean_kappa": 0.438271,
"mean_retention": 0.545421,
"n_cases": 1000,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.991,
0.985,
0.961
],
"mean_kappa": 0.963624,
"mean_retention": 0.998718,
"retention_by_severity": [
0.9962,
1,
1
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.943,
0.928,
0.903
],
"mean_kappa": 0.869948,
"mean_retention": 0.974359,
"retention_by_severity": [
1,
0.9654,
0.9577
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.767,
0.635,
0.548
],
"mean_kappa": 0.37632,
"mean_retention": 0.537179,
"retention_by_severity": [
0.8231,
0.5115,
0.2769
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.393,
0.319,
0.217
],
"mean_kappa": 0.016978,
"mean_retention": 0.024359,
"retention_by_severity": [
0,
0.0077,
0.0654
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.782,
0.166,
0.448
],
"mean_kappa": 0.30338,
"mean_retention": 0.55641,
"retention_by_severity": [
0.9115,
0.2077,
0.55
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.82,
0.76,
0.536
],
"mean_kappa": 0.517371,
"mean_retention": 0.705128,
"retention_by_severity": [
0.8462,
0.7731,
0.4962
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.409,
0.354,
0.259
],
"mean_kappa": 0.020276,
"mean_retention": 0.021795,
"retention_by_severity": [
0,
0,
0.0654
],
"scored": true
}
},
"score": 54.542125,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.346667,
"mean_retention": 0.658974,
"per_budget": {
"n100": {
"chance_corrected": 0.281333,
"n_labels": 400,
"retention": 0.811538
},
"n20": {
"chance_corrected": 0.229333,
"n_labels": 80,
"retention": 0.661538
},
"n5": {
"chance_corrected": 0.174667,
"n_labels": 20,
"retention": 0.503846
}
},
"score": 65.897436
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0.02
},
"uncertainty": {
"full_accuracy": 0.51,
"risk_coverage_auc": 0.651684,
"score": 53.557877,
"selective_acc_at_50": 0.63,
"selective_acc_at_80": 0.56375
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 97477,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "OCT",
"n_classes": 4,
"n_test": 1000,
"runtime_s": 6.5,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "octmnist",
"task_name": "Retinal OCT (4-class)",
"timestamp": "2026-08-01T19:35:30+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 68.335442
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.730116,
"auc": 0.948576,
"balanced_accuracy": 0.69216,
"ci95": [
0.68478,
0.698633
],
"n_test": 17778,
"score": 66.137654
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.667915,
"mean_agreement": 0.703167,
"mean_kappa": 0.667831,
"mean_retention": 0.836494,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.8,
0.655,
0.455
],
"mean_kappa": 0.592092,
"mean_retention": 0.73868,
"retention_by_severity": [
0.9569,
0.7775,
0.4816
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.705,
0.6092,
0.4375
],
"mean_kappa": 0.535046,
"mean_retention": 0.701277,
"retention_by_severity": [
0.8561,
0.716,
0.5318
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8958,
0.8333,
0.7883
],
"mean_kappa": 0.819722,
"mean_retention": 0.967567,
"retention_by_severity": [
0.9919,
0.9734,
0.9373
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.6708,
0.6025,
0.5425
],
"mean_kappa": 0.559443,
"mean_retention": 0.805956,
"retention_by_severity": [
0.8612,
0.8367,
0.72
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.9108,
0.8625,
0.7792
],
"mean_kappa": 0.832851,
"mean_retention": 0.968988,
"retention_by_severity": [
0.9946,
0.9795,
0.9329
],
"scored": true
}
},
"score": 83.649363,
"worst_retention": 0.481634
},
"calibration": {
"accuracy": 0.730116,
"brier": 0.385801,
"ece": 0.088792,
"mean_confidence": 0.8189,
"overconfidence": 0.088785,
"score": 55.604065
},
"corruption": {
"clean_reference_cc": 0.667915,
"mean_agreement": 0.662738,
"mean_kappa": 0.625606,
"mean_retention": 0.758491,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9292,
0.8708,
0.7292
],
"mean_kappa": 0.824511,
"mean_retention": 0.918841,
"retention_by_severity": [
0.9786,
0.9685,
0.8095
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.9483,
0.9392,
0.9108
],
"mean_kappa": 0.924607,
"mean_retention": 0.998277,
"retention_by_severity": [
1,
1,
0.9948
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.7067,
0.6067,
0.4542
],
"mean_kappa": 0.541692,
"mean_retention": 0.748585,
"retention_by_severity": [
0.9061,
0.7802,
0.5594
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.7192,
0.5458,
0.325
],
"mean_kappa": 0.474377,
"mean_retention": 0.563255,
"retention_by_severity": [
0.8202,
0.5816,
0.288
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.6825,
0.4117,
0.2058
],
"mean_kappa": 0.384139,
"mean_retention": 0.544223,
"retention_by_severity": [
0.8692,
0.4848,
0.2787
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.8525,
0.7275,
0.5058
],
"mean_kappa": 0.660097,
"mean_retention": 0.825845,
"retention_by_severity": [
0.9661,
0.891,
0.6205
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.725,
0.6308,
0.4908
],
"mean_kappa": 0.569819,
"mean_retention": 0.710414,
"retention_by_severity": [
0.8345,
0.7188,
0.5779
],
"scored": true
}
},
"score": 75.849148,
"worst_retention": 0.278659
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.661377,
"mean_retention": 0.689644,
"per_budget": {
"n100": {
"chance_corrected": 0.53303,
"n_labels": 1100,
"retention": 0.80594
},
"n20": {
"chance_corrected": 0.481495,
"n_labels": 220,
"retention": 0.72802
},
"n5": {
"chance_corrected": 0.353818,
"n_labels": 55,
"retention": 0.534972
}
},
"score": 68.964388
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 42.053571,
"worst_class_recall": 0.473214
},
"uncertainty": {
"full_accuracy": 0.730116,
"risk_coverage_auc": 0.905082,
"score": 89.559015,
"selective_acc_at_50": 0.935538,
"selective_acc_at_80": 0.82056
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 34561,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 17778,
"runtime_s": 14.3,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organamnist",
"task_name": "Abdominal CT organs — axial (A)",
"timestamp": "2026-08-01T19:32:39+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 73.811877
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.762537,
"auc": 0.961531,
"balanced_accuracy": 0.738154,
"ci95": [
0.729918,
0.748199
],
"n_test": 8216,
"score": 71.196888
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.693623,
"mean_agreement": 0.759944,
"mean_kappa": 0.730621,
"mean_retention": 0.898495,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.8492,
0.7367,
0.5408
],
"mean_kappa": 0.671226,
"mean_retention": 0.841846,
"retention_by_severity": [
1,
0.9159,
0.6096
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.7633,
0.6367,
0.4833
],
"mean_kappa": 0.584684,
"mean_retention": 0.76938,
"retention_by_severity": [
0.925,
0.8119,
0.5712
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9275,
0.8725,
0.8217
],
"mean_kappa": 0.857618,
"mean_retention": 0.975999,
"retention_by_severity": [
0.9905,
0.9872,
0.9503
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.7483,
0.6892,
0.6625
],
"mean_kappa": 0.664728,
"mean_retention": 0.927608,
"retention_by_severity": [
0.9803,
0.9273,
0.8752
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.935,
0.8975,
0.835
],
"mean_kappa": 0.874848,
"mean_retention": 0.977644,
"retention_by_severity": [
1,
0.9887,
0.9442
],
"scored": true
}
},
"score": 89.849529,
"worst_retention": 0.571248
},
"calibration": {
"accuracy": 0.762537,
"brier": 0.350824,
"ece": 0.095132,
"mean_confidence": 0.857428,
"overconfidence": 0.094891,
"score": 52.434091
},
"corruption": {
"clean_reference_cc": 0.693623,
"mean_agreement": 0.723254,
"mean_kappa": 0.688899,
"mean_retention": 0.833011,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9567,
0.9292,
0.8433
],
"mean_kappa": 0.897906,
"mean_retention": 0.963546,
"retention_by_severity": [
1,
0.9787,
0.9119
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.96,
0.9508,
0.9342
],
"mean_kappa": 0.941619,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.7958,
0.6808,
0.5058
],
"mean_kappa": 0.623212,
"mean_retention": 0.84149,
"retention_by_severity": [
0.976,
0.8735,
0.675
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.7975,
0.6675,
0.4
],
"mean_kappa": 0.572054,
"mean_retention": 0.702493,
"retention_by_severity": [
0.9241,
0.7735,
0.4099
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.785,
0.4825,
0.3742
],
"mean_kappa": 0.490758,
"mean_retention": 0.679907,
"retention_by_severity": [
0.997,
0.5317,
0.511
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.8692,
0.7983,
0.6017
],
"mean_kappa": 0.725989,
"mean_retention": 0.893988,
"retention_by_severity": [
1,
0.9633,
0.7187
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.7483,
0.6317,
0.4758
],
"mean_kappa": 0.570757,
"mean_retention": 0.749651,
"retention_by_severity": [
0.9124,
0.7626,
0.574
],
"scored": true
}
},
"score": 83.30109,
"worst_retention": 0.409878
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.711969,
"mean_retention": 0.799722,
"per_budget": {
"n100": {
"chance_corrected": 0.639506,
"n_labels": 1100,
"retention": 0.898221
},
"n20": {
"chance_corrected": 0.564064,
"n_labels": 220,
"retention": 0.792259
},
"n5": {
"chance_corrected": 0.504562,
"n_labels": 55,
"retention": 0.708685
}
},
"score": 79.972192
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 53.039443,
"worst_class_recall": 0.573086
},
"uncertainty": {
"full_accuracy": 0.762537,
"risk_coverage_auc": 0.922642,
"score": 91.490629,
"selective_acc_at_50": 0.951558,
"selective_acc_at_80": 0.850753
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 12975,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 8216,
"runtime_s": 12.9,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organcmnist",
"task_name": "Abdominal CT organs — coronal (C)",
"timestamp": "2026-08-01T19:27:44+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 55.959617
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.572561,
"auc": 0.910188,
"balanced_accuracy": 0.529589,
"ci95": [
0.519811,
0.541522
],
"n_test": 8827,
"score": 48.254782
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.478396,
"mean_agreement": 0.639833,
"mean_kappa": 0.588619,
"mean_retention": 0.80476,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.7425,
0.5867,
0.405
],
"mean_kappa": 0.507784,
"mean_retention": 0.727903,
"retention_by_severity": [
0.9292,
0.8181,
0.4363
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.6283,
0.5225,
0.3908
],
"mean_kappa": 0.447266,
"mean_retention": 0.658726,
"retention_by_severity": [
0.8299,
0.6791,
0.4672
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8817,
0.7933,
0.7325
],
"mean_kappa": 0.772857,
"mean_retention": 0.889678,
"retention_by_severity": [
0.9461,
0.8791,
0.8439
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.555,
0.5,
0.4575
],
"mean_kappa": 0.44469,
"mean_retention": 0.791547,
"retention_by_severity": [
0.859,
0.7817,
0.7339
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.8767,
0.8117,
0.7133
],
"mean_kappa": 0.770498,
"mean_retention": 0.955944,
"retention_by_severity": [
0.9796,
0.9665,
0.9217
],
"scored": true
}
},
"score": 80.475974,
"worst_retention": 0.436349
},
"calibration": {
"accuracy": 0.572561,
"brier": 0.587273,
"ece": 0.133767,
"mean_confidence": 0.706328,
"overconfidence": 0.133767,
"score": 33.116598
},
"corruption": {
"clean_reference_cc": 0.478396,
"mean_agreement": 0.589881,
"mean_kappa": 0.532134,
"mean_retention": 0.726144,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.8925,
0.8183,
0.68
],
"mean_kappa": 0.767062,
"mean_retention": 0.928232,
"retention_by_severity": [
0.9995,
0.9464,
0.8388
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.9375,
0.9342,
0.8858
],
"mean_kappa": 0.906912,
"mean_retention": 0.983085,
"retention_by_severity": [
0.9907,
0.9882,
0.9703
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.635,
0.4758,
0.3333
],
"mean_kappa": 0.425541,
"mean_retention": 0.767172,
"retention_by_severity": [
0.9233,
0.8025,
0.5757
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.6467,
0.4925,
0.3333
],
"mean_kappa": 0.397083,
"mean_retention": 0.548704,
"retention_by_severity": [
0.8015,
0.5972,
0.2474
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.6133,
0.3933,
0.1433
],
"mean_kappa": 0.316965,
"mean_retention": 0.550947,
"retention_by_severity": [
0.9023,
0.5117,
0.2389
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.78,
0.6233,
0.3292
],
"mean_kappa": 0.52582,
"mean_retention": 0.7288,
"retention_by_severity": [
0.9542,
0.7827,
0.4495
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.58,
0.4858,
0.3742
],
"mean_kappa": 0.385557,
"mean_retention": 0.57607,
"retention_by_severity": [
0.7149,
0.6068,
0.4065
],
"scored": true
}
},
"score": 72.614422,
"worst_retention": 0.238868
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.482548,
"mean_retention": 0.583446,
"per_budget": {
"n100": {
"chance_corrected": 0.358958,
"n_labels": 1100,
"retention": 0.743881
},
"n20": {
"chance_corrected": 0.284312,
"n_labels": 220,
"retention": 0.589189
},
"n5": {
"chance_corrected": 0.201352,
"n_labels": 55,
"retention": 0.417269
}
},
"score": 58.34461
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 27.619048,
"worst_class_recall": 0.341991
},
"uncertainty": {
"full_accuracy": 0.572561,
"risk_coverage_auc": 0.746626,
"score": 72.128816,
"selective_acc_at_50": 0.738786,
"selective_acc_at_80": 0.638346
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 13932,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "CT",
"n_classes": 11,
"n_test": 8827,
"runtime_s": 12,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "organsmnist",
"task_name": "Abdominal CT organs — sagittal (S)",
"timestamp": "2026-08-01T19:30:25+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 44.228081
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.372841,
"auc": 0.770577,
"balanced_accuracy": 0.340428,
"ci95": [
0.330496,
0.350416
],
"n_test": 7180,
"score": 25.798203
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.244602,
"mean_agreement": 0.562667,
"mean_kappa": 0.499798,
"mean_retention": 0.751016,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.5208,
0.3283,
0.2
],
"mean_kappa": 0.257196,
"mean_retention": 0.327105,
"retention_by_severity": [
0.5944,
0.2877,
0.0992
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.5325,
0.5025,
0.4475
],
"mean_kappa": 0.420577,
"mean_retention": 0.916811,
"retention_by_severity": [
0.9571,
0.8774,
0.9159
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8442,
0.7867,
0.7325
],
"mean_kappa": 0.758426,
"mean_retention": 0.950389,
"retention_by_severity": [
0.961,
0.9716,
0.9186
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.4317,
0.3575,
0.3008
],
"mean_kappa": 0.270113,
"mean_retention": 0.560773,
"retention_by_severity": [
0.7731,
0.5259,
0.3834
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.8758,
0.8383,
0.7408
],
"mean_kappa": 0.792676,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
}
},
"score": 75.101552,
"worst_retention": 0.099156
},
"calibration": {
"accuracy": 0.372841,
"brier": 0.727627,
"ece": 0.098206,
"mean_confidence": 0.471047,
"overconfidence": 0.098206,
"score": 50.897211
},
"corruption": {
"clean_reference_cc": 0.244602,
"mean_agreement": 0.448532,
"mean_kappa": 0.369829,
"mean_retention": 0.56366,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9492,
0.8183,
0.6333
],
"mean_kappa": 0.772115,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8417,
0.83,
0.78
],
"mean_kappa": 0.791493,
"mean_retention": 0.999582,
"retention_by_severity": [
0.9987,
1,
1
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.5392,
0.3783,
0.2583
],
"mean_kappa": 0.29688,
"mean_retention": 0.607893,
"retention_by_severity": [
0.956,
0.5289,
0.3388
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.3517,
0.2158,
0.1475
],
"mean_kappa": 0.129664,
"mean_retention": 0.179173,
"retention_by_severity": [
0.3521,
0.1742,
0.0112
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.4658,
0.14,
0.1225
],
"mean_kappa": 0.138798,
"mean_retention": 0.286455,
"retention_by_severity": [
0.7984,
0,
0.061
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.6717,
0.4742,
0.2092
],
"mean_kappa": 0.37835,
"mean_retention": 0.716195,
"retention_by_severity": [
1,
0.9211,
0.2275
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.2733,
0.1717,
0.1475
],
"mean_kappa": 0.081505,
"mean_retention": 0.156322,
"retention_by_severity": [
0.2258,
0.165,
0.0781
],
"scored": true
}
},
"score": 56.365997,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.257982,
"mean_retention": 0.447402,
"per_budget": {
"n100": {
"chance_corrected": 0.161912,
"n_labels": 900,
"retention": 0.627609
},
"n20": {
"chance_corrected": 0.114276,
"n_labels": 180,
"retention": 0.442962
},
"n5": {
"chance_corrected": 0.070077,
"n_labels": 45,
"retention": 0.271635
}
},
"score": 44.740221
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 1.315789,
"worst_class_recall": 0.122807
},
"uncertainty": {
"full_accuracy": 0.372841,
"risk_coverage_auc": 0.591369,
"score": 54.028993,
"selective_acc_at_50": 0.537047,
"selective_acc_at_80": 0.420787
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 89996,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "Histopathology",
"n_classes": 9,
"n_test": 7180,
"runtime_s": 12.6,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "pathmnist",
"task_name": "Colorectal pathology (9-class tissue)",
"timestamp": "2026-08-01T19:41:48+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 57.914465
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.833333,
"auc": 0.94908,
"balanced_accuracy": 0.783761,
"ci95": [
0.75035,
0.819748
],
"n_test": 624,
"score": 56.752137
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.567521,
"mean_agreement": 0.89391,
"mean_kappa": 0.632604,
"mean_retention": 0.686546,
"n_cases": 624,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.9247,
0.875,
0.8157
],
"mean_kappa": 0.602683,
"mean_retention": 0.78012,
"retention_by_severity": [
0.9774,
0.9548,
0.4081
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.8606,
0.7917,
0.7708
],
"mean_kappa": 0.222292,
"mean_retention": 0.195783,
"retention_by_severity": [
0.4593,
0.1205,
0.0075
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.9744,
0.9583,
0.9279
],
"mean_kappa": 0.877847,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.8958,
0.8782,
0.8462
],
"mean_kappa": 0.568414,
"mean_retention": 0.557731,
"retention_by_severity": [
0.6762,
0.5873,
0.4096
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.984,
0.9663,
0.9391
],
"mean_kappa": 0.891784,
"mean_retention": 0.899096,
"retention_by_severity": [
0.9699,
0.9322,
0.7952
],
"scored": true
}
},
"score": 68.654618,
"worst_retention": 0.00753
},
"calibration": {
"accuracy": 0.833333,
"brier": 0.271176,
"ece": 0.111088,
"mean_confidence": 0.944058,
"overconfidence": 0.110724,
"score": 44.456178
},
"corruption": {
"clean_reference_cc": 0.567521,
"mean_agreement": 0.872558,
"mean_kappa": 0.524281,
"mean_retention": 0.553356,
"n_cases": 624,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9968,
0.9599,
0.8558
],
"mean_kappa": 0.784871,
"mean_retention": 0.744478,
"retention_by_severity": [
0.9849,
0.8268,
0.4217
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.9712,
0.9792,
0.9599
],
"mean_kappa": 0.916948,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.9247,
0.8381,
0.7724
],
"mean_kappa": 0.398003,
"mean_retention": 0.381526,
"retention_by_severity": [
0.7877,
0.3419,
0.0151
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.891,
0.8558,
0.7724
],
"mean_kappa": 0.425546,
"mean_retention": 0.524096,
"retention_by_severity": [
0.9337,
0.5843,
0.0542
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.9054,
0.8013,
0.7724
],
"mean_kappa": 0.315918,
"mean_retention": 0.35492,
"retention_by_severity": [
0.884,
0.1657,
0.0151
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.9471,
0.8846,
0.7724
],
"mean_kappa": 0.491661,
"mean_retention": 0.498494,
"retention_by_severity": [
0.9352,
0.5452,
0.0151
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.8654,
0.8253,
0.7724
],
"mean_kappa": 0.337023,
"mean_retention": 0.36998,
"retention_by_severity": [
0.747,
0.3117,
0.0512
],
"scored": true
}
},
"score": 55.335628,
"worst_retention": 0.01506
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.567521,
"mean_retention": 0.961345,
"per_budget": {
"n100": {
"chance_corrected": 0.655556,
"n_labels": 200,
"retention": 1
},
"n20": {
"chance_corrected": 0.634188,
"n_labels": 40,
"retention": 1
},
"n5": {
"chance_corrected": 0.501709,
"n_labels": 10,
"retention": 0.884036
}
},
"score": 96.134538
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 17.094017,
"worst_class_recall": 0.58547
},
"uncertainty": {
"full_accuracy": 0.833333,
"risk_coverage_auc": 0.946066,
"score": 89.213252,
"selective_acc_at_50": 0.964744,
"selective_acc_at_80": 0.901804
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 4708,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Chest X-ray",
"n_classes": 2,
"n_test": 624,
"runtime_s": 3.4,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "pneumoniamnist",
"task_name": "Paediatric chest X-ray (pneumonia)",
"timestamp": "2026-08-01T19:16:17+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 37.757002
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.41,
"auc": 0.590131,
"balanced_accuracy": 0.265322,
"ci95": [
0.230826,
0.310116
],
"n_test": 400,
"score": 8.165292
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.081653,
"mean_agreement": 0.495167,
"mean_kappa": 0.312111,
"mean_retention": 0.734893,
"n_cases": 400,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.4825,
0.2725,
0.13
],
"mean_kappa": 0.144963,
"mean_retention": 0.448138,
"retention_by_severity": [
1,
0.0688,
0.2756
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.435,
0.46,
0.3275
],
"mean_kappa": 0.163408,
"mean_retention": 0.316831,
"retention_by_severity": [
0.472,
0.411,
0.0674
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.755,
0.64,
0.515
],
"mean_kappa": 0.499839,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.5625,
0.5475,
0.52
],
"mean_kappa": 0.302054,
"mean_retention": 0.988433,
"retention_by_severity": [
1,
1,
0.9653
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.75,
0.6025,
0.4275
],
"mean_kappa": 0.450289,
"mean_retention": 0.921063,
"retention_by_severity": [
1,
1,
0.7632
],
"scored": true
}
},
"score": 73.489284,
"worst_retention": 0.067401
},
"calibration": {
"accuracy": 0.41,
"brier": 0.880558,
"ece": 0.324453,
"mean_confidence": 0.7295,
"overconfidence": 0.3195,
"score": 0
},
"corruption": {
"clean_reference_cc": 0.081653,
"mean_agreement": 0.487976,
"mean_kappa": 0.344155,
"mean_retention": 0.681512,
"n_cases": 400,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9675,
0.9225,
0.8875
],
"mean_kappa": 0.891673,
"mean_retention": 0.966083,
"retention_by_severity": [
0.9414,
1,
0.9569
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.845,
0.8575,
0.7475
],
"mean_kappa": 0.734471,
"mean_retention": 0.933493,
"retention_by_severity": [
0.8865,
0.9139,
1
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.6225,
0.5375,
0.4025
],
"mean_kappa": 0.28112,
"mean_retention": 0.90198,
"retention_by_severity": [
1,
0.7059,
1
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.195,
0.1175,
0.1075
],
"mean_kappa": 0.024896,
"mean_retention": 0.184012,
"retention_by_severity": [
0.3607,
0.0436,
0.1478
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.525,
0.4475,
0.2725
],
"mean_kappa": 0.171679,
"mean_retention": 0.60061,
"retention_by_severity": [
0.6806,
0.9028,
0.2184
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.63,
0.4075,
0.3475
],
"mean_kappa": 0.275246,
"mean_retention": 0.802828,
"retention_by_severity": [
1,
0.8915,
0.517
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.2225,
0.0975,
0.0875
],
"mean_kappa": 0.029997,
"mean_retention": 0.381577,
"retention_by_severity": [
0.5928,
0.5519,
0
],
"scored": true
}
},
"score": 68.151168,
"worst_retention": 0
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.081653,
"mean_retention": 1,
"per_budget": {
"n100": {
"chance_corrected": 0.116761,
"n_labels": 466,
"retention": 1
},
"n20": {
"chance_corrected": 0.091621,
"n_labels": 100,
"retention": 1
},
"n5": {
"chance_corrected": 0.100219,
"n_labels": 25,
"retention": 1
}
},
"score": 100
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0.05
},
"uncertainty": {
"full_accuracy": 0.41,
"risk_coverage_auc": 0.483684,
"score": 35.46052,
"selective_acc_at_50": 0.48,
"selective_acc_at_80": 0.425
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 1080,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": false
},
"modality": "Fundus",
"n_classes": 5,
"n_test": 400,
"runtime_s": 2.5,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "retinamnist",
"task_name": "Fundus photography (DR grade)",
"timestamp": "2026-08-01T19:13:11+00:00"
},
{
"code_fingerprint": "b444c02112cc",
"composite": {
"coverage": 0.89,
"dimensions_scored": [
"accuracy",
"acquisition",
"calibration",
"corruption",
"limited_data",
"subgroup",
"uncertainty"
],
"index": 44.356107
},
"device": "mps",
"dimensions": {
"accuracy": {
"accuracy": 0.464784,
"auc": 0.733885,
"balanced_accuracy": 0.275777,
"ci95": [
0.272509,
0.278636
],
"n_test": 47280,
"score": 17.231657
},
"acquisition": {
"basis": "simulated re-acquisition (gamma, window/level, resolution, FOV, detector noise)",
"clean_reference_cc": 0.182919,
"mean_agreement": 0.549111,
"mean_kappa": 0.374257,
"mean_retention": 0.685347,
"n_cases": 1200,
"per_perturbation": {
"detector_noise": {
"agreement_by_severity": [
0.5408,
0.4767,
0.4667
],
"mean_kappa": 0.074248,
"mean_retention": 0.106657,
"retention_by_severity": [
0.2753,
0.0447,
0
],
"scored": true
},
"field_of_view": {
"agreement_by_severity": [
0.7392,
0.6917,
0.5792
],
"mean_kappa": 0.537352,
"mean_retention": 0.883774,
"retention_by_severity": [
0.9554,
0.9076,
0.7882
],
"scored": true
},
"gamma_shift": {
"agreement_by_severity": [
0.8058,
0.6133,
0.4058
],
"mean_kappa": 0.505979,
"mean_retention": 0.96098,
"retention_by_severity": [
1,
1,
0.8829
],
"scored": true
},
"resolution": {
"agreement_by_severity": [
0.6783,
0.655,
0.6133
],
"mean_kappa": 0.530098,
"mean_retention": 0.937791,
"retention_by_severity": [
0.9868,
0.9405,
0.886
],
"scored": true
},
"window_level": {
"agreement_by_severity": [
0.5583,
0.2608,
0.1517
],
"mean_kappa": 0.22361,
"mean_retention": 0.537536,
"retention_by_severity": [
0.8864,
0.5215,
0.2048
],
"scored": true
}
},
"score": 68.534744,
"worst_retention": 0
},
"calibration": {
"accuracy": 0.464784,
"brier": 0.692101,
"ece": 0.048602,
"mean_confidence": 0.513386,
"overconfidence": 0.048602,
"score": 75.699094
},
"corruption": {
"clean_reference_cc": 0.182919,
"mean_agreement": 0.594286,
"mean_kappa": 0.403546,
"mean_retention": 0.531508,
"n_cases": 1200,
"per_perturbation": {
"brightness": {
"agreement_by_severity": [
0.9983,
0.9975,
0.9975
],
"mean_kappa": 0.996767,
"mean_retention": 1,
"retention_by_severity": [
1,
1,
1
],
"scored": true
},
"contrast": {
"agreement_by_severity": [
0.8683,
0.8025,
0.665
],
"mean_kappa": 0.686341,
"mean_retention": 0.913602,
"retention_by_severity": [
0.986,
0.9497,
0.8051
],
"scored": true
},
"defocus_blur": {
"agreement_by_severity": [
0.7333,
0.5683,
0.4283
],
"mean_kappa": 0.46451,
"mean_retention": 0.855896,
"retention_by_severity": [
1,
0.9069,
0.6608
],
"scored": true
},
"gaussian_noise": {
"agreement_by_severity": [
0.4817,
0.4692,
0.4683
],
"mean_kappa": 0.018005,
"mean_retention": 0.017419,
"retention_by_severity": [
0.0428,
0.0086,
0.0009
],
"scored": true
},
"pixelate": {
"agreement_by_severity": [
0.7442,
0.535,
0.2958
],
"mean_kappa": 0.361262,
"mean_retention": 0.549337,
"retention_by_severity": [
0.9201,
0.5731,
0.1548
],
"scored": true
},
"quantise": {
"agreement_by_severity": [
0.4108,
0.2433,
0.1883
],
"mean_kappa": 0.096052,
"mean_retention": 0.143116,
"retention_by_severity": [
0.3843,
0.034,
0.011
],
"scored": true
},
"shot_noise": {
"agreement_by_severity": [
0.5433,
0.5217,
0.5192
],
"mean_kappa": 0.201883,
"mean_retention": 0.241183,
"retention_by_severity": [
0.2811,
0.1082,
0.3343
],
"scored": true
}
},
"score": 53.15076,
"worst_retention": 0.000858
},
"cost": null,
"limited_data": {
"full_reference_cc": 0.172317,
"mean_retention": 0.50539,
"per_budget": {
"n100": {
"chance_corrected": 0.104855,
"n_labels": 800,
"retention": 0.608501
},
"n20": {
"chance_corrected": 0.080591,
"n_labels": 160,
"retention": 0.467691
},
"n5": {
"chance_corrected": 0.075815,
"n_labels": 40,
"retention": 0.439978
}
},
"score": 50.538977
},
"spec_sensitivity": null,
"subgroup": {
"attributes": {},
"basis": "worst diagnostic class (no demographic metadata in source)",
"has_real_metadata": false,
"score": 0,
"worst_class_recall": 0.00403
},
"uncertainty": {
"full_accuracy": 0.464784,
"risk_coverage_auc": 0.599243,
"score": 54.199143,
"selective_acc_at_50": 0.593993,
"selective_acc_at_80": 0.515625
}
},
"domain": "medicine",
"kind": "image",
"limitations": {
"has_real_subgroup_metadata": false,
"n_train_available": 165466,
"notes": null,
"subgroup_basis": "worst diagnostic class (no demographic metadata in source)",
"train_capped": true
},
"modality": "Microscopy",
"n_classes": 8,
"n_test": 47280,
"runtime_s": 26.4,
"seed": 20260727,
"spec_version": "MedEval-1 v0.1",
"split_origin": "official released split",
"task": "tissuemnist",
"task_name": "Kidney cortex cells (8-class)",
"timestamp": "2026-08-01T19:38:00+00:00"
}
],
"heldout": {
"entries": [
{
"direction": "no gap beyond sampling error",
"equating_offset": 0,
"gap": -1.767815,
"gap_se": 2.617803,
"generation": "gen1",
"merkle_root": "61c508cbafed177bb1553f05aa2cc80d4157adc1bc1ff611d88581aab9ca61c5",
"model": "hog_logreg",
"score_private_equated": 52.592004,
"score_private_raw": 52.592004,
"score_public": 50.824189,
"set_hash": "641ab45baeaf445a1057c060eb9b35205bd55fcde4e629b2b88cfa38fba30047",
"submitter": "Submitter B",
"verdict": "clear",
"version_tag": "v1",
"z": -0.675305,
"z_crit": 3
},
{
"direction": "no gap beyond sampling error",
"equating_offset": 13.753223,
"gap": -6.263166,
"gap_se": 4.642961,
"generation": "gen2",
"merkle_root": "4adc9bbb0bc17564ad2effaa4d6fc902a0f319bb3e26735ac67d1b988cbe6e2d",
"model": "hog_logreg",
"score_private_equated": 59.599916,
"score_private_raw": 45.846693,
"score_public": 53.33675,
"set_hash": "ad30d214daa285c5123df991dd0eae58038f99f3490b840299cb620279ff2e73",
"submitter": "Submitter B",
"verdict": "clear",
"version_tag": "v2",
"z": -1.348959,
"z_crit": 3
}
],
"labelling": "PUBLIC-DATA DEMONSTRATION OF THE PRIVATE PROTOCOL"
},
"sources": [
{
"path": "results/cross_site/organ_ct__hog_logreg.json",
"sha256": "c49a4fcc7dccbbd685afd6f064c5760cc7ea427c7598537fea7be72e00ab0825"
},
{
"path": "results/heldout/audit_report.json",
"sha256": "df460b6d91a6512dfa1e7dc650637fce2b4dd3a1ec32e03446bf6e11f9c597d2"
},
{
"path": "results/heldout/cycle.json",
"sha256": "e012b8978457b51b5c31d18d14d2cc92f100731897523306acae19934f943502"
},
{
"path": "results/heldout/manifests/gen1.seal.json",
"sha256": "91a7adfa29870a0719125cdd48f6dcb149f1f6485ac1cfa104c6e00057ad9b81"
},
{
"path": "results/heldout/manifests/gen2.seal.json",
"sha256": "f61470918a09cf39e074ef14fb7ade4ac3a13d36d9385f18cf9b92e4b5329547"
},
{
"path": "results/heldout/manifests/public.seal.json",
"sha256": "1a4fd930d2e781ac7163483990f15d6694e28e507d1b3ca009ff09395e47497c"
},
{
"path": "results/records/bloodmnist__hog_logreg.json",
"sha256": "d2febf8b19a5988009a8aff67ab2bdde80de417107b392684b462e165d08c7e0"
},
{
"path": "results/records/breastmnist__hog_logreg.json",
"sha256": "d75604762557b0864ca0a403c2996cf3aa0728b2e787008abc4bc8b4da219b9a"
},
{
"path": "results/records/dermamnist__hog_logreg.json",
"sha256": "ccf178af0ba9e8959491356308727a750a072c95e35ab031df057fa06d2d7b21"
},
{
"path": "results/records/octmnist__hog_logreg.json",
"sha256": "c5155cc76e8b49d696181591400e7a3c6b0c7cc911b127a466120759321db0dd"
},
{
"path": "results/records/organamnist__hog_logreg.json",
"sha256": "63cc89e81e87e9886976a9e7057fc8954074065c329c23b96bcc6999b49d2b09"
},
{
"path": "results/records/organcmnist__hog_logreg.json",
"sha256": "9008f98f8a75ca6168b557ad1e99faa67b1d0cffaf9d6c09da7e2b40ad4691f5"
},
{
"path": "results/records/organsmnist__hog_logreg.json",
"sha256": "1d531b0ba53255fc44411bf6b4447e30dc608e05f708ff6a12d9bc6831eb3aff"
},
{
"path": "results/records/pathmnist__hog_logreg.json",
"sha256": "e1c02f91a99057416458ae6eb2e93b312cfc707ffefb25e89e86856f6e054fb7"
},
{
"path": "results/records/pneumoniamnist__hog_logreg.json",
"sha256": "d1aa310d240f5f0d78abe0ebaf9a3b8c4d1348af60811b9042df8f693f077bf4"
},
{
"path": "results/records/retinamnist__hog_logreg.json",
"sha256": "6d3fa677d879d539e75751fafec225977c5b801431bfece78876547c544a4339"
},
{
"path": "results/records/tissuemnist__hog_logreg.json",
"sha256": "4e6d90310286e491c2ab3fdc28e036f08175c1c4fa5ed8835b209a57f405afa2"
}
]
},
"declared": null,
"declared_note": "No vendor declaration. This is an open reference baseline implemented by NakedSignal, not a commercial product: there is no vendor to assert an intended use or to sign an attestation, so the declared block is null rather than filled in on the vendor's behalf.",
"expiry_basis": "the sealed held-out generation this evidence cycle is anchored to (gen2, set hash ad30d214daa285c5...) is published to expire 2026-09-20; per spec, a passport lapses when its generation retires or its spec version is superseded, whichever comes first",
"expiry_utc": "2026-09-20T00:00:00Z",
"issued_utc": "2026-08-03T16:03:42Z",
"issuer": {
"algo": "Ed25519",
"key_id": "ns-passport-2026-07",
"name": "NakedSignal",
"public_key_b64": "0PBCC6mjzcppR5QdPcK7vP/AamB6MP8l1T4ypMYbsMw="
},
"observed": null,
"observed_note": "Not available: this requires continuous monitoring. Production history and drift are observed fields, and nothing is deployed, so there is nothing to observe. The object is present and null on purpose. The shape is the roadmap, not a claim.",
"passport_version": "0.1",
"signature": "VG1P+q2RihosuPA3a9EmzX5pGH8u9TGHKu4Z6VDF2aPduFbJaqKXdq8m0SQMBtrOHS3xdMmDRZ6aG/2/+ba/AA==",
"spec_version": "MedEval-1 v0.1",
"subject": {
"behaviour_version": "1.0.0",
"code_fingerprint": "b444c02112cc",
"code_fingerprints": [
"b444c02112cc"
],
"family": "classical",
"model_id": "hog_logreg",
"model_name": "HOG + logistic regression",
"params": "9 orientations, 7x7 cells, 2x2 blocks"
}
}Canonicalisation: RFC 8785 (JCS)-compatible: UTF-8, object keys sorted by code point, no insignificant whitespace, ECMAScript number formatting. every float is rounded once at emit to 6 decimal places (Python round(), banker's rounding); integers are left exact. Signed over the whole document with the `signature` member removed, canonicalised as above and encoded UTF-8. The public key for ns-passport-2026-07 is 0PBCC6mjzcppR5QdPcK7vP/AamB6MP8l1T4ypMYbsMw= — see Governance.