Shift lab

PROTOTYPE

A model is only as good as the distribution it was measured on, and nothing about its output tells you which distribution you are in. Train a real model, deploy it into machines it has never seen, watch it fail, generate the variation it is missing, and deploy it again.

What this is notThe acquisition shift here is simulated on public single-source data. It is a stand-in for the real thing -- one patient imaged on two physical machines -- which needs paired multi-site data this project does not have. The one genuine acquisition shift in public data is the organ{A,C,S} cross-plane transfer test, published separately. The model, its weights and every accuracy below are real and were produced by the command in the right-hand column; what is simulated is the second machine.

84.1%

accuracy on the reference condition — the number that would go on a slide

62.7%

worst of 15 deployment realities, same model, same weights

26.0pts

spread between the best and worst reality

785

parameters in the model — small enough to look at

624

held-out X-rays scored in every reality

A real model, small enough to look at

◼ normal◼ pneumonia

This is the entire model: 785 numbers, one per pixel plus an intercept, trained on 4,708 real paediatric chest x-ray (pneumonia) images. The square on the left is those numbers drawn as an image — blue where a bright pixel is evidence for pneumonia, red where it is evidence for normal. There is nothing else in it.

A convolutional network would score higher and show you nothing. The argument on this page is about the training distribution, not about model capacity, and a linear model makes it legible — and makes the browser’s arithmetic exact.

84.1%

accuracy, reference condition

60.3%

recall on its worse class (normal)

logistic regression

L2, C=0.5, max_iter=6000

Deploy it somewhere else

Same model, same weights, same patients. The only thing that changes is the machine the image came off. Pick a difference and a degree of it — every one of these is a thing that genuinely varies between two real radiology departments.

What differs

How different

Which model

A tighter collimation or a differently positioned patient.

One held-out X-ray

as trained on

as it arrives

What the model says about this one

pneumonia

predicted · truth is pneumonia

100.0%

confidence in pneumonia

10.7825

w·x + b, computed live

Matches the value shift_lab.py recorded for this image in this reality (10.782465). The page is running the model, not replaying an answer.

And across all 624 held-out X-rays

62.7%

accuracy in this reality

74.5%

of its reference accuracy retained

0.4%

recall on its worse class

62.5%

what you get by always answering the commoner class

At this setting the model is no better than answering pneumonia every time. It has not degraded — it has stopped working, and nothing about its output says so.

The published number is one point in a distribution nobody publishes

Every dot below is the same model, on the same held-out patients, in one of the 15 realities. The model has one published accuracy and 26.0 points of unpublished range. A buyer is told the marked value and deployed into an unmarked one, and there is nothing in the model, the paper or the marketing that says which.

always answer the commoner class

published

60%70%80%90%

Each dot is one of the 15 realities, rows by degree of difference. Click one to load it into step 2. Currently showing the baseline model.

It is not a small effect

Published work on cross-population and cross-scanner shift puts degradation at 10–25% on unseen populations, and notes that intra-centre variation — two machines in one hospital — can be as large as between-centre variation.

We measure it as a dimension

This is dimension 2 of the standard, not a side experiment. On the one genuine acquisition shift in public data — the same abdominal volumes reconstructed in three planes — the suite’s best model goes from 92.3% within-plane to 40.2% across planes. See the retention scatter →

Which is why evidence expires

A number measured on one distribution is a claim about that distribution and nothing else. That is the whole reason an Evidence Passport records the population it was measured on, and the reason it has an expiry. See the passport →

Measure the variation, then generate it

The measurement in step 2 says exactly where this model is fragile: it barely notices the three intensity differences and it falls apart under the two geometric ones. So the generator is aimed at those two, and at only 2 of the 3 degrees — leaving 11 of the 15 realities never seen in training, so what follows measures generalisation rather than memorisation.

The generated images are added to the real ones, never substituted for them: 4,708 real X-rays become 23,540 training rows. That ordering is not a detail — replacing real data with generated data is how error grows without bound, while accumulating provably does not.

This belongs in training, and the alternatives are worse for a reason worth stating: fine-tuning on the target site needs labelled data from a site you have not deployed to yet, and test-time adaptation needs a batch of that site’s images at inference. Train-time augmentation is the only one of the three that buys robustness to a machine you have never seen.

The generated training data

Real image first, then the 4 generated variants of it that go into training. Same label throughout — the transform changes how the scan was made, not what is in it.

Block	Degree	Rows	Origin
The real X-rays	—	4,708	measured
Reconstruction matrix	mild	4,708	generated
Reconstruction matrix	moderate	4,708	generated
Field of view / positioning	mild	4,708	generated
Field of view / positioning	moderate	4,708	generated

Retrain, redeploy, and report the part that still fails

Baseline weights

After generated variation

A different model, visibly. The baseline concentrated its opinion in a few bright regions; the retrained one spreads it, because a feature that survives four other machines is the only kind of feature that earns weight. Both squares are drawn on the same scale, so they are directly comparable.

15/15

realities improved

11/11

of the realities never generated, improved

60.3 → 82.1%

recall on the worse class, reference condition

Deployment reality	Degree	Generated?	Baseline	Retrained	Δ	Worse class, retrained
Reference condition	as published	—	84.1	90.5	+6.4	82.1
Detector response curve	mild	never seen	86.7	91.3	+4.6	87.2
Detector response curve	moderate	never seen	87.3	90.1	+2.7	88.5
Detector response curve	pronounced	never seen	88.6	89.6	+1.0	88.5
Windowing preset	mild	never seen	85.9	91.7	+5.8	86.8
Windowing preset	moderate	never seen	87.0	89.9	+2.9	87.2
Windowing preset	pronounced	never seen	86.4	88.0	+1.6	85.0
Reconstruction matrix	mild	generated	82.5	90.4	+7.9	80.8
Reconstruction matrix	moderate	generated	80.9	88.1	+7.2	73.9
Reconstruction matrix	pronounced	never seen	81.6	88.8	+7.2	74.8
Field of view / positioning	mild	generated	75.2	86.5	+11.4	67.1
Field of view / positioning	moderate	generated	66.0	77.9	+11.9	41.5
Field of view / positioning	pronounced	never seen	62.7	66.7	+4.0	11.1
Detector noise floor	mild	never seen	83.8	90.4	+6.6	82.1
Detector noise floor	moderate	never seen	84.1	90.2	+6.1	82.1
Detector noise floor	pronounced	never seen	82.4	89.3	+6.9	77.4

What this did not fix

The worst reality — the most pronounced field-of-view difference — went from 62.7 to 66.7. That is a real improvement and it leaves the model 4.2 points above the 62.5% you get for free by answering pneumonia every time. Nobody should deploy that. Generated variation moved every number on this page and it did not buy a model that survives everything — and the reference number went up too, from 84.1 to 90.5, which means the honest summary is “better everywhere, still unusable in one place” rather than a trade-off. Reporting the second half of that sentence is the difference between a measurement and a marketing claim.

Where this sits

The phenomenon on this page is dimension 2 of the standard — acquisition robustness — reduced to one model you can hold in your head. The published, aggregate version is synthetic variation, which measures 12 transforms at 3 severities across 7 models and 11 tasks; the one genuine acquisition shift in public data, the same abdominal volumes reconstructed in three planes, is on the leaderboard, and the same metric applied to wearable sensors is on wearable signal. If a model’s evidence is going to expire when its population changes, that fact belongs in the document that travels with it — the Evidence Passport — and in the gate that reads it, the Trust Runtime.

No model in the published suite has been improved by generated variation. The retraining on this page is a demonstration on one small model on one public dataset, and it is not a result about MedEval-1’s leaderboard.

Provenance

dataset: pneumoniamnist
model: 785 parameters
train: 4,708 → 23,540
test: 624 held out
realities: 16
seed: 20260730
fingerprint: 21a1ed929973

Re-derive it

cd plan_d
python src/shift_lab.py
python src/shift_lab.py --check

--check re-runs the training and asserts the emitted JSON is byte-identical. Two separate processes agree, which is why the transforms are seeded explicitly here rather than through corruptions.apply() — see docs/ANALYSIS-PLAN.md.