Case studies

The headline number a model reports is measured in the setting it was trained in. Change the acquisition and keep everything else identical — same patients, same organs, same labels — and most of the skill is gone.

That is ResNet-18 (ImageNet) fine-tuned on the axial plane of abdominal CT and then on the sagittal plane of the same volumes — same patients, same organs, same labels. Across the whole zoo mean retention runs from 97.4% for the best down to 36.1% for the worst. Of 37 model-and-setting pairs, 2 sit above 0.85 within-setting accuracy and below 0.60 mean retention at the same time.

That gap is the company’s whole argument in one number: the score on the paper is not the score that decides whether the model can be acted on somewhere else. See the scatter and the full table →

A held-out track only means something if it can catch a model that has seen the answers. This cycle was run end to end with a detector whose threshold was written into a tamper-evident ledger before the first case was drawn.

One submission had been trained on the sealed set. It scored 100.0 on the private track against 76.8 on the public control arm and was flagged at z = -15.11, where the 5 honest submissions in the same scan ran -0.68 to +1.63. The threshold it breached — |z| ≥ 3.0 — sits at ledger position 1, earlier in the chain than every score entry.

Rotating the set to a fresh generation removed 95.3% of its advantage: 25.0 points over the honest cohort on the leaked generation, 1.2 on the next one. Memorisation has a half-life if the set has one.

48 of 48 published scores were independently re-derived by an auditor who never saw a case: the audit bundles carry salted case identifiers and no image, feature or patient attribute. Failures and flags are published, including the operator’s own.

NakedSignal did not start in medicine. It started in labour-market signal, and the question it had to answer there is the same question it now asks about medical models.

The company built continuous-signal infrastructure over labour-market data. The raw material was labour-market exhaust — job postings and job changes — and it carried two different kinds of signal at once. A job change is a corporate leading indicator, and it is also a personal life-event trigger: a relocation, an insurance need, a banking need. The revenue came from recruitment. The harder use cases were the ones that set the engineering bar.

It was never a jobs company. It was an alternative-data and signal-extraction business that started with labour data — source data nobody else was reading, harmonise it, and turn it into something someone could act on.

What made those harder use cases hard was never the feed. It was whether a model’s output was reliable enough to act on when the action carried real cost — the underwriting question. That is the question this company has been working on since before it touched medicine, and it is why the claim to be domain-general describes where the work came from rather than where it hopes to go.

That question does not change between domains. Medicine is where it is hardest, most regulated and least well answered, which is why medicine is where the first full standard got built. MedEval-1 is one instrument. The question underneath it is older than the instrument.