Process and surface inspection
LIVESemiconductor fabrication pass/fail and steel plate surface faults. The fab corpus is split chronologically rather than at random, because a process-monitoring model is always asked to predict forward in time — and it has 590 sensor channels over 1,567 lots, so it is expected to be hard and the intervals say so. No separate specification document has been published for this benchmark — the MedEval-1 spec is the document, and this page is that instrument run on another corpus. Everything below is read from the same payload.json the medical leaderboard renders from.
The index is the weighted mean of the dimensions that were scored, on the weights published in the spec. Dimensions this corpus cannot support are excluded from it rather than counted as zero, which is why an index here is not comparable to a medical one model-for-model — it is a mean over a different set of questions, and the table below says which.
Every model, on all 9 dimensions
Ranked by index. An em-dash is a dimension the harness left null.| # | Model | Index | 1 Accuracy | 2 Cross-site | 3 Calibration | 4 Limited data | 5 Subgroup | 6 Corruption | 7 Uncertainty | Specification sensitivity | Cost, latency & footprint |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Gradient boosting200 stages, depth 3, lr 0.05 | 71.1 | 75.4 | — | 74.5 | 80.3 | 48.6 | 65.0 | 90.9 | — | — |
| 2 | Random forest400 trees | 70.3 | 76.0 | — | 54.8 | 89.7 | 48.2 | 72.4 | 91.9 | — | — |
| 3 | MLP (128, 64)2 hidden layers, adam, early stopping when n>=200 | 68.2 | 69.1 | — | 74.5 | 91.4 | 30.6 | 68.2 | 85.9 | — | — |
| 4 | Logistic regressionstandardised, L2, C=1 | 67.1 | 70.4 | — | 66.0 | 92.6 | 34.2 | 66.1 | 82.4 | — | — |
| 5 | Gradient boosting200 stages, depth 3, lr 0.05 | 23.5 | 0.0 | — | 71.9 | 0.0 | 0.0 | — | 86.2 | — | — |
| 6 | Random forest400 trees | 22.4 | 0.0 | — | 66.2 | 0.0 | 0.0 | — | 87.1 | — | — |
| 7 | Logistic regressionstandardised, L2, C=1 | 21.2 | 0.0 | — | 61.8 | 0.0 | 0.0 | — | 85.2 | — | — |
| 8 | MLP (128, 64)2 hidden layers, adam, early stopping when n>=200 | 12.8 | 0.0 | — | 24.6 | 0.0 | 0.0 | — | 84.0 | — | — |
- 2. Cross-site robustness — No re-acquisition of the same subjects exists in this corpus, so there is nothing to measure. Widening the definition to fill the cell would be a spec change, not a demo.
- . — Not computed on any record in this benchmark, and not approximated.
- . — Not computed on any record in this benchmark, and not approximated.
The measurements the scores came from
Raw metrics, not the 0–100 rescaling.| Model | Balanced accuracy (95% CI) | AUC | ECE | Risk–coverage AUC | Mean label-budget retention |
|---|---|---|---|---|---|
| Gradient boosting | 0.789[0.740–0.828] | 0.951 | 0.051 | 0.922 | 0.803 |
| Random forest | 0.794[0.746–0.834] | 0.961 | 0.090 | 0.930 | 0.897 |
| MLP (128, 64) | 0.736[0.683–0.789] | 0.934 | 0.051 | 0.879 | 0.914 |
| Logistic regression | 0.747[0.700–0.793] | 0.933 | 0.068 | 0.849 | 0.926 |
| Gradient boosting | 0.481[0.471–0.490] | 0.467 | 0.056 | 0.931 | 0.000 |
| Random forest | 0.499[0.496–0.500] | 0.448 | 0.068 | 0.935 | 0.000 |
| Logistic regression | 0.493[0.487–0.499] | 0.609 | 0.076 | 0.926 | 0.000 |
| MLP (128, 64) | 0.493[0.487–0.497] | 0.382 | 0.151 | 0.920 | 0.000 |
Balanced accuracy carries a bootstrap 95% interval; read it before reading the rank. ECE is expected calibration error, so lower is better and the 0–100 dimension score inverts it. Every value here is on the record; hover any figure for the unrounded number.
Who the best model fails
LIVEDimension five, on Gradient boosting — the top-ranked model above. Basis, in the evaluator’s own words: worst demographic subgroup (real metadata). This is the strong reading — the corpus ships the attributes, so the harness did not have to fall back to the worst-class proxy every medical imaging task uses.
| A300worst | 0.559 | |
| A400 | 0.744 |
Per-group recall, and the gap between the best and worst group. These numbers measure who the model fails on this corpus. They are not an audit of any deployed system, and the subgroup attributes are model inputs here, as they are in the standard formulation of the task.
The corpus
Semiconductor fabrication pass/fail
secom · Tabular / process sensors- Source
- UCI SECOMMcCann & Johnston, SECOM, UCI ML Repository, DOI 10.24432/C54305
- Split
- chronological split minted by MedEval-1 on the production timestamp shipped in secom_labels.data: earliest 60% train, next 15% validation, latest 25% test. A random split is not defensible on a process-monitoring corpus -- tools drift and are recalibrated, so shuffling lets the model interpolate a tool's state from wafers measured minutes on either side, while production always extrapolates forward. Missing values imputed with the TRAIN median only; channels all-missing or constant on the training period dropped (468 of 590 channels kept).
- Sizes
- 940 train · 392 held out · 2 classes
- Classes
- pass · fail
- Registry note
- 590 sensor channels over 1,567 production lots, 6.6% failures, heavy missingness. Split is CHRONOLOGICAL on the recorded timestamp, not random: a process-monitoring model is always asked to predict forward in time, and a random split lets it interpolate within a drift it would never see the far side of in production. p is more than a third of n, so this task is expected to be hard and the intervals are expected to be wide -- that is the finding, not a defect.
Steel plate surface faults (7-class)
steel_plate_faults · Tabular / imaging-derived- Source
- Semeion Research Center / UCI Steel Plates FaultsSemeion Research Center of Sciences of Communication, UCI ML Repository, DOI 10.24432/C5J88N
- Split
- minted by MedEval-1 (stratified 60/15/25, seed 20260727); no official split is published for this corpus. The label is the argmax of the seven trailing one-hot fault columns, which are dropped from the features.
- Sizes
- 1,164 train · 486 held out · 7 classes
- Classes
- pastry · z_scratch · k_scratch · stains · dirtiness · bumps · other
- Registry note
- Geometric and luminosity descriptors from real plate photographs. Subgroup metadata is the steel grade (A300/A400) -- an industrial stratum rather than a demographic one, and dimension 5 reports it as exactly that. The largest class is a catch-all 'other', so a 7-way accuracy figure is partly a measure of how well a model predicts 'none of the above'.
Re-derive every figure on this page with python src/build_site.py --emit-json in plan_d/ — the same run writes the medical leaderboard. All benchmarks reads the same 9 dimensions per domain, and the trust graph traces any figure here back to the run that produced it.