Consumer credit risk
LIVEThe same dimensions, the same evaluators and the same seed, pointed at a consumer credit decision. Calibration and deferral are the two that need no translation at all. No separate specification document has been published for this benchmark — the MedEval-1 spec is the document, and this page is that instrument run on another corpus. Everything below is read from the same payload.json the medical leaderboard renders from.
The index is the weighted mean of the dimensions that were scored, on the weights published in the spec. Dimensions this corpus cannot support are excluded from it rather than counted as zero, which is why an index here is not comparable to a medical one model-for-model — it is a mean over a different set of questions, and the table below says which.
Every model, on all 9 dimensions
Ranked by index. An em-dash is a dimension the harness left null.| # | Model | Index | 1 Accuracy | 2 Cross-site | 3 Calibration | 4 Limited data | 5 Subgroup | 6 Corruption | 7 Uncertainty | Specification sensitivity | Cost, latency & footprint |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Gradient boosting200 stages, depth 3, lr 0.05 | 58.4 | 35.2 | — | 67.7 | 63.2 | 32.8 | 95.0 | 76.5 | — | — |
| 2 | Logistic regressionstandardised, L2, C=1 | 56.0 | 32.4 | — | 62.5 | 62.9 | 27.6 | 96.7 | 75.1 | — | — |
| 3 | Random forest400 trees | 54.2 | 22.9 | — | 72.2 | 68.3 | 21.3 | 94.6 | 75.0 | — | — |
| 4 | MLP (128, 64)2 hidden layers, adam, early stopping when n>=200 | 48.9 | 0.7 | — | 89.2 | 100.0 | 0.0 | 76.9 | 93.5 | — | — |
| 5 | Logistic regressionstandardised, L2, C=1 | 47.3 | 1.6 | — | 94.0 | 66.7 | 0.0 | 81.5 | 94.4 | — | — |
| 6 | MLP (128, 64)2 hidden layers, adam, early stopping when n>=200 | 44.2 | 3.0 | — | 62.4 | 100.0 | 0.0 | 87.5 | 55.7 | — | — |
| 7 | Gradient boosting200 stages, depth 3, lr 0.05 | 43.6 | 1.3 | — | 93.6 | 66.7 | 0.0 | 63.7 | 94.3 | — | — |
| 8 | Random forest400 trees | 38.8 | 5.1 | — | 85.5 | 83.1 | 0.0 | 31.5 | 93.0 | — | — |
- 2. Cross-site robustness — No re-acquisition of the same subjects exists in this corpus, so there is nothing to measure. Widening the definition to fill the cell would be a spec change, not a demo.
- . — Not computed on any record in this benchmark, and not approximated.
- . — Not computed on any record in this benchmark, and not approximated.
The measurements the scores came from
Raw metrics, not the 0–100 rescaling.| Model | Balanced accuracy (95% CI) | AUC | ECE | Risk–coverage AUC | Mean label-budget retention |
|---|---|---|---|---|---|
| Gradient boosting | 0.676[0.616–0.730] | 0.811 | 0.065 | 0.883 | 0.632 |
| Logistic regression | 0.662[0.611–0.723] | 0.792 | 0.075 | 0.876 | 0.629 |
| Random forest | 0.614[0.560–0.675] | 0.799 | 0.056 | 0.875 | 0.683 |
| MLP (128, 64) | 0.503[0.499–0.510] | 0.693 | 0.022 | 0.968 | 1.000 |
| Logistic regression | 0.508[0.502–0.516] | 0.727 | 0.012 | 0.972 | 0.667 |
| MLP (128, 64) | 0.515[0.491–0.557] | 0.647 | 0.075 | 0.778 | 1.000 |
| Gradient boosting | 0.506[0.498–0.517] | 0.716 | 0.013 | 0.971 | 0.667 |
| Random forest | 0.525[0.513–0.540] | 0.679 | 0.029 | 0.965 | 0.831 |
Balanced accuracy carries a bootstrap 95% interval; read it before reading the rank. ECE is expected calibration error, so lower is better and the 0–100 dimension score inverts it. Every value here is on the record; hover any figure for the unrounded number.
Who the best model fails
LIVEDimension five, on Gradient boosting — the top-ranked model above. Basis, in the evaluator’s own words: worst demographic subgroup (real metadata). This is the strong reading — the corpus ships the attributes, so the harness did not have to fall back to the worst-class proxy every medical imaging task uses.
| <45worst | 0.664 | |
| 45-59 | 0.713 |
Per-group recall, and the gap between the best and worst group. These numbers measure who the model fails on this corpus. They are not an audit of any deployed system, and the subgroup attributes are model inputs here, as they are in the standard formulation of the task.
The corpus
Consumer credit risk (Statlog)
german_credit · Tabular / financial- Source
- UCI Statlog German CreditHofmann / UCI ML Repository, Statlog (German Credit Data)
- Split
- minted by MedEval-1 (stratified 60/15/25, seed 20260727)
- Sizes
- 600 train · 250 held out · 2 classes
- Classes
- good · bad
- Registry note
- Minted stratified split at the standard seed; no official split is released, and that is a weaker provenance claim than the Adult task next to it. 1,000 applicants, so the held-out split is 250 -- small, and the confidence intervals say so. Dimension 2 is undefined for the same reason as Adult.
Caravan insurance purchase (CoIL 2000)
coil2000_caravan · Tabular / insurance- Source
- UCI CoIL Challenge 2000 (tic-mld)van der Putten & van Someren (eds.), CoIL Challenge 2000, Sentient Machine Research / LIACS TR 2000-09
- Split
- official released split (CoIL Challenge 2000: ticdata2000.txt 5822 train / ticeval2000.txt 4000 test). The test labels were released separately, in tictgts2000.txt, only after the challenge closed -- they were not available to anyone tuning on this benchmark at the time. Test scored whole; validation carved from ticdata2000.txt only (stratified 15%, seed 20260727).
- Sizes
- 4,948 train · 4,000 held out · 2 classes
- Classes
- no policy · policy
- Registry note
- Official released split with the held-out labels published as a separate file -- the same provenance class as Adult. 5.9% positive. IMPORTANT: the demographic-looking columns (religion, education, social class, marital status) are POSTCODE-AREA PERCENTAGES, not attributes of an individual. Slicing on them would be an ecological analysis, so they are deliberately not offered as dimension-5 subgroups; reporting them as individual demographics would be an ecological-fallacy claim.
Financial
Consumer credit risk — the question NakedSignal started on.
A consumer credit-risk task, scored on the same dimensions as everything else on this site. It sits on the question the company was built around: whether a model's output is trustworthy enough to underwrite on. That is the question this standard formalises.
Calibration and deferral are the dimensions that matter most here, and they are the two that need no translation at all. An underwriter does not need a model that is right on average; they need one whose confidence means something, and that knows when to hand the file to a human.
- A small corpus with no official split. The split is minted at the standard seed — a weaker provenance claim than the labour task next to it, stated on the card — and the held-out set is small enough that the confidence intervals do real work. Read them.
- Age is the only subgroup attribute scored. Groups smaller than the harness's minimum are dropped rather than reported, because a gap computed on a handful of rows is noise dressed as a finding.
- Dimension two is undefined, exactly as on the labour task. Dimension six is measured, and this corpus is the mild one: retention sits between 0.91 and 0.95 across the three scored families, with a dropped field the costliest and a unit change and a transcription error close to indistinguishable from each other. The wide spread between those families on the labour task is not reproduced here, and the aggregate figure on /synthetic should not be read onto this card. Coding drift is reported unscored for the same reason as on the labour task — no one-hot block for it to touch.
Re-derive every figure on this page with python src/build_site.py --emit-json in plan_d/ — the same run writes the medical leaderboard. All benchmarks reads the same 9 dimensions per domain, and the trust graph traces any figure here back to the run that produced it.