Contract provision type
LIVEOne hundred long-tailed provision types over SEC contract text, on the LexGLUE published split. Plain accuracy flatters a frequency-matching baseline here, which is exactly why this corpus is useful to the calibration and deferral dimensions rather than to the accuracy one. No separate specification document has been published for this benchmark — the MedEval-1 spec is the document, and this page is that instrument run on another corpus. Everything below is read from the same payload.json the medical leaderboard renders from.
The index is the weighted mean of the dimensions that were scored, on the weights published in the spec. Dimensions this corpus cannot support are excluded from it rather than counted as zero, which is why an index here is not comparable to a medical one model-for-model — it is a mean over a different set of questions, and the table below says which.
Every model, on all 9 dimensions
Ranked by index. An em-dash is a dimension the harness left null.| # | Model | Index | 1 Accuracy | 2 Cross-site | 3 Calibration | 4 Limited data | 5 Subgroup | 6 Corruption | 7 Uncertainty | Specification sensitivity | Cost, latency & footprint |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | TF-IDF + MLP (256)SVD-256 then 1 hidden layer of 256 | 69.9 | 68.8 | — | 75.1 | 88.9 | 0.0 | 97.4 | 93.4 | — | — |
| 2 | TF-IDF + linear SVM (Platt-scaled)squared hinge, C=1, 5-fold Platt calibration | 66.4 | 73.1 | — | 47.1 | 89.6 | 0.0 | 97.7 | 94.5 | — | — |
| 3 | GTE-base frozen + linear probe109M frozen, mean-pooled 768-d, logistic head; contrastive sentence objective, general domain | 63.7 | 68.5 | — | 31.4 | 94.9 | 0.0 | 96.1 | 92.0 | — | 100.0 |
| 4 | BiomedBERT frozen + linear probe109M frozen, mean-pooled 768-d, logistic head | 61.3 | 65.7 | — | 32.1 | 89.7 | 0.0 | 90.3 | 90.9 | — | 100.0 |
| 5 | TF-IDF + logistic regression1-2 grams, 200k features, sublinear tf | 60.5 | 69.5 | — | 18.6 | 93.6 | 0.0 | 97.6 | 93.4 | — | — |
| 6 | PubMedBERT-sentence frozen + linear probe109M frozen, mean-pooled 768-d, logistic head; biomedical pretraining PLUS a sentence objective | 60.3 | 64.2 | — | 27.2 | 88.8 | 0.0 | 93.2 | 89.8 | — | 98.7 |
| 7 | TF-IDF + complement naive Bayes1-2 grams, alpha=0.3 | 55.7 | 63.8 | — | 0.0 | 95.0 | 0.0 | 99.0 | 91.6 | — | — |
- 2. Cross-site robustness — No re-acquisition of the same subjects exists in this corpus, so there is nothing to measure. Widening the definition to fill the cell would be a spec change, not a demo.
- . — Not computed on any record in this benchmark, and not approximated.
The measurements the scores came from
Raw metrics, not the 0–100 rescaling.| Model | Balanced accuracy (95% CI) | AUC | ECE | Risk–coverage AUC | Mean label-budget retention |
|---|---|---|---|---|---|
| TF-IDF + MLP (256) | 0.691[0.680–0.702] | 0.985 | 0.050 | 0.934 | 0.889 |
| TF-IDF + linear SVM (Platt-scaled) | 0.734[0.723–0.746] | 0.981 | 0.106 | 0.946 | 0.896 |
| GTE-base frozen + linear probe | 0.688[0.676–0.701] | 0.982 | 0.137 | 0.920 | 0.949 |
| BiomedBERT frozen + linear probe | 0.660[0.649–0.670] | 0.979 | 0.136 | 0.909 | 0.897 |
| TF-IDF + logistic regression | 0.698[0.689–0.713] | 0.988 | 0.163 | 0.935 | 0.936 |
| PubMedBERT-sentence frozen + linear probe | 0.645[0.634–0.660] | 0.974 | 0.146 | 0.899 | 0.888 |
| TF-IDF + complement naive Bayes | 0.642[0.633–0.654] | 0.971 | 0.494 | 0.917 | 0.950 |
Balanced accuracy carries a bootstrap 95% interval; read it before reading the rank. ECE is expected calibration error, so lower is better and the 0–100 dimension score inverts it. Every value here is on the record; hover any figure for the unrounded number.
The corpus
Contract provision type (100-class)
ledgar · Legal text- Source
- LexGLUE / LEDGAR (SEC EDGAR filings)Tuggener et al., LREC 2020; Chalkidis et al., LexGLUE, ACL 2022
- Split
- official LexGLUE published split (train/validation/test released files, test scored whole); train capped to 12,000 by a stratified subsample at seed 20260727
- Sizes
- 12,000 train · 10,000 held out · 100 classesTraining capped at the standard budget from 60,000 available rows, so every model in the suite sees the same amount of data.
- Registry note
- Official published split. 100 long-tailed classes over short contract provisions, so plain accuracy flatters a frequency-matching baseline and macro-averaged figures are the honest headline. The long tail is why this task is useful to the calibration and deferral dimensions rather than to the accuracy one.
Re-derive every figure on this page with python src/build_site.py --emit-json in plan_d/ — the same run writes the medical leaderboard. All benchmarks reads the same 9 dimensions per domain, and the trust graph traces any figure here back to the run that produced it.