Is ORDER BY over a Jev probability defensible? Six pre-registered gates on an easy corpus

AIndependent · Reproduction artifacts availableClassification20 Newsgroups (three topic probes) · n = 3602026-09-18pre-registeredverdict: PASS — all six gates

Rank rows by a Jev probability and test whether the resulting order is defensible, through calibration, inversion and invariant checks.

Accuracy

Every recorded metric

Model	Accuracy	Recall	FPR	AUROC	ECE	Latency	Cost
Jev	—	—	—	0.964	0.045	—	not recorded
Jev	—	—	—	—	—	—	not recorded
Jev	—	—	—	—	0.077	—	not recorded

Protocol

Six gate conditions with thresholds fixed before the run. Stratified sampling with near-miss negatives, per-row ambiguity filtering, mechanically derived negations and a paraphrase control. Three Jev call modes were exercised: jev_bool, jev_score and jev_choice. 350 requests at roughly 999 tokens per row; the author reports about $0.013 of spend for this corpus, which covers all three modes together and is therefore not recorded as a per-call cost. The run spanned 18–19 September 2026.

Dataset

20 Newsgroups (three topic probes)

n = 360

360 rows across medical, for-sale and space probes, stratified with near-miss negatives

Limitations (9)

Single domain — English newsgroups — which the author states limits transferability.
The author describes topic membership as an 'easy judgment' and the result as an upper bound for harder tasks. The same protocol on a hard probe failed four of six gates.
About 120 rows per probe makes a 10-bin ECE noisy; adaptive ECE is reported as sensitive at the ±0.003 level.
The 20 Newsgroups labels are themselves noisy, per the author.
Ties break ordering: only 45 of 101 possible probability values appeared, 53 rows tied at the maximum of 0.99, and 21.6% of pairs were tied. An ORDER BY over a column with that many ties is not a total order.
Graded ranking was proxied rather than measured directly on this corpus.
No formal confidence intervals for the primary metrics; Wilson intervals are used only for binning.
No latency figures are reported.
No per-call cost is recorded: the stated spend covers all three call modes together.

Note on this run

One of only two pre-registered runs in the dataset, and the one that passed. Read it beside the ESCI probe from the same author, which used the same gates and failed four of them.

What this benchmark does not prove

01That a Jev probability is safe to ORDER BY in general. The same gates failed on a harder probe from the same author, and the tie statistics mean the column is not a total order even here.
02That Jev is well calibrated. It is well calibrated on this corpus, on an easy discrimination the author explicitly flags as an upper bound.
03That an ECE of 0.0453 transfers to your data. At about 120 rows per probe the binning is noisy, and the author says so.
04Anything about ranking quality on graded relevance, which this corpus proxies rather than measures.

Source and artifacts

jev-orderby-bench — is ORDER BY over a Jev probability defensible? — yodablocks, verified 2026-09-19

Reproduction artifacts: github.com/yodablocks/jev-orderby-bench

Independent: yes · Vendor-affiliated: no · Independently reproduced: not yet

Six pre-registered gate conditions with stated thresholds, run against an easy corpus and a deliberately hard probe. Author states API costs were paid by the author and used TypeSafe's published $0.042/MTok rate for cost estimates.