Three-way email classification: Jev prompt variants vs TF-IDF logistic regression
Label an email as legitimate, spam or phishing.
| Model | Accuracy | Recall | FPR | AUROC | ECE | Latency | Cost |
|---|---|---|---|---|---|---|---|
Jev | 93.6% | 85.7% | — | — | — | — | not recorded |
Jev | 98.0% | 98.4% | — | — | — | — | not recorded |
Jev | 98.6% | 98.4% | — | — | — | — | not recorded |
TF-IDF + logistic regression | 98.9% | 75.3% | — | — | — | — | not recorded |
Jev + TF-IDF ensemble | 99.3% | 95.7% | — | — | — | — | not recorded |
Three-way classification. Jev accessed as jev-1.13.0 in three prompt configurations: text-only, enriched with extracted evidence, and enriched with evidence-focused wording. The TF-IDF logistic regression baseline used fixed, untuned settings. Five-fold cross-validation with near-duplicate grouping. The full paired run issued 19,772 requests in about 252 seconds at concurrency 16, consuming 32.06M input tokens.
- The source does not state a run date, so these figures cannot be placed against a specific model version or price period.
- No confidence intervals are reported at any sample size.
- The author is explicit that 'no task-specific fitting is not no supervision': the detailed binary criteria used in the enriched prompts were informed by labelled errors, so the enriched configurations are not cleanly zero-shot.
- The author notes pretraining exposure to these messages is unknown and the labels are confounded, and describes the set as 'not a locked benchmark'.
- Calibration was not measured. The author notes that probability-shaped outputs still require calibration checks.
- The repository does not state the author's affiliation or who funded API costs.
- No per-call cost is recorded. The source reports about $1.35 of measured spend, but that total covers the full paired run across every prompt configuration and every set, so it cannot be attributed to a single configuration on this dataset.
What this benchmark does not prove
- 01That Jev has a phishing recall of 98.4%. That is the enriched configuration on this dataset; the text-only configuration on the same data scored 85.7%, and a different dataset scored lower still.
- 02That prompt enrichment is free. The author states the enriched criteria were informed by labelled errors, so part of the gain is supervision that a genuinely cold-start deployment would not have.
- 03That the TF-IDF baseline is the better choice because it has the highest accuracy. It has the worst phishing recall of the serious configurations, and on a detection workload that is usually the number that decides it.
- 04That the ensemble's 99.3% is production performance. It is an unvalidated combination on a set the author describes as not locked, with no intervals reported.
jev-spam-eval — three-way email classification: Jev variants vs TF-IDF logistic regression — bitnovus, verified 2026-09-19
Reproduction artifacts: github.com/bitnovus/jev-spam-eval
Independent: yes · Vendor-affiliated: no · Independently reproduced: not yet
Public code and stated protocol. The repository does not state a run date, the author's affiliation, or who funded API costs; it does report measured spend of about $1.35 for the full paired run.