← All evidence
Three-way email classification, held-out fresh set
AIndependent · Reproduction artifacts availableSecurity detectionFresh set (author's own assembly) · n = 3,300date not stated by source
Label an email as legitimate, spam or phishing, on a set held out from the main run.
Accuracy
95.4%
Jev jev-1.13.0, enriched evidence
no interval reported
96.3%
TF-IDF + logistic regression enriched
no interval reported
supervised — trained on this dataset, not zero-shot
Every recorded metric
| Model | Accuracy | Recall | FPR | AUROC | ECE | Latency | Cost |
|---|---|---|---|---|---|---|---|
Jev | 95.4% | — | — | — | — | — | not recorded |
TF-IDF + logistic regression | 96.3% | — | — | — | — | — | not recorded |
Protocol
Same three-way task and the same enriched configurations as the main run, on a separately assembled set. Recorded as its own run because it is a different dataset.
Dataset
Fresh set (author's own assembly)
n = 3,300
1,100 legitimate / 1,100 spam / 1,100 phishing
Limitations (4)
- The source does not state a run date.
- No confidence intervals reported.
- Only overall accuracy is reported for this set; per-class recall is not, so the accuracy-versus-recall tension visible on the main set cannot be checked here.
- The author describes the sets as not locked benchmarks and notes pretraining exposure is unknown.
Note on this run
Both systems score lower here than on the main set, and the ordering flips: the regression edges ahead on overall accuracy. Held-out replication moving the numbers is the normal case, not an anomaly.
What this benchmark does not prove
- 01That the TF-IDF baseline is more accurate than Jev in general. It leads by 0.94 points on one held-out set with no interval reported.
- 02That either system degrades on new data in a predictable way. Two datasets is not a trend.
- 03That the main-set recall findings carry over. Per-class recall was not reported for this set at all.
Source and artifacts
jev-spam-eval — three-way email classification: Jev variants vs TF-IDF logistic regression — bitnovus, verified 2026-09-19
Reproduction artifacts: github.com/bitnovus/jev-spam-eval
Independent: yes · Vendor-affiliated: no · Independently reproduced: not yet
Public code and stated protocol. The repository does not state a run date, the author's affiliation, or who funded API costs; it does report measured spend of about $1.35 for the full paired run.