Does Jev give the same answer twice?
RUN 003 reported one run. Every number on that page assumed a second run would say the same thing. RUN 004 tested that assumption: the same 200 frozen texts and the same request, sent to jev-1.13.0 twice more, 24 hours apart. The procedure and how each outcome would be read were committed before the first repeat request.
- TEXTS × RUNS
- 200 × 3
- LABELS THAT CHANGED
- 2 / 200
- ACCURACY, EACH RUN
- 94.0%
- NO ERROR KEPT AT 0.70
- 1 of 3 runs
2 of 200 texts did not get the same label in all three runs (1.0%; 95% Wilson interval 0.3%–3.6%). Both came back near zero confidence every time — texts Jev reported as close to a coin flip.
| TEXT | ID | LABEL | RUN 003 | REPEAT 1 | REPEAT 2 |
|---|---|---|---|---|---|
| LIFE has never been this much fun and great until you came in. You made it truly special for me. I won't forget you! enjoy @ one gbp/sms | VE001-0130 | AD | NOT_AD 0.06 ✗ | AD 0.00 | AD 0.20 |
| If i reach 100 subscribers i will tazz my self and my friend | VE001-0195 | AD | AD 0.01 | NOT_AD 0.09 ✗ | NOT_AD 0.19 ✗ |
| RUN | DATE | MODEL | CORRECT | ACCURACY | ERRORS | KEPT AT 0.70 | WRONG KEPT AT 0.70 | COST |
|---|---|---|---|---|---|---|---|---|
| RUN 003 | 2026-09-23 | jev-1.13.0 | 188 / 200 | 94.0% | 12 | 173 | 0 | $0.004929 |
| Repeat 1 | 2026-09-23 | jev-1.13.0 | 188 / 200 | 94.0% | 12 | 174 | 1 | $0.004929 |
| Repeat 2 | 2026-09-24 | jev-1.13.0 | 188 / 200 | 94.0% | 12 | 175 | 1 | $0.004929 |
The error count is the same in every run because the two texts that changed label traded places: one error became right and one right answer became an error.
Each row is a text Jev got wrong in at least one run; each mark is one run’s answer. Almost every error stays well left of 0.70. One text sits on the line and crosses it depending on the run.
| TEXT | ID | LABEL | RUN 003 | REPEAT 1 | REPEAT 2 |
|---|---|---|---|---|---|
| 1,000,000 VIEWS NEAR | VE001-0165 | NOT_AD | AD 0.69 ✗ | AD 0.72 ✗ | AD 0.70 ✗ |
| this song is awesome. these guys are the best. love this video too its hilarious lol. im getting popular fast because i rap with meaning. thumbs up if you piss next to the water in the toilet so it doesnt make noise | VE001-0197 | NOT_AD | AD 0.52 ✗ | AD 0.54 ✗ | AD 0.58 ✗ |
| Katy has the voice of gold. this video really brings the lyrics to life. I am a rapper with 25000 subscribers.. thumbs up if you hate when you take a shit and the water splashes your nuts | VE001-0191 | NOT_AD | AD 0.57 ✗ | AD 0.51 ✗ | AD 0.54 ✗ |
| yeah sure thing mate haunt got all my stuff sorted but im going sound anyway promoting hex for .by the way who is this? dont know number. Joke | VE001-0023 | NOT_AD | AD 0.29 ✗ | AD 0.32 ✗ | AD 0.42 ✗ |
| Thumbs up if FE-FE-FE-FE-FEGELEIN brought u here | VE001-0194 | NOT_AD | AD 0.34 ✗ | AD 0.38 ✗ | AD 0.36 ✗ |
| A Boy loved a gal. He propsd bt she didnt mind. He gv lv lttrs, Bt her frnds threw thm. Again d boy decided 2 aproach d gal , dt time a truck was speeding towards d gal. Wn it was about 2 hit d girl,d boy ran like hell n saved her. She asked 'hw cn u run so fast?' D boy replied "Boost is d secret of my energy" n instantly d girl shouted "our energy" n Thy lived happily 2gthr drinking boost evrydy Moral of d story:- I hv free msgs:D;): gud ni8 | VE001-0097 | NOT_AD | AD 0.25 ✗ | AD 0.23 ✗ | AD 0.05 ✗ |
| LIFE has never been this much fun and great until you came in. You made it truly special for me. I won't forget you! enjoy @ one gbp/sms | VE001-0130 | AD | NOT_AD 0.06 ✗ | AD 0.00 | AD 0.20 |
| lets get it to 1 BILLION | VE001-0163 | AD | NOT_AD 0.12 ✗ | NOT_AD 0.18 ✗ | NOT_AD 0.20 ✗ |
| If i reach 100 subscribers i will tazz my self and my friend | VE001-0195 | AD | AD 0.01 | NOT_AD 0.09 ✗ | NOT_AD 0.19 ✗ |
| http://www.bubblews.com/news/6401116-vps-solutions | VE001-0051 | NOT_AD | AD 0.13 ✗ | AD 0.00 ✗ | AD 0.00 ✗ |
| More views than nikki minaj Anaconda | VE001-0160 | NOT_AD | AD 0.07 ✗ | AD 0.04 ✗ | AD 0.09 ✗ |
| Pathaya enketa maraikara pa' | VE001-0040 | NOT_AD | AD 0.02 ✗ | AD 0.01 ✗ | AD 0.07 ✗ |
| Hi babe its Chloe, how r u? I was smashed on saturday night, it was great! How was your weekend? U been missing me? SP visionsms.com Text stop to stop 150p/text | VE001-0182 | AD | NOT_AD 0.03 ✗ | NOT_AD 0.03 ✗ | NOT_AD 0.02 ✗ |
| PAIR | LABEL FLIPS | IDENTICAL ROWS | CONFIDENCE UNCHANGED | MEDIAN |Δ| | P95 |Δ| | MAX |Δ| | CROSSED 0.70 |
|---|---|---|---|---|---|---|---|
| RUN 003 vs Repeat 1 | 2 | 130 / 200 | 137 / 200 | 0.00 | 0.06 | 0.13 | 3 |
| RUN 003 vs Repeat 2 | 2 | 136 / 200 | 141 / 200 | 0.00 | 0.07 | 0.20 | 2 |
| Repeat 1 vs Repeat 2 | 0 | 136 / 200 | 139 / 200 | 0.00 | 0.08 | 0.20 | 1 |
“Identical” means the label, both probabilities and the confidence all matched. A threshold only sees confidence, so a text whose confidence moves across it is routed differently from one run to the next even when its label never changes.
- That Jev is this repeatable on other tasks, other label sets, or on Banking77.
- Why answers change. The API exposes no sampling settings, and three runs cannot separate time of day, load or anything else from ordinary variation.
- That the two repeats agreeing with each other and not with RUN 003 means anything. With three runs that pattern is as likely to be chance as a change on the provider’s side; the resolved model was jev-1.13.0 every time.
- Anything about accuracy against other labels. The gold labels are RUN 003’s AI-adjudicated ones, with the same uncertainty described there.
Both repeats used RUN 003’s runner unchanged, with its dataset and request hashes checked on every run: 400 new API calls, $0.009858 at list price. Results are published in RUN 003’s public shape, and the comparison is compare-runs.mjs’s own output.
- ve-001/results.jsonl · sha256 472bd995a01267dd7050a7288ac83a5cdf5005209717f94d0035d11305747823
- ve-001-repeat/repeat-1/results.jsonl · sha256 1b6a086c4f1f24a1781dd1e89b93163381f8ec51d8aa8147ecb93a4587e6efe5
- ve-001-repeat/repeat-2/results.jsonl · sha256 ab1678a9eda586d2e1a37e7ef52de5bf1749c2b6644938cdfabe13e4b8c781f1
The texts come from the UCI SMS Spam Collection (Almeida and Hidalgo, 2011, doi:10.24432/C5CC84) and YouTube Spam Collection (Alberto and Lochter, 2015, doi:10.24432/C58885), redistributed under CC BY 4.0. They are untrusted user content and are shown as plain text.