Open benchmark
JevBench — AI Decision Model Benchmark
JevBench by Benchmark Heaven is a benchmark for AI decision models, including Jev-compatible systems. It measures typed decision accuracy, probability calibration, latency and cost. Maintained by Benchmark Heaven, independently of TypeSafe AI. This fully static, crawlable snapshot preserves the v1.6.1 release feed captured on 2026-10-05; the table below is rendered at build time, not loaded by JavaScript. Capability is the mean of Intelligence and Calibration, ranked within the official cost and median-latency caps. Composite is a weighted harmonic mean with configured penalties; its equal 25% weights do not make it an arithmetic average.
92ranked systems
94systems listed in total
4axes · equal 25% weights in Composite
v1.6.1snapshot captured 2026-10-05
Current leaderboard — JevBench Capability Score
Data: JevBench v1.6.1, generated 2026-10-05 · 92 ranked systems · only system-level aggregates are published. Rows inside the official caps first, in Capability rank order; the other ranked rows follow, in Composite order. Click a column header to sort.
| Capability rank | Model | Capability score | Intelligence | Calibration | Speed | Cost axis | USD per 1,000 decisions | p50 latency s (raw) |
|---|---|---|---|---|---|---|---|---|
| Quyet-1.0-Large (Chinh Nguyen, Gemma-4-31B decoder, option-letter logits) | 81.7 | 73.4 | 90.0 | 86.9 | 50.5 | $0.0445 est. | 0.1 | |
| Sage 1.3.0 (Levanto Labs) | 78.6 | 65.6 | 91.6 | 93.6 | 58.2 | $0.0247 est. | 0.2 | |
| deck-31B (krishna765, frozen Gemma-4-31B-it, TorchAO FP8 dynamic) | 77.6 | 73.0 | 82.2 | 86.7 | 49.7 | $0.0475 est. | 0.1 | |
| Jev 1.13.0 (TypeSafe AI) | 77.1 | 63.6 | 90.6 | 91.5 | 54.7 | $0.0323 | 0.2 | |
| torchcast-decision-12b (Torchcast AI, Gemma-4-12B fine-tune, option-letter logprob readout) | 71.7 | 60.5 | 82.9 | 91.7 | 56.4 | $0.0285 est. | 0.0 | |
| Jev-Omni (akhilaaa3, Gemma-4-12B merged) | 71.3 | 55.5 | 87.0 | 85.4 | 56.1 | $0.0292 est. | 0.2 | |
| Winnow-12B Q8 | 71.2 | 59.5 | 83.0 | 86.7 | 56.6 | $0.0281 est. | 0.1 | |
| Cygnet (blockbrain, frozen Gemma-4-12B-it) | 70.9 | 54.8 | 87.0 | 91.8 | 56.4 | $0.0283 est. | 0.0 | |
| Plumb-4B (crh225, JevK5 v0.2 + LoRA) | 65.4 | 43.0 | 87.9 | 93.8 | 63.1 | $0.017 est. | 0.0 | |
| decider-4b v2 (Mapika) | 64.9 | 40.1 | 89.6 | 91.8 | 64.5 | $0.0152 est. | 0.0 | |
| JevK5 v0.3 (4B) | 64.2 | 38.5 | 89.9 | 93.9 | 63.1 | $0.017 est. | 0.0 | |
| Clef-Flash (Cloudflare, Qwen3.5-9B post-train with a joint schema head, multimodal, measured on text) | 63.8 | 42.2 | 85.5 | 88.4 | 46.8 | $0.0593 est. | 0.1 | |
| deck-4B v1.0 (krishna765, JevK5 + LoRA, FP8 weight-only) | 63.6 | 38.6 | 88.6 | 89.9 | 63.5 | $0.0165 est. | 0.1 | |
| JevK5 v0.2.0 | 62.8 | 36.3 | 89.4 | 91.6 | 63.1 | $0.017 est. | 0.0 | |
| Hopper | 62.4 | 34.7 | 90.1 | 91.1 | 62.3 | $0.0181 est. | 0.1 | |
| Imajev-4B (RTX 5090) | 61.9 | 34.6 | 89.2 | 90.5 | 63.3 | $0.0168 est. | 0.1 | |
| jqv (Qwen3-32B zero-shot) | 61.9 | 33.9 | 89.9 | 83.1 | 51.2 | $0.0424 est. | 0.1 | |
| metask-jev-4b | 61.8 | 39.2 | 84.4 | 90.8 | 57.7 | $0.0256 est. | 0.1 | |
| lev (Interfaze AI, Qwen3.5-4B + LoRA r32, label-token readout + candidate-path head, per-bucket temperatures) | 61.7 | 41.1 | 82.3 | 88.0 | 61.8 | $0.0188 est. | 0.1 | |
| Malkuth-4B (newfull5, Kev post-train) | 61.1 | 33.8 | 88.4 | 89.5 | 55.8 | $0.0297 est. | 0.1 | |
| Decision 4B v1.2 (FlyMyJev, Qwen3.5-4B + LoRA) | 60.4 | 32.4 | 88.4 | 93.9 | 63.1 | $0.017 est. | 0.0 | |
| Quyet-1.0-Medium (Chinh Nguyen, Qwen3.5-4B decoder, option-letter logits) | 59.2 | 41.7 | 76.8 | 89.5 | 63.4 | $0.0166 est. | 0.1 | |
| Manchego v2.1 | 58.3 | 32.5 | 84.0 | 90.7 | 64.3 | $0.0155 est. | 0.1 | |
| jev-local (Qwen3.5-9B) | 57.5 | 41.9 | 73.1 | 75.6 | 58.9 | $0.0235 est. | 0.4 | |
| Decision 4B v1.1 (FlyMyJev, Qwen3.5-4B + LoRA) | 57.2 | 29.1 | 85.2 | 94.0 | 63.1 | $0.017 est. | 0.0 | |
| JEV Qwen3.5-9B Base NVFP4 | 57.1 | 29.3 | 85.0 | 93.6 | 47.6 | $0.0558 est. | 0.0 | |
| typecastlm (Mikhail Gribov, Qwen3.5-4B computed head) | 56.5 | 29.8 | 83.3 | 92.8 | 64.0 | $0.0159 est. | 0.0 | |
| SemIf, formerly OpenJev (Qwen3.5-4B, TheoLeeCJ) | 55.4 | 27.1 | 83.6 | 90.8 | 63.1 | $0.017 est. | 0.1 | |
| local-jev Qwen3.5-4B | 55.0 | 24.1 | 85.8 | 86.1 | 59.3 | $0.0228 est. | 0.1 | |
| Decision 2B (FlyMy.AI, v59) | 54.8 | 22.9 | 86.8 | 90.4 | 66.4 | $0.0132 est. | 0.1 | |
| spark-s1-4b-v6 (Open Spark Jev, abhishek085) | 54.0 | 43.2 | 64.8 | 87.0 | 60.4 | $0.0209 est. | 0.1 | |
| ZeroEntropy zerank-2 | 49.4 | 15.6 | 83.2 | 82.9 | 48.4 | $0.0525 est. | 0.2 | |
| open-alternative-jev (Qwen3.5-4B, IkerMoel) | 49.1 | 22.3 | 75.9 | 91.3 | 63.3 | $0.0168 est. | 0.0 | |
| OpenSourceJev (Qwen3.5-4B Q4_K_M, native llama.cpp) | 48.6 | 23.2 | 73.9 | 77.1 | 69.1 | $0.0107 est. | 0.3 | |
| Malkuth-2B (newfull5, Kev post-train) | 47.6 | 15.9 | 79.3 | 92.0 | 65.6 | $0.0141 est. | 0.0 | |
| Nemotron Diffusion 8B (pst2154, optimized vLLM) | 47.5 | 20.4 | 74.5 | 93.8 | 55.5 | $0.0305 est. | 0.0 | |
| Qwen3-Reranker-4B | 44.7 | 16.3 | 73.2 | 82.1 | 48.4 | $0.0525 est. | 0.2 | |
| decider-2b (Mapika) | 44.6 | 20.7 | 68.5 | 94.6 | 64.9 | $0.0148 est. | 0.0 | |
| BAAI bge-reranker-v2-m3 | 44.1 | 0.1 | 88.1 | 93.9 | 59.3 | $0.0227 est. | 0.0 | |
| kev 4B (research preview) | 44.0 | 19.5 | 68.5 | 90.5 | 65.8 | $0.0138 est. | 0.1 | |
| Mixedbread mxbai-rerank-base-v2 | 43.9 | 1.1 | 86.6 | 92.6 | 60.3 | $0.021 est. | 0.0 | |
| lev-350m (Franck Verrot, LFM2.5-350M) | 43.3 | 4.8 | 81.9 | 94.0 | 80.0 | $0.00463 est. | 0.0 | |
| Certo v1 (AltSlate Labs) | 43.2 | 0.2 | 86.1 | 92.5 | 96.9 | $0.00127 est. | 0.0 | |
| smalljev semantic-v9 | 43.0 | 8.7 | 77.2 | 90.2 | 60.8 | $0.0203 est. | 0.1 | |
| Alibaba GTE Reranker ModernBERT-base | 41.7 | 0.0 | 83.5 | 93.8 | 69.3 | $0.0106 est. | 0.0 | |
| Decision Fast (FlyMy.AI, v53a) | 40.6 | 6.1 | 75.1 | 91.5 | 80.1 | $0.00461 est. | 0.1 | |
| openJev Verdict 1.4 | 40.4 | 5.0 | 75.7 | 81.5 | 86.6 | $0.00279 est. | 0.3 | |
| Quyet-1.0-Small-EN (Chinh Nguyen, ModernBERT-base encoder with fixed heads) | 39.5 | 6.1 | 73.0 | 94.9 | 89.1 | $0.00231 est. | 0.0 | |
| OpenDecision (ModernBERT-large zero-shot) | 38.0 | 7.4 | 68.6 | 87.1 | 79.1 | $0.00497 est. | 0.1 | |
| verdict-small (Manavarya09, multilingual-e5-small 118M) | 37.6 | 2.3 | 72.9 | 89.5 | 100.0 | $0.000866 est. | 0.0 | |
| kev 0.6B (research preview) | 36.9 | 7.6 | 66.3 | 91.5 | 80.1 | $0.00461 est. | 0.1 | |
| Raw Phi-4 mini direct logits | 35.3 | 13.7 | 57.0 | 91.9 | 53.2 | $0.0363 est. | 0.0 | |
| Fastino GLiNER-2.5-Decide (hosted API) | 34.6 | 7.0 | 62.1 | 84.7 | 45.8 | $0.064 | 0.4 | |
| Raw Qwen3 4B Instruct 2507 direct logits | 34.6 | 34.0 | 35.2 | 91.2 | 63.4 | $0.0166 est. | 0.0 | |
| kev 0.5B | 32.9 | 4.6 | 61.2 | 92.4 | 80.1 | $0.00461 est. | 0.0 | |
| Quyet-1.0-Small (Chinh Nguyen, SEA-LION-ModernBERT-300M encoder with fixed heads) | 32.5 | 4.0 | 61.0 | 94.9 | 79.6 | $0.0048 est. | 0.0 | |
| Quyet-1.0-Tiny (Chinh Nguyen, mmBERT-small encoder (16 layers kept) with fixed heads) | 31.1 | 3.7 | 58.5 | 95.1 | 88.4 | $0.00243 est. | 0.0 | |
| Open Jev JSON Canvas (JoshuaSP) | 30.6 | 61.2 | 0.0 | 85.5 | 49.3 | $0.049 est. | 0.1 | |
| openJev Verdict (heman10x, ModernBERT-base 151M) | 26.6 | 4.1 | 49.1 | 79.4 | 86.6 | $0.00279 est. | 0.3 | |
| CLM-8B (Contrastive-LM, clm-latest) | 20.1 | 0.1 | 40.0 | 92.6 | 50.5 | $0.0447 est. | 0.0 | |
| Deem 0.8B v1 | 18.1 | 6.5 | 29.6 | 85.4 | 80.3 | $0.00454 est. | 0.2 | |
| Laya multilingual | 17.0 | 0.3 | 33.7 | 77.8 | 82.2 | $0.00393 est. | 0.4 | |
| Raw Qwen3 1.7B direct logits | 8.3 | 5.1 | 11.5 | 93.2 | 68.5 | $0.0112 est. | 0.0 | |
| Raw Qwen3 0.6B direct logits | 8.2 | 5.6 | 10.9 | 93.5 | 77.6 | $0.00559 est. | 0.0 | |
| wity-1 (Wity, reasoning auto) | 79.1 | 70.3 | 87.9 | 71.8 | 58.8 | $0.0236 | 1.6 | |
| swanOne (blockbrain, Qwen3.8-Flash-Next NVFP4) | 72.4 | 56.2 | 88.6 | 82.6 | 42.2 | $0.0848 est. | 0.3 | |
| NInfer Qwen3.8-Flash-Next mixed | 69.7 | 49.0 | 90.4 | 89.3 | 42.5 | $0.0823 est. | 0.1 | |
| Qwen3.5-9B Jev-like data-mix v2 | 60.9 | 41.9 | 79.8 | 85.4 | 45.7 | $0.0646 est. | 0.2 | |
| JevOne | 68.9 | 46.0 | 91.8 | 89.8 | 39.8 | $0.101 est. | 0.1 | |
| decider-35b-a3b (Mapika) | 66.2 | 48.7 | 83.6 | 91.3 | 34.4 | $0.154 est. | 0.0 | |
| Bespoke Nimble 9B (Bespoke Labs) | 56.5 | 43.1 | 69.9 | 86.1 | 36.8 | $0.128 est. | 0.1 | |
| reflex 4B (kshetrajna12) | 59.0 | 30.9 | 87.0 | 68.8 | 63.1 | $0.0169 est. | 1.4 | |
| AutoJev-27B (denis-pplx, Qwen3.8-27B) | 71.6 | 55.6 | 87.6 | 88.6 | 29.4 | $0.226 est. | 0.1 | |
| Eikos-27B (caiovicentino1, Qwen3.8-27B) | 77.2 | 64.4 | 90.0 | 88.1 | 28.7 | $0.238 est. | 0.1 | |
| NInfer Qwen3.8-27B NVFP4 | 68.0 | 51.8 | 84.2 | 90.5 | 29.1 | $0.231 est. | 0.1 | |
| OpenJev (thinking, BF16) | 75.9 | 70.3 | 81.4 | 72.6 | 28.5 | $0.241 est. | 0.9 | |
| Open-Jev 9B (Zefan Cai) | 61.8 | 43.9 | 79.8 | 74.4 | 33.1 | $0.17 est. | 0.6 | |
| Standard One 8B (Standard Thinking) | 58.4 | 32.8 | 84.0 | 92.8 | 43.3 | $0.0777 est. | 0.0 | |
| Clef (Cloudflare, Qwen3.8-27B post-train with a joint schema head, multimodal, measured on text) | 75.3 | 59.1 | 91.6 | 83.4 | 28.1 | $0.249 est. | 0.2 | |
| Bev / Bonsai 27B | 66.1 | 43.8 | 88.5 | 73.0 | 28.2 | $0.247 est. | 1.0 | |
| LitJev (Qwen3.8-27B) | 66.6 | 43.3 | 89.9 | 68.2 | 28.4 | $0.244 est. | 1.5 | |
| system-one (Qwen3-8B, Sean Goedecke) | 29.7 | 29.3 | 30.2 | 90.9 | 45.0 | $0.0683 est. | 0.0 | |
| kev 8B (research preview) | 50.0 | 29.3 | 70.8 | 89.0 | 40.4 | $0.0968 est. | 0.1 | |
| Raw Qwen3 8B direct logits | 30.1 | 26.5 | 33.7 | 88.7 | 45.5 | $0.0654 est. | 0.1 | |
| Open-Jev 2B (Zefan Cai) | 47.7 | 21.8 | 73.6 | 77.1 | 33.1 | $0.17 est. | 0.4 | |
| Open-Jev 27B v1.1 (Zefan Cai, LoRA + decision head on Qwen3.8-27B) | 68.0 | 52.5 | 83.5 | 72.5 | 14.4 | $0.716 est. | 0.8 | |
| SimpleJev (Qwen3.5-0.8B, CPU) | 31.6 | 13.0 | 50.2 | 58.9 | 70.3 | $0.00976 est. | 4.3 | |
| Qwen3.5-0.8B Decision Model (Mourad Ghafiri) | 42.6 | 7.3 | 78.0 | 55.9 | 79.5 | $0.00482 est. | 5.6 | |
| Laya typed-decisions | 44.2 | 5.5 | 83.0 | 70.8 | 82.5 | $0.00382 est. | 0.9 | |
| open-jev-deberta-v3-large (local CPU) | 35.4 | 2.8 | 68.0 | 67.6 | 77.6 | $0.00558 est. | 1.7 | |
| Laya (Convai Innovations, ModernBERT-large 421M) | 32.5 | 1.8 | 63.2 | 73.3 | 84.9 | $0.00319 est. | 0.8 | |
| Mirror | 25.8 | 0.0 | 51.6 | 73.5 | 89.3 | $0.00228 est. | 0.7 |
Best Jev alternatives
The strongest other systems in this snapshot, by Capability Score: Quyet-1.0-Large (81.7) · Sage 1.3.0 (78.6) · deck-31B (77.6). The Jev reference row (TypeSafe AI) is excluded from this list. For the full comparison — open-weight options, use-case fit and per-row evidence — see the Jev alternatives page.
How the score works
IntelligenceChance-corrected accuracy on the frozen public and sealed tiers.
CalibrationDistribution quality of the typed answers a system produces.
SpeedMedian (p50) latency on a logarithmic scale; self-hosted runs carry a published adjustment.
CostUSD per 1,000 complete decisions — a whole question, not per 1,000 tokens.
JevBench Capability Score: Arithmetic mean of Intelligence and Calibration; ranked models within both official caps. The official caps are 2× the Jev reference (jev-1.13.0@v1.5.7): at most 0.0646 USD per 1,000 decisions — 1.23 s median latency. The Composite is a weighted harmonic mean of Intelligence, Calibration, Speed and Cost, with equal 25% weights (not an arithmetic average) and configured squared penalties; it is secondary on open weights and headlines the hosted API board. Cost figures are labeled by evidence basis; “est.” means estimated, not a measured charge.
Listed, not ranked
These systems appear on the board but are not part of the ranking in this release.
- wity-1 (Wity, reasoning always) — listed: variant listed for reference, not ranked in this release
- wity-1 (Wity, reasoning off) — listed: variant listed for reference, not ranked in this release
More
- Live JevBench board — full interactive view, filters and head-to-heads
- Jev alternatives, compared
- How to choose a Jev-class model
- Is Jev open source?
- Public results API (JSON)
- Submit a system to JevBench
- JevBench method repository (GitHub)
- JevBench results dataset (Hugging Face)