Jev 决策模型在 Agent Harness 中的基准评测：能做什么、不能做什么

Model under test: jev-1.13.0 · Date: Sep 2026 · Code & raw results: github.com/Aitejiu/jev-harness-lab A black-box engineering evaluation: 10 public datasets, ~22,500 API calls, 52.2M input tokens, $2.19 total. Jev is TypeSafe AI's "System One" decision model. You send it a state plus typed questions; it returns typed answers with probabilities and confidence. It never generates text — and it costs about $0.00004 and 0.3s per call. That makes it a candidate for the "fast-think layer" of an agent harness: the flood of narrow decisions an agent makes every turn. I spent a few days measuring where that actually works. TL;DR Works (directly usable in a harness): Capability Dataset Result Indirect prompt-injection detection InjecAgent (1,105) at threshold 0.10: P/R 100%, 0% false positives Reranking BEIR SciFact (900 pairs) BM25 → Jev: MRR 0.622 → 0.843, Hit@1 50% → 78.3% Intent classification SNIPS / Banking77 97.9% (7 classes) / 80.3% (77 classes) Tool catalog routing MetaTool (199 tools) 96.5% with similar distractors (k=5) Skill routing SkillRetBench (501 skills) two-stage R@1 75.8% vs 38.0% best baseline Shell command risk gate 130 hand-built commands 100% dangerous caught, 98.2% safe passed Tool-relevance detection BFCL (1,140) 77.0% live accuracy after iteration (59.5% first try) Doesn't work (negative results): Task Result Why Model-difficulty routing 51% accuracy (no signal) requires predicting another model's failure modes Trajectory failure attribution AUROC 0.560 (random) requires cross-step causal reasoning Non-English tasks KO R@1 48.9% vs EN 61.5% English-primary training Two engineering lessons worth more than the benchmarks: For fuzzy semantic judgments, orthogonal decomposition + code composition beats single-question prompting (shell-gate false positives 14.5% → 1.8%). For multi-label decisions, let choice compete first, then let noul verify — multi-skill routing went from 9.0% to 81.0% R@1. Model gives calibrated local judgments; code holds the control flow. 1. The model and its interface POST https://api.typesafe.ai/v1/systemone Authorization: Bearer $TYPESAFE_API_KEY { "state": "...", # string | object | array, text only "model": "jev-latest", "questions": { "..." : Question } } Enter fullscreen mode Exit fullscreen mode Primitive Meaning Returns noul yes/no probability 0–1 (no separate confidence field) choice pick one of a closed set choice + probability distribution + confidence score rate on an ordered rubric probability-weighted score + legend + confidence Known limits, all verified during testing: choice caps at 255 options — beyond that, split into stages. state + questions share ~32k tokens; large corpora must be retrieved first. Text only; English-primary (CJK accuracy drops). Pricing reference: $42 / billion input tokens. 2. How I evaluated Every experiment follows the same shape (eval/ in the repo): load a dataset → build state/questions → concurrent calls with JSONL caching → threshold sweeps, coverage curves, calibration and cost. Metric definitions matter, so being explicit: Threshold sweep — treat the returned probability as a score; report precision/recall/F1/FPR at each threshold. Confidence gating curve — noul has no confidence field, so certainty is proxied by max(p, 1-p); report coverage vs accuracy-within-coverage. Conditional Hit@1 (SkillRet) — fraction of queries where the gold candidate was in the shortlist and Jev picked it; isolates the selector from retrieval. Cost — input tokens × $42/B (output-token billing, if any, would add slightly). Datasets: InjecAgent, neuralchemy prompt-injection, synthesized tool-output injections, BEIR SciFact, SNIPS, Banking77, MetaTool ToolE, BFCL v3, SkillRet, SkillRetBench, RouterBench, Who&When, plus a 130-command shell risk set. 3. Results 3.1 Indirect injection detection Real data: 17 user tools × 62 attacker instructions (InjecAgent). Threshold Precision Recall F1 Benign FPR 0.50 100% 89.8% 94.7% 0% 0.10 100% 100% 100% 0% By attack type (t=0.5): data-stealing 98.5% recall, direct-harm 80.6%; benign mean score 0.03. Gating: certainty ≥ 0.90 auto-handles 41.1% of traffic at 100% accuracy. Cost: $0.0213 for 1,105 calls; P50 0.30s. Synthetic tool outputs (injections wrapped inside WebFetch/Read/GitHub/Email results): at 0.5, P 97.1 / R 85.0 / FPR 4.9%. Misses are encoding bypasses (Cyrillic homoglyphs, code-snippet disguises). Cautionary dataset (neuralchemy, 942): the labels include "requests for harmful content," which my criteria did not ask about — recall drops to 55.1% at 0.5, purely a task-definition mismatch. Also note: the 0–0.1 score bucket still contains 16.6% malicious samples. Low score is not a safety guarantee. 3.2 Reranking (BEIR SciFact) 60 queries × 15 candidates; Jev scores each (query, document) pair 0–3, then reranks. Ranker Recall@5 MRR nDCG@5 Hit@1 BM25 74.2% 0.622 0.632 50.0% Jev 90.0% 0.843 0.848 78.3% Mean score: 2.38 for relevant vs 0.65 for irrelevant. $0.028 / 900 pairs, P50 0.31s. 3.3 Intent and catalog routing Dataset Classes Top-1 Gating SNIPS (1,400) 7 97.9% conf≥0.90: 93.6% coverage, 99.1% acc Banking77 (3,080) 77 80.3% conf≥0.90: 67.8% coverage, 92.8% acc MetaTool (199 tools) 199 96.5% (k=5, similar distractors) — MetaTool comparison: the paper reports 69.1% for ChatGPT on the same "similar choices" subtask (different exact setup — treat as magnitude reference). Errors cluster on near-duplicate tools: descriptions need explicit not_for boundaries. 3.4 Shell command risk gate Design: four orthogonal noul questions (destructive / touches secrets / exfiltrates / irreversible) + code that combines them into deny/review/allow. Variant 3-class agreement Dangerous caught Safe passed v1 (loose criteria) 66.2% 85.0% 87.3% v2 (tightened criteria) 78.5% 100% 98.2% Single-question choice baseline 77.7% 100% 85.5% v1 missed availability harms (fork bombs, reboot, firewall lockouts) until the criteria explicitly included them. The decomposition's payoff shows up in false positives: 14.5% → 1.8%. 3.5 Tool-relevance detection (BFCL) Task: given a request plus a function list, should any function be called? 1,140 samples. Version Method classic acc live-irrelevance acc FPR v1 single noul "is it relevant" 84.2% 59.5% 35.2% v2 criteria tightened to "purpose match" 89.6% 67.3% 27.9% v2c composed: purpose × (1 − incidental) 90.8% 77.0% 20.1% Typical failure mode: "technically possible" mistaken for "semantically intended" (a generic requests.get judged relevant). 3.6 Skill routing: 60% → 76% Goal: stop injecting the entire skill catalog into the main model. Route each request with one fast judgment, load only the winner. SkillRet (6,006 skills, 300 queries) — isolating selector from retrieval: Setup BM25 Hit@1 shortlist recall end-to-end conditional Hit@1* k=10, name+description 52.7% 71.3% 63.7% 89.3% k=20, + body in criteria 52.7% 77.3% 65.3% 84.5% k=10, body in BM25 index 58.3% 78.7% 68.3% 86.9% * conditional = gold in the shortlist. The selector is strong (85–89%); the bottleneck is retrieval. Adding bodies to the BM25 index helped retrieval; adding them to Jev's criteria did not. SkillRetBench (501 skills, official baselines, macros over 5 settings): Metric BM25 NaiveLLM Jev (chunked) Jev (hybrid) Recall@1 38.0% 30.2% 60.4% 75.8% Recall@10 59.8% 55.6% 87.6% 93.0% nDCG@10 53.4% 45.1% 60.3% 70.2% The architecture iteration is the interesting part (multi-skill composition R@1): Setting v1 chunked (one pick per chunk) v2 multiselect (per-candidate noul) v3 hybrid multi-skill composition 9.0% 15.0% 81.0% overall macro R@1 60.4% — 75.8% v2 failed because per-candidate noul scores have no competition: everything clusters around 0.5 (top-10 mean 0.56) and the merged ranking is noise. v3 fixes it: chunked choice produces 15 candidates with real probability spread, then one request with 15 noul questions verifies set membership. Compete first, verify second. Cost: hybrid $0.650 / 500 queries (P50 2.07s); the production-shaped "BM25 top-50 + Jev" is $0.072 / 500 queries at P50 0.33s (R@1 54.8%). 3.7 Negative results Model-difficulty routing (RouterBench, 825 samples): routing decision accuracy 51.3% — no signal; scores skew toward "a small model can handle it" (median 0.11). Chinese subset 14.6% vs English 57.7%. Predicting which model will fail is not a natural-language property. Trajectory failure attribution (Who&When, 184 trajectories × 8 steps): step-level AUROC 0.560, top-1 localization 16.3% (random 12.5%). The label requires "never corrected" — cross-step causality, outside a System One model's shape. 4. Versus published numbers On identical candidate lists (SkillRouter paper, ~80K pool, top-20): System Hit@1 GPT-4o-mini (listwise judge) 67.3% GPT-5.4-mini (listwise judge) 66.0% Qwen3-Reranker-8B 71.4% SkillRouter 1.2B (fine-tuned) 74.0% R3 (fine-tuned, bilingual) 77.1% Jev (zero-shot, conditional) 86.9–89.3% The paper's own conclusion: LLM-as-judge baselines are "not competitive." Letting a general LLM rank directly loses to purpose-built rerankers. Jev, zero-shot and untrained, sits in the same band as fine-tuned rerankers under the conditional metric (different pool sizes and metrics — read as order-of-magnitude, not a controlled comparison). One more finding from the SkillRet paper worth remembering: off-the-shelf rerankers can hurt when the first-stage retriever is strong (nDCG@10 dropped 83.5 → 74–78 there). If you bolt Jev onto a strong embedding retriever, re-measure the marginal gain. 5. Six engineering rules Criteria are the decision boundary. Merge "pressure" and "legitimate escalation" into one question and you get a mushy 0.47; split them and you get 0.04 / 0.96. Orthogonal decomposition + code composition beats one clever prompt. Shell gate false positives 14.5% → 1.8%; tool relevance 59.5% → 77.0%. High scores are reliable; low scores are not safe. In injection data, everything ≥0.1 was 90%+ malicious — but the 0–0.1 bucket still contained 16.6% malicious. Keep a human-review band. Confidence ≠ correctness. High confidence means the model is sure it applied your definition — including your mistakes (I saw wrong routings at confidence 1.0). Multi-label: compete, then verify (section 3.6). Respect the capability boundary. Only ask about locally observable patterns in text — not model capability (51%), not cross-step causality (AUROC 0.56). 6. Getting started Two integrations ship with this work: MCP server (three tools: scan_injection, bash_risk, rank_candidates): uvx jev-mcp # PyPI; or: uvx --from git+https://github.com/Aitejiu/jev-harness-lab jev-mcp Enter fullscreen mode Exit fullscreen mode Agent skill (skill routing, load only the winner): npx skills add Aitejiu/jev-harness-lab --skill jev-skill-router Enter fullscreen mode Exit fullscreen mode Both need TYPESAFE_API_KEY. 7. Reproducing git clone https://github.com/Aitejiu/jev-harness-lab cd jev-harness-lab uv venv --python 3.12 .venv uv pip install -r requirements.txt echo "TYPESAFE_API_KEY=" > .env .venv/bin/python eval/run_injecagent.py --concurrency 6 # injection .venv/bin/python eval/run_rerank.py --queries 60 # reranking .venv/bin/python eval/run_bash.py --variant v2 # shell gate .venv/bin/python eval/run_skillretbench.py --variant hybrid --per-setting 100 Enter fullscreen mode Exit fullscreen mode All raw results are committed under eval/results/; add --report-only to regenerate reports from cache. Datasets are public and downloaded separately per eval/README.md. 8. Limitations Single model version (jev-1.13.0); re-run after upgrades. Some datasets are samples or hand-built (130 shell commands, 121 synthetic injections, 60 SciFact queries) — selection bias possible. Thresholds are reference values; calibrate on your own data (use the coverage–accuracy curve). Cost estimates are input-token only. English-primary; CJE tasks need dedicated validation. SkillRetBench's NaiveLLM/SADO baselines are simulated per the dataset authors. If you build agent infrastructure and want to swap notes on decision models in the harness, the repo issues are open — or find me on X. The 22,500 calls cost $2.19; the lessons were cheaper than the tokens.