JevOut: Natural Context Can Flip Decision Models
cs.CL
Quick verdict
Claims short natural context flips decision models (TFR 61-73%); rigorous adaptive attack study, though white-box access and auto answer checks temper it.
Summary
The paper shows that short, fluent context additions can redirect 'decision models' - systems like Jev that map language to probability distributions over finite choices - from a correct answer to a wrong option fixed in advance. Using probability-guided context optimization, a Gemma4-12B proposer inserts one to four sentences (median 31 added words) while a target-blind checker enforces that question, options, and gold answer remain intact; a Gibbs-style archive allocation concentrates search using log-probability margins toward the fixed target. Within 64 accepted target evaluations, the optimizer flips 312/508 (61.4%) of Jev's initially correct decisions across seven benchmarks (MMLU-Pro, SuperGPQA, MuSR, ToMBench, LAR-ECHR, SATA-Bench, BFCL V4), with 229 flips reaching 0.7 or higher probability on the wrong option; OpenSourceJev, Von, and a plain Qwen scorer show 64.9-73.2%. Probability feedback beats label-only allocation by 7.1 points at matched budget, contexts transfer across models (21.1-47.8%), and LoRA-trained proposers improve direct generation (one-generation TFR 16.1% to 21.9%). Limitations: the attack assumes white-box probability access and 64 adaptive queries per decision; answer preservation is verified only by an automated checker with no human audit; headline rates keep rising with budget and are protocol-specific; the primary target is a proprietary hosted model; per-dataset denominators are small.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Short, fluent context additions that preserve the question, options, and gold answer can redirect dedicated decision models to a wrong option fixed in advance, with TFR of 61.4% on Jev and 64.9-73.2% on three other systems within 64 accepted evaluations.
- Access to full option-probability distributions improves attack efficiency: probability-guided allocation beats label-only allocation by 7.1 percentage points at a matched 64-evaluation budget.
- Answer-preserving context additions transfer across decision models (21.1-47.8% targeted transfer over 12 directed pairs), strongest between implementations sharing Qwen3-1.7B weights.
- Proposers trained on development trajectories improve direct context generation on held-out items, raising one-generation TFR from 16.1% to 21.9% (V2).
- Structured probability outputs over finite options do not by themselves make model decisions dependable as routing or action interfaces.
Strengths
Directed, constraint-respecting attack design
The wrong target is fixed before any construction (Eq. 1), only context is added while source, question, options, and gold answer stay byte-identical, and the neutral vs target-aware one-shot vs optimized contrast (2.2% / 16.9% / 61.4% TFR on Jev, Table 1) cleanly separates target knowledge from iterative search effects.
Matched-budget causal ablation of the probability signal
The full/probability-only/label-only comparison on the same 140 Jev decisions (Sec. 4.3, Table 12) isolates feedback granularity with a paired, dataset-stratified bootstrap: 63.6% vs 56.4%, +7.1 points, 95% CI 2.1-12.9, with numerical history and allocation effects separated.
Exceptional experimental accounting and reproducibility record
Interface-isolation and transfer design
Honest reporting of mixed and negative results
Weaknesses
White-box, query-heavy threat model
Answer preservation verified only by an automated checker
Narrow target class with a proprietary primary target
Budget-dependent, non-saturated headline rates
Small per-task denominators without multiplicity control
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is an empirical robustness study of 'dedicated decision models' - systems that consume unstructured text and emit a probability distribution over a finite, typed option set. The primary target is Jev 1.13.0 (a hosted commercial model), joined by OpenSourceJev (a Jev-style interface over frozen Qwen3-1.7B Q8 weights), Von 1.0 (a 395M non-autoregressive decision model), and a plain Qwen scorer on the same Qwen weights. Across seven benchmarks (MMLU-Pro, SuperGPQA, MuSR, ToMBench, LAR-ECHR, SATA-Bench, BFCL V4; 100 held-out items each), the authors claim that short, fluent, answer-preserving context additions redirect initially correct decisions to a wrong option fixed before construction. Headline numbers: TFR 61.4% for Jev (312/508; Wilson 57.1-65.5%), then 72.6%/73.2%/64.9% for the other three systems; 45.1% of Jev flips put probability at least 0.7 on the wrong option. The framing claim - that structured probability outputs do not make decisions dependable as routing/action interfaces - is reasonable and honestly budget-qualified throughout ('within 64 accepted target evaluations' appears in nearly every quantitative statement).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the JevOut paper about?
It shows that one or two short, ordinary-sounding background sentences - which do not change the question, the options, or the correct answer - can make specialized 'decision models' (such as Jev, which outputs probabilities over finite choices) switch from a correct decision to a specific wrong option. Across seven benchmarks, such additions were found for 61.4% of Jev's initially correct decisions, often with over 0.7 probability assigned to the wrong choice.
What is Targeted Flip Rate (TFR)?
TFR is the percentage of decisions a model initially answers correctly for which the authors' optimizer found an answer-preserving context addition that makes the model select one specific wrong option fixed in advance, within a budget of at most 64 accepted evaluations per decision. In this paper TFR was 61.4% for Jev and 64.9-73.2% for three other decision systems.
How long are the context additions that flip these models?
They are short: 81.1% of successful Jev contexts used at most two added sentences, and the median total added text was 31 words. A single target-aware sentence already flips 16.9% of Jev's initially correct decisions, versus 61.4% with full iterative optimization.
Do these adversarial contexts transfer to other models?
Yes. Contexts optimized against one system were frozen and tested on others, producing targeted flips on 21.1-47.8% of matched decisions across 12 directed model pairs; transfer was strongest (44.2% and 47.8%) between two implementations sharing the same Qwen3-1.7B weights.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 are assessed only through their embedded caption text; heatmap cells and curves could not be inspected visually.
- arXiv v1 preprint (24 Sep 2026), apparently not yet peer-reviewed; external artifacts (code repository, homepage, hosted Jev API) could not be verified from the text.
- The 'Jev', 'Von', 'OpenSourceJev' and 'Gemma 4' systems are recent and little documented outside this paper; their characterization could not be independently checked.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.