IBBench-Light: A Paired Evaluation of Task-Conditioned Responses to External Directives
Kainan Zhou, Gangzhen Qian, Zhaoyi Li, Hang Xiao
cs.AI · cs.LO
Quick verdict
Paired execute/process benchmark showing what marginal averages hide; candid, well-audited, but 12-base design and unlogged historical runs limit generality.
Summary
IBBench-Light is a paired benchmark for instruction-data separation in LLMs. Twelve synthetic semantic bases (token emission, arithmetic, transformation, selection) are each expanded into twelve records via four source wrappers and three embedding forms; each record is tested twice against byte-identical context — once to execute an embedded benign directive, once to report a stored reference — yielding 144 matched pairs (288 prompts) per model. Four quantized instruction models (Qwen3-4B, Mistral-7B, SmolLM2-1.7B, Phi-4-mini) produced 1,152 archived greedy responses scored with a case-insensitive exact contract; paired exact-contract accuracy (PECA) counts records succeeding in both conditions. Qwen scores 91.7% (execute) and 75.7% (process) but only 67.4% paired, showing what marginal averages omit. Logged CPU follow-ups (1,722 generations) add directive-absent controls (SmolLM2 succeeds 36/36 on direct reference extraction), twelve new bases, wording changes, and stopping tests: switching Phi-4-mini's EOS set moves paired success from 0/144 to 62/144. A bounded 30-item IHEval comparison locates interface differences without score conversion. Limitations: archived runs omit model revisions and stop events; only 12 designed bases (wide resampling intervals); 50% of records expose the literal target; no human adjudication, training study, or adaptive attacks; new experiments cover only SmolLM2 and Phi on CPU.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Marginal execute/process accuracies systematically overstate conditional task success; PECA over matched records reveals the gap (Qwen: 83.7% average marginal vs 67.4% joint).
- Generation stopping policy is a first-order experimental variable: switching Phi-4-mini's EOS set changes exact paired success from 0/144 to 62/144 in the pinned BF16 rerun.
- Half of the records (72/144) contain the execute target literally under the scorer's case-insensitivity, so exact scores partly admit copying rather than computation.
- Directive-absent controls separate extraction from operation skill: SmolLM2 returns the reference on 36/36 direct prompts but only 11/36 when the same operation is embedded in a record.
- The benchmark measures conditional task and output-contract success within a single-user-turn serialization, and does not establish privilege-boundary robustness or operational security.
Strengths
Paired design exposes the marginal-vs-joint gap
Byte-identical records probed under execute and process conditions; Qwen's 132/144 execute and 109/144 process exact successes collapse to 97/144 complete pairs (Section VI-A), with the Frechet-style bounds of Eq. (3) stated explicitly.
Self-auditing of scoring artifacts
Section IV-B quantifies literal-target exposure (60/144 case-sensitive, 72/144 case-insensitive, Table II); Section VI-A's case-sensitive recheck moves SmolLM2 execute success 28 to 20 without changing its paired count of nine.
EOS stopping finding backed by token-level evidence
Explicit threat-model scoping
Reproducibility hygiene for new runs
Weaknesses
Historical environment under-logged
Tiny designed sample yields weak inference
Confounded factor assignments
Exact string scoring with post-hoc relaxations
Quantized archived panel and partial new coverage
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
IBBench-Light asks whether instruction-tuned LLMs condition behavior on the user's request when an unchanged external record contains a benign directive. Each of 144 records is probed twice with byte-identical context: an execute request (apply the directive, return its deterministic result) and a process request (report a stored reference). The headline metric, paired exact-contract accuracy (PECA, Eq. 2), counts records passing a case-insensitive exact string contract under both conditions. The authors claim: (i) marginal accuracies misrepresent conditional success - Qwen3-4B posts 91.7%/75.7% execute/process but only 67.4% PECA; (ii) scoring and stopping protocols are first-order variables - literal targets occur in 72/144 records under case-insensitive matching (Table II), and switching Phi-4-mini's EOS set moves paired success from 0/144 to 62/144 (Section VI-D); (iii) the benchmark measures conditional task and output-contract success within a single-user-turn serialization, explicitly not privilege robustness or operational security (Section IV-C). Claims and scope are unusually well matched; the candor is a distinguishing feature of the paper.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PECA in IBBench-Light?
Paired exact-contract accuracy: the fraction of the 144 record pairs where a model's response satisfies the output contract under both task conditions for the same byte-identical record - executing the embedded directive AND reporting the stored reference. Qwen3-4B reaches 67.4% PECA despite 91.7% (execute) and 75.7% (process) marginal accuracies.
Why did Phi-4-mini score 0/144 paired in the original run?
The archived run used only the tokenizer's EOS token (199999) for stopping, but Phi's configuration also declares token 200020 as an end-of-sequence token. Under tokenizer-only EOS, all 288 responses continued past token 200020, hit the 32-token cap, and failed the exact contract. Restoring the two-token model EOS set in a pinned CPU rerun raised paired success to 62/144.
Does IBBench-Light test real prompt-injection attacks?
No. Directives are benign and deterministic, prompts are fixed (no adaptive attacks), everything sits in a single user turn, and nothing is executed. The paper measures conditional task and output-contract success at the response-string interface, not operational security, privilege boundaries, or defense effectiveness.
Which models were evaluated?
Four quantized instruction models spanning 1.7B-7B parameters: Qwen3-4B-Instruct-2507, Mistral-7B-Instruct-v0.3, SmolLM2-1.7B-Instruct, and Phi-4-mini-instruct. The logged CPU follow-ups cover only SmolLM2 and Phi at BF16 precision with pinned repository commits.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Supplementary Appendices S1-S10 and Tables S2/S4 are cited but are separate from the provided main text; their contents (leading-target rule, six model contrasts, control prompts) could not be checked against the extraction.
- No artifact URL or repository link appears in the provided text; public availability of the claimed artifact (raw outputs, code, checksums) is unconfirmed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.