Benchmarking Factual Robustness of LLMs via Multi-conversation Persuasion
cs.CL
Quick verdict
SAST-IR stateless-target benchmark shows 96% cold-start attack success on DeepSeek, but N=50, one model family, and no significance tests limit the claims.
Summary
The paper proposes SAST-IR (Stateful Attacker, Stateless Target – Iterative Refinement), a red-teaming framework that wipes the target LLM's memory every turn so each attack is a cold-start, single-turn persuasion attempt, eliminating what the authors call 'Refusal Inertia' — refusals propagating through a dialogue for consistency rather than genuine robustness. The attacker agent, CP-Agent, uses a Plan-Reflect-Optimize loop with 20 psychological persuasion patterns across 7 categories, diagnosing each refusal as soft (refine) or hard (re-plan). On CounterFact-Strict, a 50-question QA set derived from the CounterFact corpus, with DeepSeek-Chat serving as attacker, target, and judge and an 8-turn budget across five ablation groups, all groups exceed 90% persuasion success rate: the memory-less random baseline G1 reaches 96% (48/50) versus 90% (45/50) for the full hybrid agent G5. A judge-based quality analysis finds a 'Complexity Paradox': refined attacks elicit hedged compliance more often (21.4% for G5 vs 15.3% for G1), while simple diverse attacks achieve a higher share of genuine belief adoption (84.7% vs 78.6%). Key limitations, acknowledged by the authors: the small, relation-skewed dataset, evaluation of a single model family, and reliance on an LLM judge; no statistical significance testing is reported.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 3
● low 0
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Standard stateful multi-turn red-teaming suffers 'Refusal Inertia', where early refusals propagate for contextual consistency and mask true single-turn vulnerability.
- Under SAST-IR (stateless target), a memory-less random attacker achieves 96% PSR@8 (48/50) on CounterFact-Strict against DeepSeek-Chat, with all five ablation groups exceeding 90%.
- A 'Complexity Paradox' holds: iteratively refined attacks elicit more hedged Compliance (21.4% for G5 vs 15.3% for G1) while simple diverse attacks achieve a higher share of genuine belief adoption (84.7% vs 78.6%).
- Complex, refined attacks trigger 'Performative Alignment' — the model outputs the false answer under hypothetical framing while preserving internal alignment — so sophistication buys output compliance, not persuasion.
Strengths
Clean, unambiguous threat-model formalization
Sec 3.1 defines the asymmetric MDP (r_t = M_T(p_t)) with an implementation note confirming the target request contains only the current system prompt and attacker turn; Sec 5.1's turn protocol states explicitly that PSR@t counts subjects with a hit in turns 1..t where each turn is an independent conversation, removing history carryover by construction.
Candid, pre-emptive limitations section
Sec 7 acknowledges the N=50 relation-skewed subset, single model family (DeepSeek-Chat), and LLM-judge bias — matching what an external reviewer would flag, which is rare candor.
Mechanism-isolating ablation design
Success-criterion hygiene against false positives
Genuinely counterintuitive qualitative finding
Weaknesses
Scale and scope undermine the 'SOTA models' framing
No statistical treatment of group differences
Judge circularity on the core qualitative claim
The headline 96% is a best-of-8 figure
Refusal Inertia motivates but is never measured
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv:2609.16777, cs.CL, single author, HKUST) benchmarks the "cold-start" factual robustness of LLMs under persuasion attacks. Its three stated contributions (Sec 1) are: (1) the SAST-IR framework — a Stateful Attacker interacting with a Stateless Target, where the target's memory is wiped every turn (Eq 1-2) so each attack must succeed as a standalone prompt, removing what the authors call "Refusal Inertia"; (2) CP-Agent, a Plan-Reflect-Optimize attack agent with 20 psychological persuasion patterns (PAP) across 7 dimensions and a Reflector that diagnoses refusals as soft (→ Refinement) or hard (→ Re-planning); (3) the empirical finding of a "Complexity Paradox" — iteratively refined attacks raise Compliance rather than genuine belief adoption. The headline claim is that a memory-less random attacker reaches 96% persuasion success (48/50) on CounterFact-Strict against DeepSeek-Chat, exposing brittle memory-less defenses.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the SAST-IR framework in this paper?
SAST-IR (Stateful Attacker, Stateless Target – Iterative Refinement) is a red-teaming setup where the attacking agent keeps its memory across attempts while the target LLM is reset before every turn. Each attack must therefore work as a standalone single-turn prompt, measuring 'cold-start' persuasion robustness instead of multi-turn consistency effects.
What does the 96% success rate mean?
On 50 CounterFact-derived QA questions, a simple random attacker got the target model (DeepSeek-Chat) to output the false answer at least once within 8 independent attempts for 48 of 50 subjects (96%). Because each turn is a fresh conversation, this is a best-of-8 cumulative figure, not the success rate of a single prompt.
What is the 'Complexity Paradox' in this paper?
It is the finding that more sophisticated, iteratively refined persuasion attacks tend to make the model comply in a hedged, hypothetical framing rather than genuinely adopt the false belief. Simple, diverse attacks achieved a higher share of genuine persuasion (84.7% for the G1 baseline vs 78.6% for the full G5 agent) in the paper's LLM-judge analysis.
What is 'Refusal Inertia'?
The authors' term for the tendency of an LLM to repeat its earlier refusal in later dialogue turns mainly to stay consistent with its own conversation history, rather than because its safety filters are robust — which they argue masks single-prompt vulnerability in standard multi-turn red-teaming evaluations.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are referenced with captions but their charts are not analyzable from the extracted text; quantitative statements rely on the inline tables.
- Minor extraction artifacts: o_true/o_false appear garbled as 'otrue'/'of alse' in the source text; interpreted as the true/false answer fields throughout.
- The metadata abstract is truncated mid-sentence; the complete abstract in the full text was used.
- The linked GitHub repository was not inspected; code/data availability rests on the paper's statements.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.