CIPHER: Benchmarking Cross-record Inference over Privacy-Hardened Evidence Records
Suparno Roy Chowdhury, Manan Roy Choudhury, Dhruv Madhwal, Vivek Gupta
cs.CR · cs.AI
Quick verdict
A well-built hybrid-record benchmark (3,427 questions, SQL provenance); findings credible but hinge on a lenient judge and a one-model restoration ablation.
Summary
CIPHER is a benchmark of 3,427 expert-validated questions over hybrid records that combine structured attributes with free-text narratives, drawn from Indian First Information Reports (1,571 questions), CFPB consumer complaints (1,356), and MIMIC-IV discharge notes (500). Every question carries an executable SQL specification, a reference answer, and supporting-record provenance, so gold answers can be regenerated and failures traced to specific stages. The authors evaluate five LLMs (Llama 3.3 70B, GPT-OSS 120B/20B, Gemini 2.0 Flash, Qwen 3 Next 80B) with prompting, retrieval, hybrid symbolic-neural (WEAVER, BlendSQL), and table-specialist baselines under redacted versus surrogate-restored evidence. Performance is uniformly weak: the median M3 across 135 model-method-domain configurations is 26.5, none reaches 50, and retrieval baselines collapse to roughly 2.8. Failure analysis attributes most errors to record selection and predicate interpretation rather than arithmetic. Restoration effects are non-uniform - mean delta-M3 of +7.6 (FIR), +3.7 (MIMIC), and -5.5 (CFPB) - but this comparison uses only Llama 3.3 70B. Strengths include high annotation agreement (kappa 0.93-0.96; judge-human kappa 0.91) and honest framing of surrogates as non-private. Main limitations: a deliberately lenient LLM judge, no significance testing, a retrieval pool capped at ten candidates with no recall reported, and no reasoning-model baselines.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Providing all supporting records does not make hybrid reasoning reliable: the median M3 across 135 model-method-domain configurations is 26.5 and none reaches 50.
- Most failures arise from evidence (record) selection and narrative predicate interpretation rather than arithmetic execution - e.g., 72.5% of failed aggregation questions are dominated by evidence-selection errors.
- Privacy transformations have non-uniform, method- and domain-dependent effects: surrogate restoration raises mean M3 by +7.6 on FIR and +3.7 on MIMIC but lowers it by 5.5 on CFPB (measured on Llama 3.3 70B only).
- CIPHER is presented as the first resource combining institutionally redacted narratives with structured attributes across three domains, paired with executable SQL supervision and supporting-record provenance.
Strengths
Executable supervision with full provenance
Every question pairs a SQL specification, a regenerable reference answer, and supporting-record IDs (Section 3.1), so golds can be re-executed and failures attributed to selection vs. interpretation vs. execution - the design the RQ3 taxonomy (Figure 6) directly exploits.
Unusually rigorous validation chain
Three-reviewer template audit with Cohen's kappa 0.93-0.96 (Table 2, Section 4.3); the M2 judge validated against 750 human decisions at kappa 0.91 overall (Table 7); all 895 surrogate replacements dual-reviewed at 84.4-94.6% agreement (Table 8, Appendix E).
Actionable failure diagnosis
Honest privacy framing
Complete metric transparency
Weaknesses
Lenient judging makes every headline number judge-relative
Retrieval baseline is structurally handicapped and recall is never reported
The restoration analysis rests on a single backbone
No uncertainty quantification
Failure taxonomy lacks reported agreement and author independence
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CIPHER (arXiv 2609.07022) is a benchmark paper: 3,427 expert-validated questions over hybrid records that pair structured attributes with free-text narratives, from three sensitive domains - 1,571 questions on 322 OCR'd Indian First Information Reports (masked by the authors with Presidio), 1,356 on 400 CFPB complaints, and 500 on MIMIC-IV discharge notes. Every question carries an executable SQL specification, a regenerable reference answer, and supporting-record identifiers (Section 3.1). Four headline claims: (1) systems fail even with all supporting evidence in context (median M3 26.5 over 135 configurations, max 48.6, none at 50); (2) failures concentrate in record selection and predicate interpretation, not arithmetic; (3) privacy transformations have non-uniform effects (+7.6 FIR / +3.7 MIMIC / -5.5 CFPB mean delta-M3, Llama 3.3 70B only); (4) CIPHER is the first resource combining institutionally redacted narratives with structured attributes across domains with executable supervision. Claims 1-2 are well supported by the presented data; claim 3 is supported but under-powered; claim 4 is plausible but not independently verified here.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the CIPHER benchmark?
CIPHER is a dataset of 3,427 expert-validated questions that require reasoning over hybrid records combining structured fields (dates, categories, numbers) with free-text narratives, drawn from three sensitive sources: Indian police First Information Reports, US CFPB consumer complaints, and MIMIC-IV hospital discharge notes. Each question ships with executable SQL that regenerates the gold answer and identifies the supporting records, enabling failures to be traced to specific stages rather than just scored.
Why do LLMs fail on CIPHER?
The paper's analysis shows most failures come from picking the wrong records and misreading conditions expressed inside narratives, not from arithmetic. For example, 72.5% of failed aggregation questions and 73.8% of failed ranking questions are dominated by evidence-selection errors, and this holds even when all required records are placed directly in the model's context. Across 135 tested configurations the best combined score never reaches 50%.
Does de-identification redaction hurt LLM accuracy on documents?
Not uniformly. Comparing redacted records against versions where masked spans were filled with plausible surrogate values, the tested model (Llama 3.3 70B) improved on police reports (+7.6 mean M3) and clinical notes (+3.7) but got worse on consumer complaints (-5.5). Restored text can remove distracting gaps or, conversely, inject misleading entities, and the effect depends heavily on the prompting method used.
Is the CIPHER dataset publicly available?
The authors commit to releasing the benchmark, templates, SQL supervision, prompts, transformation metadata, and evaluation code under a research-only, non-commercial license, subject to source-data licenses. The CFPB and FIR source records are public, but the MIMIC-IV portion remains gated behind PhysioNet credentialing, so full replication requires those credentials.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 4 and Figure 6 cell values are extracted as jumbled number streams out of visual order; all figures cited in this review were cross-checked against the prose and Tables 3-6 rather than the figure layouts.
- The judge prompt in Appendix A.6 ends mid-sentence at 'Discrepancies between counts is too hig' where a page break occurs - a PDF extraction artifact; the tail of the judge's 'no' criteria list may not be visible, though the 'yes' criteria and the leniency instruction are fully present.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.