From Tables to Quantified Statements: Evaluating LLM Inference Generation through Executable Verification
Mai Mohamed Eida, Gunjan Anand, Ayush Singh, Aleksandre Maskharashvili
cs.CL
Quick verdict
Sound idea: executable Python checkers verify LLM quantified table claims; honest human audit, but synthetic data, confounded models, no significance tests.
Summary
The paper introduces STAT-TO-TEXT, a controlled table-to-text task in which LLMs generate quantified natural-language inferences (all, some, no, most, conditionals) from statistical tables, and an LLM-generated Python checker verifies each statement by executing pandas code against the source table. One hundred synthetic tables (15 rows, 5-10 variables) were produced by GPT-5.5; four open-weight models (LLaMA 3.1 8B/70B, GPT-OSS 20B/120B) generated 20 inferences per table under identical prompts. After verification, GPT-OSS-120B leads on accuracy (93.8%), grounding (99.5%), and coverage (100% row coverage, 95.6% variable coverage), while LLaMA-3.1-8B achieves perfect grounding yet only 48.7% accuracy, showing grounding alone does not ensure logical validity. A human audit of 2,400 statements over 120 sampled tables finds 91.2% overall checker-human agreement (97.5% for GPT-OSS-120B), with false negatives (172) far outnumbering false positives (39), so automated scores are conservative lower bounds. The paper documents concrete checker failure modes, including majority claims treated as universal and verdicts contradicting their own justifications. Main limitations: tables are fully synthetic, model family, scale, and reasoning paradigm are confounded in the four-model design, pipeline throughput is uneven (78-95 tables per model), no significance testing is reported, and the accuracy metric depends entirely on the LLM-generated verifier being audited.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- GPT-OSS-120B generates the most faithful quantified inferences of the four tested open-weight models (93.8% post-checker accuracy, 99.5% grounding) while maintaining top table coverage (TRC 1.0, VC 0.956).
- Grounding alone does not ensure logical validity: LLaMA-3.1-8B achieves 100% grounding yet only 48.7% accuracy after checker verification.
- LLM-generated Python checkers align closely with human judgments (91.2% agreement overall on 2,400 annotations; 97.5% for GPT-OSS-120B), and because false negatives (172) dominate false positives (39), automated accuracy scores are conservative lower bounds.
- Model family and scale strongly influence quantified-statement faithfulness, while quantifier diversity appears to trade off against faithfulness for all models except GPT-OSS-120B.
Strengths
Executable verification is a principled evaluation instrument
Translating each inference into pandas checker code executed against the source table (Step 2, Listing 1) yields deterministic, table-grounded verdicts, sidestepping the unreliable learned/LLM metrics the authors motivate via Onderková et al. (2025). Withholding cell values from the checker prompt (§4 Step 2) is a sound anti-leakage design choice.
Honest, reasonably powered human audit with a useful error taxonomy
Table 3 audits 600 statements per model (2,400 total): overall checker-human agreement 91.2%, from 79.67% (LLaMA-8B) to 97.5% (GPT-OSS-120B). The 172 false negatives vs 39 false positives split is correctly interpreted as automated scores being conservative lower bounds. Appendix E documents four concrete checker failure modes with full examples.
Multi-dimensional metric suite separating faithfulness, coverage, and diversity
Controlled synthetic design isolates model effects from table variation
Weaknesses
Fully synthetic, small-scale data
Confounded design with no statistical inference
Unequal post-pipeline throughput biases cross-model comparison
Internal inconsistency and metric definition/formula mismatch
Abstract overstates the diversity result; human protocol underspecified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper defines STAT-TO-TEXT, a controlled table-to-text task in which LLMs generate quantified natural-language inferences (universal, existential, negative, majority, conditional) from statistical tables, and an LLM-generated Python checker verifies each statement by executing pandas code against the source table (§1, Fig. 1). The framing is grounded in generalized quantifier theory (Barwise & Cooper 1981, §2.2): most(R,S) requires |R∩S| > |R|/2, so faithfulness reduces to a checkable set relation over rows. Four claims are advanced: (i) the pipeline reliably generates and verifies quantified inferences (RQ1), (ii) model family, scale, and reasoning paradigm affect logical faithfulness, with GPT-OSS-120B best (RQ2), (iii) coverage is easy but diversity trades off against faithfulness (RQ3), and (iv) the automated checker aligns closely with human judgments. This is positioned against LogicNLG/Logic2Text (Chen et al. 2020a,b), which mix quantifier-like operations into a broader inventory, and against PAL (Gao et al. 2023), here repurposed from answering to evaluation.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is STAT-TO-TEXT?
A task and evaluation pipeline in which LLMs generate quantified natural-language statements (using all, some, no, most, or conditionals) from statistical tables, and a Python checker script written by an LLM is executed against the source table (via pandas) to verify whether each statement's truth conditions hold.
Which model performed best on STAT-TO-TEXT?
GPT-OSS-120B: after checker verification it scored 93.8% accuracy, 99.5% grounding, 100% row coverage, and 95.6% variable coverage, and its automated verdicts agreed with human annotators 97.5% of the time - the highest of the four open-weight models tested (LLaMA 3.1 8B/70B, GPT-OSS 20B/120B).
Can the LLM-generated Python checker be trusted?
Largely yes for stronger models. A human audit of 2,400 statements found 91.2% overall agreement between the checker and human labels (96.0-97.5% for the GPT-OSS models, 79.7% for LLaMA-8B), and errors skewed toward false negatives (172 vs 39 false positives), meaning reported accuracy likely underestimates true validity.
Was the evaluation run on real-world data?
No. All 100 tables are synthetic, generated by GPT-5.5 with 15 rows and 5-10 variables each across seven themes (healthcare, retail, logistics, finance, sports, agriculture, HR). Evaluating on real, larger, or noisier tables is listed as future work.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 2 (quantifier-category distributions) is referenced but its numeric values cannot be recovered from the extracted text, so distribution-skew claims rely on the prose alone.
- The extraction duplicates Table 2 as Table 5 (before/after checker split); minor layout artifacts suggest slight extraction noise, but all headline numbers appear consistently twice.
- Human annotation protocol details (annotator count, IAA) are absent from the text itself, limiting assessment of audit quality beyond the reported agreement rates.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.