Reading Between the Lines: Can LLMs Discover the Question Behind the Text?
Claudiu Creanga, Claudiu Creanga, Liviu P. Dinu, Liviu P. Dinu
cs.CL
Quick verdict
'Question archaeology' (infer a text's genesis question) on a 335-text benchmark; Gemini beats humans, but headline metrics and stats are under-specified.
Summary
The paper introduces 'question archaeology': inferring the single authentic 'genesis question' that motivated a text, distinct from question generation (any plausible question) and utterance-level discourse frameworks. The authors commissioned 335 hand-written texts from 55 classroom participants who recorded their research question before writing and supplied two plausible distractors, yielding a contamination-free benchmark. Evaluation covers two tasks: free-form question prediction and three-way multiple choice, benchmarked against 553 human annotations (Fleiss' Kappa 0.36). Gemini Pro 2.5 leads both tasks (0.75 and 0.76 F1) versus humans (0.66 and 0.60), with a clear progression across Gemini versions (Flash 2.0: 0.68/0.65); Mistral 7B and Qwen 2.5 14B trail on task 2 (0.57/0.59), and fine-tuned MLMs score only 0.47-0.52 versus an XGBoost baseline of 0.35. Confidence scores correlate weakly with accuracy (r=0.25), and summarization preprocessing hurt performance. Main limitations: small skewed dataset (evaluative n=8, comparative n=1), heavy reliance on one proprietary model family, ambiguous task-1 metric (table says F1 Macro, narrative reports semantic similarity), no significance tests for the headline human-LLM comparison, and non-native-English human raters. Code and data are promised public.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- 'Question archaeology' - inferring the single authentic genesis question behind a complete text - is a distinct evaluation task, separable from question generation, reverse prompting, and QUD-style discourse modeling.
- A 335-text dataset of freshly commissioned, hand-written texts with pre-registered questions avoids pre-training contamination and is released publicly.
- Gemini performance progresses monotonically across versions on both tasks, from Flash 2.0 (0.68/0.65) to Pro 2.5 (0.75/0.76).
- Current LLMs outperform human raters on both tasks, suggesting advanced modeling of authorial intent.
- Fine-tuned masked language models perform poorly (0.47-0.52), indicating masked-word pre-training is ill-suited to recovering global authorial purpose.
Strengths
Contamination-free dataset with recorded provenance
Data Collection section: 335 texts hand-written in classroom by 55 participants, with the genesis question logged before writing and two author-crafted distractors per text; this design blocks memorization-based shortcuts that plague public-dataset benchmarks, and the data is promised public.
Distractor quality is quantitatively validated
Sentence-Transformer similarity analysis shows the true question is measurably closer to its text (0.66) than decoys (0.57/0.54), decoy distributions approximate normality, and a paired t-test (t=3.66, p<0.001) characterizes decoy asymmetry - evidence the three-way task is well-defined rather than arbitrary.
Baselines span multiple paradigms
Negative and calibration results reported openly
Clear task positioning
Weaknesses
Task-1 metric is internally inconsistent
Headline 'LLMs surpass humans' claim lacks statistical support
Model coverage is too narrow for the generalization claims
Dataset scale and skew undermine fine-grained conclusions
Presentation quality-control lapses
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'question archaeology' in NLP?
A proposed evaluation task where a model must infer the single original question that motivated a text's creation - not just any plausible question answerable from the text. It is tested two ways: predicting the question from the text alone, and picking the true question out of three candidates.
Can LLMs outperform humans at identifying the question behind a text?
On this paper's benchmark, yes: Gemini Pro 2.5 scored 0.75 and 0.76 on the two tasks versus 0.66 and 0.60 for human raters. Caveats: the gaps lack statistical significance testing, human raters were mostly non-native English speakers, and only the Gemini family was tested on both tasks.
Is the QuestionArchaeology dataset contaminated by LLM training data?
The authors argue no: all 335 texts were newly hand-commissioned in classroom settings, with each writer's motivating question recorded before the text was written, so neither texts nor question-text pairs existed publicly before evaluation.
Why did BERT-based models perform poorly on this task?
Fine-tuned MLMs (BERT-large, DeBERTa V3, XLM-RoBERTa, ModernBERT, and an ensemble) reached only 0.47-0.52 F1 on the multiple-choice task. The authors attribute this to masked-language pre-training emphasizing local context rather than global authorial intent, compounded by a small fine-tuning set (~134 texts in a 40-60 split).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Review is based on the extracted full text; figures (2, 5, 8-10) appear only as caption text and embedded axis fragments, so plotted distributions could not be visually verified.
- The referenced GitHub repository containing code and data was not inspectable, so 'codeAvailable' reflects the paper's stated promise rather than verified release.
- Figure 5's heatmap values and any appendix material beyond the extracted text were unavailable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.