One Prompt Does Not Fit All: Self-Meta-Evolve for Personalized Information Extraction
Hongliang Li, Lu Wang, Yong Xu, Hanyang Chen, Zhitao Hou, Xiaoting Qin, Song Ge, Qingwei Lin, et al.
cs.AI
Quick verdict
Claims +13.56pp SR over ProTeGi via per-user prompt evolution on a 292-persona synthetic benchmark; broad evidence but fully LLM-judged, code deferred.
Summary
The paper reformulates enterprise information extraction as per-user prompt adaptation: instead of one globally optimized prompt, each user owns a structured extraction prompt refined by persona-conditioned feedback. Self-Meta-Evolve runs a dual loop — an inner loop where an LLM editing agent patches section-level point_ids of a user's prompt from an 'AI User' critique (weighted revise/delete counts), and an outer loop that rewrites the meta-prompt from a buffer of successful edit traces. The authors also build a benchmark of 292 LLM-generated enterprise personas seeded from O*NET occupations, with 8-12 synthetic documents each, split 60/20/20, plus STEM and Humanities test sets (N=50 each). On the 59-persona held-out set with GPT-5.1, the method reaches 74.58% success rate versus 61.02% for the strongest baseline ProTeGi (paired bootstrap p=0.0038) and hits 52.54% within two iterations; gains hold across five backbones (+11.87 to +15.25pp) and the STEM-Humanities degradation shrinks to 4.0 points. Ablations attribute most gains to the outer loop (SR 74.6 to 57.6 without it). Double-blind studies with 20 professionals show a 71% win rate against a static prompt, and synthetic personas rate as realistic as real ones. Main caveats: the entire training-and-evaluation loop is LLM-simulated (documents, preferences, feedback), the efficiency claim in Section 6.3 conflicts with Table 3, and code/data release is deferred to acceptance.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Enterprise information extraction should be formulated as per-user prompt adaptation under persona-conditioned interaction feedback, replacing the single global prompt optimization paradigm.
- Self-Meta-Evolve's dual loop (per-user structured prompt editing plus cross-user meta-prompt evolution) reaches 74.58% success rate on a 59-persona held-out set with GPT-5.1, +13.56 points over ProTeGi (paired bootstrap p=0.0038), and 52.54% within two iterations.
- The outer meta-evolution loop is the principal driver: removing it drops success rate from 74.6 to 57.6, and the evolved meta-prompt cuts the STEM-to-Humanities degradation to 4.0 points versus 8.0-16.0 for baselines.
- The 292-persona O*NET-grounded synthetic benchmark is a valid proxy for real workplaces: synthetic personas rate as realistic as real ones (all dimension gaps at most 0.06, n.s., kappa=0.71), and 20 real professionals prefer the adapted prompts in 71% of 200 double-blind pairwise comparisons against a static prompt.
Strengths
Clear problem reframing with usable formalization
Section 3 recasts enterprise IE as per-user prompt adaptation (Eqs. 1-3) with a concrete scalar loss (lambda_d = 2*lambda_r from a 5-user, 30-case Likert pilot) and a uniformly applied success threshold calibrated to the strongest baseline's dev median.
Unusually transparent synthetic benchmark
Section 4.1 documents the full 412-to-400-to-292 persona reduction with per-filter counts (dedup -47, coherence -38, specificity -23), O*NET v28.0 seeds, and a leakage control: the document generator is conditioned on role/domain only, not the preference profile.
Broad, internally consistent empirical matrix
Human validation beyond the synthetic loop
Honest negative controls and failure accounting
Weaknesses
Fully LLM-simulated evaluation loop
Efficiency claim contradicted by own table
Reproducibility deferred
Internal configuration inconsistencies
Thin statistical reporting around small sets
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets enterprise information extraction (IE) where 'correct' output is role-relative: a project manager wants deadlines and owners from the same email a financial analyst reads for budget entities. It makes three claims: (1) enterprise IE should be formalized as per-user prompt adaptation under interaction feedback (Section 3, Eqs. 1-3); (2) Self-Meta-Evolve — an inner loop that patches section-level point_ids of a structured prompt from an 'AI User' critique, plus an outer loop that rewrites the meta-prompt from a buffer of successful edit traces — beats seven prompt-optimization baselines; (3) a 292-persona synthetic benchmark grounded in O*NET is a faithful proxy for real workplace heterogeneity. Headline numbers: 74.58% success rate (SR) versus 61.02% for ProTeGi on the 59-persona held-out set with GPT-5.1 (paired bootstrap p=0.0038), SR@2 = 52.54%, and a 71% win rate across 200 double-blind pairwise judgments by 20 professionals against a static prompt.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Self-Meta-Evolve?
A framework that gives each enterprise user their own LLM extraction prompt instead of one shared prompt. An inner loop edits the user's structured prompt based on critiques from a simulated 'AI User'; an outer loop distills successful editing patterns across users into an improved meta-prompt. It reports a 74.58% success rate, 13.56 points above the strongest baseline ProTeGi, on a 59-persona held-out test set using GPT-5.1.
Is the benchmark based on real enterprise data?
No. All 292 user personas are generated by GPT-5.1 from public US O*NET job descriptions, and each persona's 8-12 documents (emails, chats, reports) are LLM-synthesized, so no real or private data is used. Validation includes a double-blind realism comparison against real professional profiles and a pairwise study with 20 real professionals. The authors promise to release the data under CC-BY 4.0 only upon paper acceptance.
How is 'success rate' defined in this paper?
A persona's run succeeds if its deployed loss — a weighted count of revise and delete feedback items (delete weighted twice as heavily as revise, based on a small pilot) — falls at or below a threshold tau = 0.75, which is set to the median loss of the strongest baseline (ProTeGi) on a 50-persona development set. Success rate is the fraction of evaluation personas meeting this bar.
Does this method help when all users want the same extraction?
Barely. In a control experiment (Appendix I) where all 'users' share one generic extraction objective, the unoptimized initial prompt already scores 88.0% success and the full framework adds only 3.5 points (91.5%). The authors present this as consistent with the method's purpose: it targets heterogeneous, role-specific preferences, not shared objectives.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 and 9 are available only as embedded text; curve shapes and graphical values could not be independently verified, though all key numbers appear in tables.
- The extraction includes main text and Appendices A-I with no obvious truncation, but human-study raw materials and exact prompts (Appendix D is abridged) are unverifiable from text alone.
- This is arXiv v1 (September 2026); peer-review status is unknown and the promised artifact release has not yet occurred.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.