LLMs as Feature Engineers for Text-and-Tabular Prediction
cs.LG
Quick verdict
Error-guided LLM loop turns text into interpretable categorical features for tabular models; AUC gains are modest yet consistent; evaluation may be optimistic.
Summary
This method paper introduces an iterative agentic framework that turns unstructured text into interpretable, schema-bound categorical features for tabular prediction. A generator LLM (GPT-5.4) proposes feature definitions — name, finite value set, description — in batches of ~90; a cheaper extractor LLM (GPT-4.1-nano) applies them zero-shot row-by-row; a HistGradientBoostingClassifier scores predictive utility. The search is steered by translating model errors into natural-language feedback: for AUC, sampled misranked positive/negative pairs, with well-ranked pairs as negative constraints. On three public datasets (Kickstarter 80K, Amazon Books 80K, Stack Overflow 60K rows), the top-10 selected LLM features lift 5-fold CV AUC over the tabular baseline (0.7444→0.7639, 0.7611→0.8366, 0.8544→0.9320), and combining LLM features with TF-IDF and MiniLM embeddings is best everywhere (0.7692/0.8510/0.9700); notably, LLM features alone trail embeddings on Stack Overflow. A controlled ablation shows error feedback reaches the no-feedback variant's 15-iteration peak within 3–9 iterations. SHAP analysis finds LLM features top every per-dataset ranking (e.g., mean |SHAP| 2.25 vs 0.09 for goal_usd on Kickstarter). A production CTR deployment reports +5–6.6% revenue/margin lifts at 30–50% higher serving latency. Key limitations: proprietary models and no code release, random stratified CV despite temporal features, feature selection and evaluation sharing folds, no reported p-values or intervals, and no fine-tuned-transformer text baseline.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
4 strengths · 0 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Translating downstream model errors (AUC ranking inversions) into natural-language feedback lets the LLM reach the no-feedback baseline's 15-iteration peak in 3–9 iterations across three datasets (Table 3).
- LLM-generated schema-bound categorical features are complementary to TF-IDF and dense embeddings: the three-view combination achieves the highest 5-fold CV AUC on all three datasets (Table 2).
- Discovered features dominate SHAP importance rankings on every dataset, topping each list and enabling per-prediction semantic audit trails (Sec. 4.4, Table 4).
- In two production CTR models the features delivered roughly +5–6.6% revenue/margin lifts at the cost of 30–50% higher inference latency (Sec. 5).
Strengths
Well-isolated core ablation
Table 3 holds the generator model, the 90-feature budget and the 15-iteration cap constant, isolating the feedback signal; reaching the no-feedback baseline's peak at iterations 3/4/9 (Amazon/Kickstarter/Stack Overflow) is the paper's most convincing quantitative result.
Honest reporting of a negative single-view result
Table 2 shows LLM features alone losing to MiniLM embeddings on Stack Overflow (0.9320 vs 0.9510); the complementarity framing is built around this rather than hiding it, and the text-baseline comparisons are fully enumerated.
Concrete interpretability artifacts
Deployment costs reported, not just wins
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper introduces an iterative, agentic loop that converts unstructured text into schema-bound categorical features for tabular prediction. A generator LLM (GPT-5.4) proposes ~90 feature definitions per iteration — a name, a finite value set, and a description; a cheaper extractor LLM (GPT-4.1-nano) applies them zero-shot row-by-row; a HistGradientBoostingClassifier scores predictive utility. The claimed novelty (Table 1) is twofold: prior LLM feature-engineering work (CAAFE, LLM-FE) operates only on tabular columns, and prior LLM optimizers (OPRO) use scalar feedback — whereas this system targets raw text and feeds back explicit model errors (sampled AUC ranking inversions, with well-ranked pairs as negative constraints, Sec. 3.4). Four headline claims: (1) error-driven feedback accelerates discovery (Table 3); (2) LLM features are strictly complementary to TF-IDF and MiniLM embeddings, the three-view combination being best on all datasets (Table 2); (3) the features dominate SHAP importance and give per-prediction semantic audit trails (Sec. 4.4, Table 4); (4) production CTR deployments gained +5–6.6% revenue/margin at +30–50% serving latency (Sec. 5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 0 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the paper 'LLMs as Feature Engineers for Text-and-Tabular Prediction' about?
It presents an iterative framework where one LLM proposes human-readable categorical feature definitions (e.g., title orthography quality: professional/casual/noisy), a second cheaper LLM labels each text row into those categories zero-shot, and a gradient-boosted tabular model is trained with the resulting features to improve predictions on text-plus-tabular tasks.
How much do LLM-generated features improve prediction in this paper?
On three public datasets, adding the top 10 LLM features to a tabular baseline raised 5-fold cross-validated AUC from 0.7444 to 0.7639 (Kickstarter), 0.7611 to 0.8366 (Amazon Books) and 0.8544 to 0.9320 (Stack Overflow). Combining them with TF-IDF and sentence embeddings was best everywhere (0.7692, 0.8510, 0.9700 respectively).
Do the LLM features replace TF-IDF or sentence embeddings?
No — the paper finds them complementary. On Stack Overflow, LLM features alone (0.9320 AUC) actually trail MiniLM embeddings (0.9510), but adding LLM features on top of both TF-IDF and embeddings yields the best AUC on all three datasets studied.
Is code or data available for this paper?
No code release is mentioned. The three datasets are public Kaggle sets and the exact generator/extractor prompts are reproduced in the appendices, but the pipeline uses proprietary models (GPT-5.4 and GPT-4.1-nano), so exact replication is not possible.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (architecture diagram) is not analyzable from the text extraction; the loop description relies on Sec. 3 prose
- Production A/B metrics in Sec. 5 are internal company numbers and cannot be independently audited
- The extraction contains minor OCR hyphenation artifacts (e.g., 'auto- mates'), but all sections, tables and appendices appear complete
- Shaded-cell highlighting in Table 4 is inferred from the surrounding prose, not from the visual formatting
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.