CORAL: An LLM-Native Harness for Production Recommender Systems
Muhammad Rafay Azhar, Yuhang Zhou, Gilbert Jiang, Yuchen Wang, Rahul Sharma, Matthew DeSousa, Jiayi Liu, Xin Guo, et al.
cs.CL
Quick verdict
Meta's LLM agent loop tunes live recommenders in A/B-verified cycles; real production evidence, but tiny effects, no baselines, and few cycles limit claims.
Summary
CORAL is a Meta system paper that places an LLM agent in a closed loop around a production recommender. Every three days the agent observes per-component operating statistics, consults a memory of past configurations and their attributed outcomes, proposes reallocations, and a constrained numerical optimizer projects any proposal onto the budget-feasible set before deployment; A/B-measured outcomes feed the next cycle. The task is formulated as partially observed, non-stationary, constrained optimization, improved in context without parameter updates. On a video service's retrieval-budget allocation, three rounds moved from +0.13% watch time (R1) through a neutral overcorrection (R2) to a deployed configuration with +0.15% watch time, +0.16% sessions overall (+0.77% in the largest market), and +0.23% sessions for new low-signal users. A second service allocated serving treatments across user segments, saving 'millions of USD' annualized, increased 44% in round two, with engagement statistically unchanged. Inference costs about 10^6 tokens (tens of USD) per deployment. Main limitations: no baseline comparisons (e.g., Bayesian optimization or bandit allocators), only 2-3 evaluation rounds supporting the 'improves over iterations' claim, effect sizes reported without confidence intervals, proprietary systems preventing reproduction, and a single decision class (resource allocation).
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A general-purpose LLM wrapped in the CORAL harness (memory, deterministic tools, constrained optimizer) can continually reconfigure a live production recommender and improve it through A/B-verified closed-loop feedback.
- The same harness generalizes across two platforms and decision types: retrieval-budget allocation (watch time +0.15%, sessions +0.16%, no added serving cost) and serving-capacity allocation (millions of USD annualized savings, engagement statistically unchanged).
- The agent's policy improves in context over successive cycles, without parameter updates, as attributed outcomes accumulate in memory (m=3 cycles).
- Per-decision LLM cost is negligible (~10^6 tokens, tens of USD per deployment) and independent of traffic because the agent acts on the control surface rather than per request.
Strengths
Real production A/B evidence at scale
Both case studies (§5) are evaluated with A/B experiments spanning millions of users on live Meta services; Table 1 reports the honest R1→R3 trajectory including the failed R2 overcorrection rather than only wins.
Sound division of labor: LLM proposes, solver disposes
The constrained optimizer (§4.2) guarantees budget feasibility by projection onto {s : c(s) ≤ B}, so model errors cannot overspend; fixed control flow (§4.3) explicitly avoids letting the model steer the loop.
Transparency about mechanism and cost
Covers both objective orientations
Honest failure and limitation reporting
Weaknesses
No baselines for the core implicit claim
Learning claim rests on 2-3 iterations
No uncertainty quantification
Reproducibility is near zero
Attribution methodology underspecified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
CORAL: An LLM-Native Harness for Production Recommender Systems — Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CORAL?
CORAL is a Meta research system that places an LLM agent in a closed loop around a live recommender: every 3 days it observes per-component metrics, recalls its past decisions from memory, proposes budget reallocations, a constrained optimizer enforces the operating budget, and A/B-measured outcomes feed the next cycle — all without retraining the model.
What improvements did CORAL achieve?
On a video service it raised watch time +0.15% and sessions +0.16% (+0.77% in its largest market) at no added serving cost, plus +0.23% sessions for new low-signal users. On a second service it cut serving capacity cost, saving millions of USD annualized (44% more in round two) without statistically degrading engagement.
Is CORAL's code or data public?
No. Both deployments run on proprietary Meta platforms with private telemetry; the paper releases only an abstracted prompt template and cost estimates, so third parties cannot currently reproduce the results.
Does the LLM in CORAL get trained or fine-tuned?
No. The policy stays a frozen general-purpose LLM; it improves 'in context' by reading a memory of its last 3 cycles — observations, its own assessments, and past decisions with their attributed outcomes.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Provided metadata lists 8 authors; the embedded paper text lists 10 (adds Lizhu Zhang and Xiangjun Fan). This review follows the paper text.
- Extraction appears complete from title through references and Appendix A (Fig. 2, Tables 2-3); no truncation detected, so none is claimed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.