Context Window Failures in Relational Foundation Models
Denis Oliveira Correa, Francisco Galuppo Azevedo
cs.LG
Quick verdict
Animus shows RT/Griffin/RelGT collapse on high-cardinality relational data; a convincing stress test, but synthetic-only, single-task, test-set-selected.
Summary
The paper introduces Animus, a synthetic financial relational dataset built to stress-test the neighborhood budgets of relational deep learning foundation models. Each customer's monthly income is decomposed into K credit transactions that sum exactly to the label, so a model sampling only k of K transactions underestimates income by roughly k/K; 5% of customers generate 1,000-10,000 transactions per month, up to 88,717 raw nodes. Four models (GraphSAGE, RT, Griffin, RelGT) are evaluated on raw versus monthly-aggregated representations of the same data under identical hyperparameter grids. The three budget-limited foundation models score at most R² 0.184 on raw data (RelGT runs out of memory), while pre-aggregation lifts RT to 0.654, Griffin to 0.384 and RelGT to 0.230; GraphSAGE, with a much larger sampling budget, moves only from 0.693 to 0.752. A stratified analysis attributes the gains to high-frequency customers. Main limitations: a single synthetic, purely additive task; no seed variance or confidence intervals; headline numbers come from the best test-set configuration in the grid search (validation-best differs for Griffin and RelGT); budgets are never varied; and code is promised only upon acceptance.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 19 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- All three recent relational foundation models (RT, Griffin, RelGT) impose hard per-entity row budgets and silently discard records beyond the limit (Sections 1-2).
- On raw Animus the three foundation models reach at most R² 0.184, while a routine monthly pre-aggregation lifts RT to 0.654 and Griffin to 0.384 (Table 1).
- GraphSAGE, with a much larger sampling budget, is nearly representation-insensitive (0.693 to 0.752), showing the failure is budget-specific rather than task difficulty.
- Per-bin analysis localizes the failure to customers whose post-cutoff transaction counts exceed sampling budgets, with +0.443 R² gains in both high-frequency bins (Table 2).
- Current relational foundation models are therefore not ready for high-cardinality real-world data (Abstract, Section 5).
Strengths
Analytically precise adversarial mechanism
Section 3 constructs credits via Dirichlet(1) weights summing exactly to the monthly income, so the failure mode is derived, not just observed: a model sampling k of K credits has expected estimate y·k/K. The benchmark isolates one causal factor cleanly.
Controlled two-representation design
Raw and Agg-Simple encode identical aggregate information (Section 3), and the same hyperparameter grids are applied to both representations with no model-specific tuning (Section 4), making cardinality the only manipulated variable.
Honest disclosure of failures and gaps
Failure localization with an honest inverse result
No-signal control table
Weaknesses
Headline numbers appear test-set-selected
No variance quantification
Budgets are never varied
External validity rests on one additive synthetic task
Incomplete metric coverage
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a benchmark/empirical paper by Correa and Azevedo (Kunumi Institute / UFMG), submitted to the 2nd ICML Workshop on Foundation Models for Structured Data (FMSD @ ICML 2026). It introduces Animus, a synthetic financial relational database of 100,000 customers whose income-prediction task requires aggregating up to tens of thousands of transactions per entity. Central claims: (1) all three recent relational foundation models (RT, Griffin, RelGT) impose hard per-entity row budgets that silently discard records; (2) on raw Animus they reach at most R² 0.184, while a single monthly pre-aggregation lifts RT to 0.654 and Griffin to 0.384 (Table 1); (3) GraphSAGE, with a much larger sampling budget, is nearly representation-insensitive (0.693 → 0.752); (4) hence current relational foundation models are not ready for high-cardinality real-world data.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Animus dataset?
Animus is a synthetic relational financial dataset of 100,000 customers with three linked tables (Customers, Transactions, Payments). Its income-prediction task requires summing a customer's credit transactions in the following month, and 5% of customers generate 1,000-10,000 transactions per month, so models must aggregate neighborhoods of up to 88,717 records.
How badly do relational foundation models fail on Animus?
On the raw event-level representation, RT reaches test R² 0.184, Griffin 0.110, and RelGT cannot even build the graph due to memory limits. After a simple monthly pre-aggregation of the same data, RT improves to 0.654, Griffin to 0.384, and RelGT to 0.230, while GraphSAGE moves only from 0.693 to 0.752.
Why does pre-aggregation help so much?
The prediction label equals the sum of all credit transactions in the target month. A model that can only sample k of K transactions underestimates income by roughly k/K, so collapsing thousands of events into monthly sums lets small-budget models see the complete aggregate.
Is the Animus code or data publicly available?
Not yet. The paper states dataset generation code and model configurations will be made publicly available upon acceptance. However, the generative process is specified in enough detail in Section 3 to reimplement, and all four evaluated models have public code.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 are image content; only captions and axis labels were extractable, so the visual shape of the transaction-count distribution (Figure 2) could not be assessed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.