Can LLMs Use Relational Transformer Embeddings?
Francisco Galuppo Azevedo, Clarissa Lima Loures
cs.LG
Quick verdict
Negative result: RT soft-token fusion into Qwen3.5-4B rarely beats standalone RT on RelBench; honest analysis, but single-seed and no text-only LLM baseline.
Summary
The paper tests whether frozen Relational Transformer (RT) embeddings, injected as soft tokens into Qwen3.5-4B via a two-layer MLP projection plus LoRA, can improve binary node classification on relational databases. Training follows a BioReason-style recipe: SFT on 7,869 chain-of-thought traces (generated by Claude Sonnet 4.6 from ego-graph dumps, up to three style variations each, with noise filtering), followed by GSPO reinforcement learning with a binary correctness reward and a format reward, run per task-regime on 100 instances. Across 10 RelBench tasks on 6 databases and four supervision regimes (ST, WD, CD, ALL), standalone RT dominates: mean AUROC 71.9 (ST) and 69.7 (CD) versus 48.3-61.4 for hybrid variants; the GSPO ALL mean of 61.4 is inflated by two outlier tasks, with the remaining eight averaging 57.9. An ablation on two tasks shows that switching from flat to JSON serialization shifts training reward by +0.63/+0.36 — evidence the LLM leans on textual scaffolding rather than decoding the soft tokens. The authors frame this honestly as a negative result and diagnose failure modes: serialization sensitivity, token-budget saturation above L=64, inconsistent attention-mask effects, and RL instability. Limitations include single-seed results, 100 GSPO samples per task-regime, one 4B LLM, binary tasks only, no text-only LLM baseline isolating the soft tokens' contribution, and an SFT warm-up objective that never uses labels.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Injecting frozen RT embeddings as soft tokens into Qwen3.5-4B (MLP projection + LoRA, SFT then GSPO) does not consistently outperform standalone RT across 10 RelBench binary tasks and four supervision regimes (Table 3: RT 71.9 ST / 69.7 CD vs hybrids 48.3-61.4).
- The hybrid is frequently below random, e.g., SFT-only mean AUROC 48.3 and multiple per-task cells in Table 6 in the 36-46 range.
- Performance is highly sensitive to serialization format: switching flat→json shifts training reward by +0.632 (rel-f1) and +0.359 (rel-trial), implying the LLM leans on textual scaffolding rather than the continuous embeddings.
- SFT warm-up on description traces provides no measurable benefit over GSPO-only (50.7 vs 51.1 ST mean AUROC), and the two-stage recipe is unstable under RL.
- Soft-token fusion requires stronger alignment objectives and schema-aware projection design before it can serve as a reliable route to relational prediction.
Strengths
Honest negative-result reporting
Table 6 color-codes below-random per-task AUROC cells; Table 3's caption discloses that the GSPO ALL mean of 61.4 is driven by two outliers (user-clicks 81.5, user-visits 68.9; remaining eight average 57.9); Section 5 self-enumerates scope limits.
Transfer-oriented evaluation grid
Four supervision regimes (ST/WD/CD/ALL) across 10 tasks on 6 RelBench databases directly probe the cross-schema gap that prior Rel-LLM (Wu et al., 2025) left untested — the regime grid itself is the paper's main methodological contribution.
Sharp failure-mode diagnosis
Unusual reproducibility documentation
Weaknesses
Missing text-only control arm
Single-seed everything
SFT objective misaligned with the task
Tiny RL budget behind the instability claim
Internal configuration inconsistency
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This ICML 2026 FMSD workshop paper (arXiv 2609.00457) asks whether frozen Relational Transformer (RT; Ranjan et al., 2026) embeddings, injected as soft tokens into Qwen3.5-4B through a two-layer Tanh MLP (256→2560) plus LoRA (r=16, α=32), can improve binary node classification on relational databases. The recipe mirrors BioReason: SFT on 7,869 chain-of-thought traces distilled from Claude Sonnet 4.6 over ego-graph dumps (three style variations, four-type noise filtering; Table 1), then per task-regime GSPO with a binary correctness reward plus a format reward (G=4, 100 instances, 2 epochs). Evaluation covers 10 RelBench tasks over 6 databases under four supervision regimes (ST/WD/CD/ALL), measured by AUROC on temporal test splits. The headline claim is explicitly negative: the hybrid never closes the gap to standalone RT (71.9 ST / 69.7 CD mean AUROC vs 48.3-61.4 for hybrids, Table 3), is frequently below random, and is fragile to design choices.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the paper conclude — can LLMs use Relational Transformer embeddings?
Mostly no. Injecting frozen Relational Transformer embeddings as soft tokens into a 4B LLM did not consistently beat the standalone RT model across 10 RelBench binary tasks; the hybrid was often near or below random (mean AUROC 48.3-61.4 vs RT's 69.7-71.9), and results were highly sensitive to how the relational tokens were serialized into the prompt.
Why did the RT-to-LLM fusion fail according to the authors?
Their ablation suggests the LLM relies on the text around the embeddings rather than the embeddings themselves: switching from flat soft-token prefixes to a JSON-interleaved layout changed training reward by +0.63/+0.36 on two tasks, and adding relational tokens beyond 64 gave diminishing returns. RL training was also unstable on the small 100-sample-per-task budget.
What datasets and models were used in this study?
Ten binary classification tasks from RelBench spanning six relational databases (Amazon, Avito, F1, HM, Stack Exchange, clinical trials); a pretrained Relational Transformer as the frozen encoder; and Qwen3.5-4B adapted with LoRA, trained first with SFT on teacher-generated reasoning traces and then with GSPO reinforcement learning.
Is code available for this paper?
The paper does not state a code release. It does provide full hyperparameters (Appendix A), exact prompts (Appendix C), and the SFT data-generation procedure (Appendix B); all core components (RelBench, the RT checkpoint, Qwen3.5-4B) are publicly available, so reimplementation is feasible.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Table 2's column layout is scrambled by PDF extraction (the header reads 'Config. Data Task Format Time rel-attn mask'); the reward and 'Time' column semantics are inferred, and Time's unit is unstated in the text itself.
- Figures 1-4 appear only as captions/inline descriptions in the extraction; the training-curve shapes in Appendix Figure 4 could not be inspected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.