Hakken: Predicting future discoveries to fill the gaps in today's knowledge
Tarek R. Besold, Uchenna Akujuobi, Pablo Sanchez, Alessandra Toniato, Kana Maruyama, Jihun Choi, Samy Badreddine, Frederick Gifford, et al.
cs.LG · cs.AI
Quick verdict
Hakken (Sony AI): temporal KG+LLM predicts undiscovered relations; strong benchmarks, 2/3 wet-lab confirmed, but leakage risk and n=3 validation temper claims.
Summary
Hakken is an AI system for knowledge prediction: given a pair of scientific entities, it predicts which as-yet-unpublished relation types hold between them. The predictor, THiGERLLM, encodes temporal knowledge-graph snapshots with a GraphSAGE-style encoder and hierarchical temporal transformer, injects the graph embeddings as prefix tokens into Mistral-7B-Instruct, and ensembles graph- and text-branch scores under a positive-unlabeled training objective with per-label calibration. The companion PHELInE framework explains each prediction via path-based sufficiency and necessity scores from a trained GraphSAGE surrogate. On a cleaned biomedical corpus of 254,806 entities and 7,127,960 triples (from PMC Open Access, MEDLINE, and licensed data), THiGERLLM outperforms eight baselines—including THiGER, ComplEx, tNodeEmbed, and a re-implemented Agatha—at temporal cut-offs from 1990 to 2020, with the largest gains in macro recall; historic evaluation shows micro recall of 0.645–0.739 ten years past the training cutoff versus random baselines near 0.16–0.22, with nDCG around 0.98–0.99. From 1.5M above-threshold aging hypotheses, biologists selected three for wet-lab testing; two were confirmed (TP53–BAMBI and RAF1–TNF) and one (SOAT1–STAT3) was not. Main limitations: the authors concede residual temporal leakage from LLM pretraining cannot be excluded; the corpus includes commercially licensed data and no code release is mentioned in the provided text; and wet-lab evidence rests on three expert-selected hypotheses.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- THiGERLLM establishes a new benchmark for time-aware multi-label relation prediction on biomedical literature, beating eight baselines at 1990–2020 cut-offs.
- Predicted hypotheses remain coherent and highly ranked up to ten years beyond the training cutoff, with micro recall 0.645–0.739 versus ~0.16–0.22 for random baselines.
- Two previously undocumented biomedical interactions predicted by Hakken — TP53–BAMBI and RAF1–TNF — were confirmed in wet-lab experiments via a CRO.
- PHELInE provides model-agnostic, path-based sufficiency/necessity explanations whose supporting evidence qualitative feedback from seven organizations rated as highly valuable.
- Hakken's confidence scores are calibrated probabilities, proposed for reuse as 'knowledge scores' in preclinical drug-development filtering.
Strengths
Full discovery cycle through wet-lab validation
Section 4.4 and Appendix F go beyond back-tests: from 1,543,297 above-threshold aging hypotheses, three were tested via a CRO and two confirmed across initial and confirmatory assays (TP53–BAMBI expression increase under p53 induction in HepG2; RAF1 inhibition raising TNF mRNA in THP-1). The failed SOAT1–STAT3 hypothesis is reported honestly rather than buried.
Rigorous temporal evaluation protocol
Appendix C trains at cut-offs 1990/2000/2010/2020, holds out a 100,000-pair validation sample from the test set, and evaluates only on relations first observed after the cut-off — a leakage-aware design many LBD papers skip. Sliding-window analyses (Fig. 3) show a stable decay profile across three cutoffs rather than one favorable snapshot.
Architecture is well-motivated for incomplete supervision
Explanation framework with measurable sufficiency/necessity
Unusually complete reproducibility materials for the wet-lab portion
Weaknesses
Wet-lab evidence is anecdotal and selection-biased
Direction-agnostic relation types make confirmation easier
Near-ceiling ranking metric and absent uncertainty reporting
Reproducibility blocked by proprietary components
Baselines may be understated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Hakken (Sony AI) is presented as a domain-general knowledge-prediction system: for an entity pair, THiGERLLM predicts which of 23 biomedical relation types hold but are not yet documented, with calibrated confidence, while PHELInE supplies path-based explanations. Four headline claims are advanced: (1) a new benchmark for time-aware multi-label relation prediction at 1990–2020 cut-offs; (2) hypotheses remain informative ten years beyond training (micro recall 0.645–0.739 vs ~0.16–0.22 random, Fig. 3a–c); (3) two wet-lab-confirmed novel interactions, TP53–BAMBI and RAF1–TNF, with the third tested hypothesis (SOAT1–STAT3) honestly reported as unsupported; (4) PHELInE explanations that domain experts valued. The evidence largely supports a weaker version of each claim, with systematic caveats the authors partially acknowledge.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Hakken and what does it predict?
Hakken is Sony AI's knowledge-prediction system. Given two scientific concepts, it predicts which relation types between them are likely true but not yet documented in the literature, assigning each prediction a confidence score and path-based explanations. Its predictor (THiGERLLM) combines temporal knowledge-graph evidence with a large language model (Mistral-7B-Instruct), and an explanation framework (PHELInE) shows which known literature paths support each hypothesis.
Which new biological discoveries did Hakken make?
The paper reports that cell-line experiments confirmed two of three selected predictions: TP53 affects BAMBI expression (BAMBI expression rose when p53 was induced in HepG2 cells) and RAF1 down-regulates TNF (RAF1 inhibitors increased TNF mRNA in THP-1 cells). A third hypothesis, SOAT1–STAT3, was not supported in the tested design. Assays were run by an independent contract research organization; the authors state these two interactions were previously undocumented in biomedical literature.
Is Hakken's code or dataset publicly available?
No code or data release is mentioned in the provided text of the paper. The biomedical dataset combines public sources (PMC Open Access full texts, MEDLINE titles/abstracts) with a commercially licensed component and an external rule-based relation-extraction pipeline, which would limit full reproduction even if code were released. The wet-lab methods (Appendix F), by contrast, are documented in reproducible detail.
How accurate is Hakken at predicting future discoveries?
In historic back-tests with training cut-offs in 1990, 2000 and 2010, the model recovered about 0.65–0.74 micro recall of relations first appearing up to ten years later, versus roughly 0.16–0.22 for a random baseline; macro recall was about 0.42–0.51 versus ~0.05. However, the authors acknowledge that language-model pretraining leakage cannot be fully excluded, so some of this apparent foresight may reflect knowledge already inside the LLM rather than pure graph-structure prediction.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with three marked '[...sampling gap...]' regions; portions of Section 4.2/4.3, related work, and parts of Appendices A/C/D/F were not visible, so judgments about those parts rely only on the surrounding visible text.
- No code/data availability statement appears in the provided excerpt; one may exist in skipped sections, so codeAvailable=false reflects the visible text only.
- Detailed per-baseline benchmark values (e.g., Fig. C4 breakdowns) were not visible in the excerpt; figure-internal recall/nDCG values were readable from extracted axis labels and are cited from those.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.