Discovery-Driven Integration of Disjoint Tables via Text

Summary

The paper formalizes Text-Mediated Join Path Discovery: given tables and text documents in a data lake, discover which rows of disjoint tables are connected through specific sentences, without supplied row-text links or schemas. LOKI couples a frozen sentence encoder with trainable bidirectional cross-attention (shared LoRA projections, gated attention), producing contextualized row and sentence embeddings and a pair-score matrix. Training uses only coarse table-document contrastive triples plus a multiple-instance-learning local objective, so fine-grained row-sentence associations emerge without row-level labels. At discovery time, adaptive thresholding and mutual top-k filtering turn scores into atomic links; shared sentences form text-mediated join paths, which are clustered per table pair, named by an LLM generalization function G, and materialized as typed tables with sentence-level provenance. On FEVEROUS, ProTrix, Pharma, and MIMIC-IV, ablations show bidirectional contextualization beats encoder fine-tuning and unidirectional attention (MIMIC-IV row-sentence AP 0.54 vs 0.42–0.44) while training roughly 30× faster on a 16GB GPU. In end-to-end integration on 382 MIMIC-IV admissions, LOKI reaches about 0.98 macro typed-pair precision but recall of only ~0.44–0.46 (typed-pair F1 0.606–0.629); direct LLM prompting achieves higher F1 (up to 0.779). LOKI needs about one third of the tokens — roughly 11× cheaper than a local Qwen baseline and ~44× cheaper than a frontier API per Table 8. Stated limitations: pairwise candidate scoring grows with |T||D|, coverage/recall is the binding constraint, and relationship naming is delegated to G.