ARGUS: Role-Aware Event Knowledge Graphs for U.S. Employment-Discrimination Complaints
Sriram Kannan, Swetha Saseendran, Vishnu Vardhan Reddy Kandi, Leslie Barrett, Madhavan Seshadri, Enrico Santus
cs.CL
Quick verdict
ARGUS converts discrimination complaints into source-grounded event graphs; scoped-QA gains are well supported, classification rests on 22 test docs only.
Summary
ARGUS is a staged pipeline that converts the FACTS sections of U.S. employment-discrimination complaints from CourtListener into document-level Event Knowledge Graphs (EKGs). It fine-tunes LEGAL-BERT/LexLM for sentence-level fact extraction (with an LLM fallback for low-confidence sentences), semantically chunks facts, extracts chunk-level mini-graphs with a 5W1H-inspired schema via two-stage LLM generation (entities/events first, then source-supported temporal and causal edges), and merges them into document-level EKGs using deterministic deduplication plus LLM-assisted cross-chunk resolution. GEPA optimizes extraction configurations. Evaluation combines human annotation, automatic structural metrics, and multi-model LLM judging. Key results: LEGAL-BERT fact F1 0.727 on a 100-sentence gold sample; 78-85% of mini-graphs judged aligned with source; merging preserves nearly all events; the graph-structured GNN reaches 0.642 Micro-F1 versus 0.590 raw-text on a 22-document claim-classification test set; and EKG-only retrieval lifts document-scoped FAQ F1 to 0.446 from 0.323 (n=299, strong paired statistics). Bar-exam experiments are null or limited: no significant retrieval-condition difference (117 MBE questions), and open-retrieval gains are capped by 1-8% Recall@10. Main limitations: small evaluation samples (24 chunks, 5 documents, 22 test docs), LLM-generated silver classification labels, graph-targeted question design in the FAQ benchmark, project-member annotators, and unstated n behind identical p=0.031 judge-agreement values. No code release is stated; annotations are promised upon publication.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A staged, source-grounded pipeline converts employment-discrimination complaints into document-level EKGs with high source alignment (78-85% Aligned) and near-zero event loss at merging (1 of 134 events).
- Graph-structured EKG representations outperform raw-text and linearized baselines for 23-label SALI claim classification (0.642 vs 0.590 Micro-F1 on a 22-document held-out set).
- EKG-only retrieval substantially improves document-scoped legal QA (+0.123 Token-F1 over raw text, n=299, p<10^-30), with the largest gains on temporal and causal question types.
- EKG benefits do not extend to open-retrieval bar-exam QA when first-stage candidate recall is low (Recall@10 of 1-8%), establishing a boundary condition on the method's utility.
Strengths
Honest boundary-condition reporting
The paper pre-announces its own negative results in the abstract: Experiment A finds no significant retrieval-condition difference (McNemar p>0.68, Table 20), open-retrieval gains cap at 8% Recall@10 (Table 10), and the confounded 82% oracle-document configuration is explicitly flagged as not a graph-only ablation in the Limitations section.
Component-wise evaluation protocol isolates error sources
Fact extraction (Table 2), mini-graph quality (Tables 3-4), and document-level merging (Tables 5-7) are each evaluated separately with distinct instruments, rather than only end-to-end — more rigorous than most GraphRAG-adjacent work.
Exemplary paired statistics on the FAQ benchmark
Grounding is enforced by construction
Weaknesses
Every headline rests on a very small sample
Classification claim has no uncertainty quantification
FAQ benchmark design plausibly favors the proposed representation
Multi-judge statistics are opaque and internally inconsistent
Reproducibility is unaddressed
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ARGUS (arXiv 2609.30184, cs.CL) presents a staged pipeline that converts the FACTS sections of U.S. employment-discrimination complaints into document-level Event Knowledge Graphs (EKGs) annotated with participant roles, temporal relations (BEFORE/AFTER/OVERLAP/SAME-TIME), and causal relations (CAUSES/ENABLES/PREVENTS). Three research questions structure the paper: whether the staged pipeline yields reliable source-grounded graphs, whether the graphs improve claim classification and document-scoped QA over text baselines, and whether benefits extend to open-retrieval settings. Claims are scoped carefully: extracted events are explicitly framed as allegations rather than adjudicated findings, and the abstract itself pre-announces the boundary conditions (open-retrieval gains limited by first-stage recall).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the ARGUS system?
ARGUS is a research pipeline that converts the FACTS sections of U.S. federal employment-discrimination complaints (from the CourtListener repository) into document-level Event Knowledge Graphs that capture participants and their legal roles, temporal order, and causal relations between events, using fine-tuned legal language models plus LLM-based structured extraction.
Does building event knowledge graphs actually improve legal question answering?
In this paper, yes for document-scoped QA: when the relevant complaint is already known, retrieving from the event graph reached 0.446 Token-F1 versus 0.323 for raw-text retrieval over 300 questions (statistically significant). However, on open-retrieval bar-exam questions the graphs added little because the first-stage search rarely surfaced the relevant passage in the first place (1-8% Recall@10).
What data was used to evaluate ARGUS?
The authors collected 153 federal employment-discrimination complaints (Nature of Suit 442) from five U.S. district courts via CourtListener, plus two Multistate Bar Examination question sets (117 and 100 questions) for generalization tests. All complaint events are plaintiff allegations, not adjudicated facts.
Is ARGUS's code or data available?
As stated in the paper, only the human annotations are promised for release upon publication; no code, prompts, or extracted graphs are stated to be released. The pipeline also depends on several proprietary commercial LLM APIs, which limits independent reproduction.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-7 appear only as captions in the extracted text; the pipeline diagrams, GEPA loop, and clustering plots could not be assessed.
- Appendix content is only partially included (Tables 11-12, 16-21 present; some procedural details referenced but not shown).
- This is an arXiv preprint (2609.30184); peer-review status unknown.
- The exact arithmetic behind the Table 7 p-values cannot be verified because comparison counts are not reported.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.