NLPCC 2026 Task 10: Citation-Level Faithfulness Verification with DeBERTa Ensembles and Class-Wise Calibration
Yanling Li, Zirui Li, Mingyu Wan
cs.CL
Quick verdict
NLPCC 2026 Track 2 runner-up (82.99 overall) with thorough ablations and honest limitations; key conclusions rest on OOF diagnostics, no significance tests.
Summary
This system report describes the second-place entry for Track 2 of the NLPCC 2026 Shared Task 10 on citation-level faithfulness in AI-assisted scientific reporting. Given an atomic claim and the cited paper's full text, the system predicts a four-way relation label and up to three evidence paragraphs, fully offline. The label head ensembles a DeBERTa-v3-base paragraph-aware cross-encoder (jointly trained with an evidence head via attention pooling) with a DeBERTa-v3-large document classifier (fused 0.70/0.30), followed by class-wise logit adjustment (FULL2 = [1.40, 0.90, 0.65, 1.00]) that corrects an over-prediction of Topical Match. Evidence scores fuse top-20/top-30 joint-model scores with a small BM25 term. On the official leaderboard it scored 82.99 overall (89.55 Macro-F1, 76.43 Joint@3), averaged over two phases. Ablations show BM25-only input beats forced abstract/conclusion context (87.95 vs 83.91 OOF Macro-F1), model scale helps (+1.45), calibration adds about 1.9 Phase 1 points, and the relation head is worth 3.95 OOF points within a single run. Oracle analyses show gold-evidence input changes label Macro-F1 negligibly (+0.0055) while evidence ordering remains the Joint@3 bottleneck. Main limitations: instance-level cross-validation folds that ignore document overlap (inflating OOF estimates), a single seed with no significance tests, calibration chosen partly with public-leaderboard feedback, and no dependency lockfile or committed weights.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A two-head offline system (paragraph-aware joint cross-encoder + DeBERTa-v3-large classifier, probability-fused and class-wise calibrated) achieved 82.9898 overall (89.5491 Macro-F1, 76.4305 Joint@3), ranking second in NLPCC 2026 Task 10 Track 2.
- Class-wise logit adjustment targeting the Macro-F1 decision boundary improves the Phase 1 score by 1.89 points (79.39→81.28, Table 4) by correcting systematic Topical Match over-prediction.
- Pure BM25-retrieved context beats forced abstract/conclusion injection (87.95 vs 83.91 OOF Macro-F1) and base→large model scaling adds 1.45 points (Table 3).
- Oracle analyses show candidate retrieval is nearly saturated (BM25 H@20 98.15%) so gold-evidence input changes label Macro-F1 negligibly (+0.0055), while within-candidate evidence ordering remains the Joint@3 bottleneck (top-30 OOF Joint@3 81.96%).
Strengths
Systematic, multi-axis ablation suite
Input construction (Table 3: forced context top-8 83.91 vs pure BM25 top-10 87.95 vs large 89.40 OOF Macro-F1), a fusion-weight sweep justifying the 0.30 large weight (Phase 1 76.71–79.39, §5.2), a staged calibration ladder (Table 4: 79.39→81.28), a flat BM25 evidence sweep (Table 5), and a relation-head ablation (−3.95 points within-run, §4.2).
Oracle decomposition cleanly localizes the bottleneck
§7.3: gold-evidence input changes Macro-F1 by only +0.0055 (91.9019→91.9074) while a gold-label oracle lifts Joint@3 to 88.16% (1906/2162) — establishing that evidence ordering, not candidate recall (H@20 98.15%, H@30 99.10%), limits Joint@3.
Unusually candid limitations and compliance reporting
Calibration gains backed by a stability check and confusion analysis
Weaknesses
All OOF ablation conclusions inherit fold leakage
No inferential statistics; single seed and single run
Test-set-informed selection of the calibration vector
Reproducibility gaps despite a released repository
Evidence head comparatively under-explored
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: NLPCC 2026 Task 10 Track 2 System (Li, Li & Wan)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What score did this system achieve in NLPCC 2026 Shared Task 10 Track 2?
It scored 82.9898 overall on the final leaderboard — 89.5491 Macro-F1 and 76.4305 Joint@3, averaged over the two evaluation phases — ranking second in Track 2.
What is class-wise decision calibration in this paper?
It is a per-class adjustment applied before the final argmax decision: adding log multipliers [1.40, 0.90, 0.65, 1.00] to the fused class probabilities, boosting the Supported margin and lowering the Topical Match margin. It targets Macro-F1 rather than probability calibration and changed only 33 of 2,162 out-of-fold decisions in the authors' analysis.
Is the code and data available?
Training/inference scripts, fixed fold files, notebooks, submitted prediction archives and intermediate outputs are on GitHub (Task10-Team-Lenormand, branch 'final'), but organizer-provided JSONL data and model weights are excluded and dependency versions were not locked, so it is not a self-contained reproduction package.
Why is Joint@3 much lower than Macro-F1 for this system?
Evidence localization is the bottleneck: BM25 candidate recall is already about 98–99%, yet picking the right paragraphs within those candidates reaches only 85.62% top-30 out-of-fold Evidence Hit@3, while relation labeling exceeds 89 Macro-F1 — a 13.12-point final gap.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (pipeline diagram) cannot be analyzed from extracted text; the assessment relies on its caption and the prose in Sections 4–5.
- arXiv text extraction shows minor formatting artifacts around equations, but all tables and key numbers appear intact.
- Leaderboard results cannot be independently verified since Phase 2 test annotations are unavailable; the review takes Table 7 values as reported.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.