The record is part of the task: matched-record evaluation of text classifiers across maintenance, safety and recall reporting
Hisham Ihshaish, Peter Mayhew, Tasnim M. A. Zayet, Ana Del Amo
cs.LG · cs.CL
Quick verdict
Claims record choice moves scores more than model choice; rigorous matched-record design across 3 systems, but GE data is proprietary and causes confounded.
Summary
This paper argues that the choice of which written record of a case a classifier reads is itself part of the evaluation, not a preprocessing detail. The authors compare matched records of the same cases under fixed labels and splits in three systems: GE Aerospace avionics repair events, NASA ASRS safety reports and NHTSA vehicle recalls. On the GE task, where labels come from parts transactions independent of the narratives, held-out macro-F1 ranged from 0.327 on customer-reported faults to 0.910 on post-repair repair actions; the 0.456 gap between the customer and technician fields exceeds the largest modelling difference tested (0.092 for sequence reading over mean pooling). On ASRS, the analyst synopsis beat the reporter narrative by +0.015 macro-F1 under learned sequence models but not under lexical baselines, and the ranking of TF-IDF versus a BiLSTM reversed across records. On NHTSA, the defect summary remained strongest under all tested model families, robust to class-label masking, out-of-time evaluation and near-duplicate grouping. Limitations: the GE dataset is proprietary and covers a single LRU family with roughly a dozen authors; the GE contrasts lack exportable confidence intervals; record effects confound information timing with authorship and style; and the misclassification audit was unblinded and exploratory. Code and public-data snapshots are released, making the framework a valuable template for decision-time-valid evaluation.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Swapping which record of the same case a classifier reads moves held-out macro-F1 on GE repair events by up to 0.583, far exceeding the largest modelling difference tested (0.092 for BiLSTM sequence reading over mean pooling).
- The 0.456 gap between the customer and technician fields exists before the label-generating parts transaction and is not relieved by any tested model or embedding change.
- On ASRS, the analyst synopsis's +0.015 advantage appears only under learned sequence models (not lexical baselines), and the TF-IDF-versus-BiLSTM ranking reverses across records (interaction D = +0.017 [0.013, 0.021]).
- The NHTSA defect summary remains the strongest field under every model family tested and under class-label masking, out-of-time and near-duplicate robustness conditions.
- Valid classifier evaluations must restrict inputs to records available at the intended decision time and report how both the record and the label were produced.
Strengths
Novel, well-formalised estimand
Section 3 defines the record contrast (Eq. 1), record-model interaction D (Eq. 2) and cross-record transfer T, placing the choice of record on the same held-out scale as modelling choices, with a decision-time availability formalism (Eq. 3, Fig. 7) that cleanly separates pre-decision from post-outcome records.
Exceptional statistical hygiene
Paired case-bootstrap CIs on shared held-out cases, approximate-randomisation tests with Holm correction within declared families (resample counts documented in Table S3), three-trainings-per-configuration, and the analysis-provenance labelling in S7/Table S22 (preplanned / prospective extension / post hoc / prior-study design) that pre-empts cherry-picking accusations.
Robust public-system findings
Verification infrastructure
Honest self-scoping
Weaknesses
Headline number is the least protected result
Confounded estimand by construction
Narrow external validity of the largest effect
Dated model suite weakens the ceiling claim
Exploratory audit cannot support its headline share
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a deceptively simple question: when a real-world case is documented several times (customer complaint, technician diagnosis, repair log; reporter narrative versus analyst synopsis; defect/consequence/remedy fields), how much does the choice of record - rather than the choice of model - determine measured classifier performance? The authors formalise this as a record contrast (Eq. 1), a record-model interaction D (Eq. 2) and a cross-record transfer T, applied to three systems with deliberately different label-record relationships: GE Aerospace LRU repairs (label from parts transactions, independent of all narratives), NASA ASRS (label co-produced by the analyst who writes the synopsis) and NHTSA recalls (regulator codes the filing containing the fields). Headline claims: record choice moves held-out macro-F1 by up to 0.583 on GE (0.327 customer field to 0.910 repair action) against 0.092 for the largest modelling change tested; the pre-transaction 0.456 customer-to-technician gap is unrelieved by any tested model; some model rankings are record-dependent; and evaluations should be decision-time valid. The claims are carefully scoped and mostly supported by the presented evidence.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is matched-record evaluation?
An evaluation design in which the same real-world cases each have several text records (for example a customer complaint, a technician's diagnosis note and a repair log), and a classifier is trained and tested separately on each record while the cases, labels and test split stay identical. This isolates how much the choice of record, rather than the choice of model, drives measured performance.
How large was the record effect on the GE Aerospace data?
On the proprietary GE avionics repair dataset, held-out macro-F1 rose from 0.327 on customer-reported faults to 0.910 on post-repair repair-action text. The gap between the customer field and the technician field (both written before the parts transaction that generates the label) was 0.456 macro-F1, versus 0.092 for the largest model or architecture change tested on the same events.
Is the code or data available?
The NASA ASRS and NHTSA datasets are public, and the analysis code, public-data snapshots with hashes, and analysis plans are released at github.com/ihshaish/source-variation. The GE Aerospace dataset and the Avi2Vec embeddings are proprietary and cannot be shared, so the largest reported effect cannot be independently reproduced.
Why should the ASRS synopsis result be interpreted cautiously?
The analyst synopsis is written by the same ASRS analyst workflow that assigns the classification label, so the +0.015 macro-F1 advantage over the reporter narrative is best read as evidence about curation within one workflow rather than a clean record-quality effect. The authors support the finding with label-token masking and robustness checks but acknowledge the label and record are co-produced.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (Fig. 4-7, S1-S6) are only partially recoverable from the extracted text; some axis values are inferred from captions and surrounding prose.
- GE field-level medians and internal statistics are proprietary or deferred to the supplement and could not be independently verified.
- The public repository (github.com/ihshaish/source-variation) was not inspected; archived results and plans are taken on trust.
- Review is based on the provided extraction; the supplement appears complete but minor truncation cannot be excluded.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.