Data Auditing and Quality Assurance in a Federated Learning Consortium; Getting the Best of Both Worlds from Cross-Institutional and In-House Data Quality Inspection
Hogenboom, J., Perez, N., Filori, Q., Sans, A., Lobo Gomes, A., Dekker, A., van der Graaf, W., Husson, O., et al.
health informatics
Quick verdict
The paper claims in-house and federated data-quality dashboards are complementary in FL consortia; plausible and useful, but a small, simulated, purely qualitative demonstration.
Summary
The paper presents two complementary data-quality dashboards for the STRONG AYA federated oncology consortium: an in-house Power BI tool for intra-clinic checks on raw EHR data, and a federated Plotly Dash/Vantage6 tool operating over FAIR-ified RDF graphs that displays per-station aggregates without revealing individual records. The authors adapt the Kahn framework into completeness, consistency, and semantic-consistency dimensions, re-purposing the semantic axis toward interoperability for the federated case, with rule sets grounded in ENCR/JRC guidance. Detection is evaluated on 839 consented AYA breast-cancer records from Centre Léon Bérard, processed via OSIRIS-RWD, split into two simulated stations with date-shifting and ICD-code relabelling, then seeded with artificial inconsistencies. Results are a side-by-side capability comparison: the in-house dashboard uniquely traces errors to individual patients, while the federated dashboard detects cross-site coding divergence and non-IID distribution patterns. Table 1 consolidates what each level can and cannot show. The conclusion—that the two are complementary—is credible but rests on a qualitative demonstration rather than a quantitative comparison: only a few illustrative numbers are given (9.2% missing TNM M values; 4 implausible BMI values; 4 impossible sex–topography combinations), with no systematic quantification of what each tool detected and no comparison against existing solutions. The authors concede a temporal-consistency blind spot, and the two-station simulation derived from a single French cohort limits the scalability claims.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The Kahn et al. data-quality framework can be adapted to serve both intra-clinic (in-house) and inter-clinic (federated) quality assessment in the STRONG AYA consortium.
- The in-house dashboard uniquely provides patient-level error traceability (e.g. 4 implausible BMI values, 4 sex–topography conflicts), which the federated dashboard cannot offer under privacy constraints.
- The federated dashboard uniquely reveals ecosystem-wide patterns — coding-system divergence (ICD-O-3 vs ICD-10) and non-IID value distributions — that single-institution tools cannot see.
- The two dashboards are complementary, and a combined schema-agnostic local-plus-federated model is the optimal balance for evolving multi-institutional ecosystems.
- Both dashboards sufficed to assess feasibility of two consortium use cases (survival rates; high-cost treatment proportions), though the federated arm has a temporal-consistency blind spot.
Strengths
Grounded, standards-aligned framework implementation
The Methodology section operationalises Kahn et al.'s harmonised DQ terminology into completeness/consistency/semantic-consistency dimensions and integrates ENCR-recommended validation rules (JRC procedure, ref 21) as CSV, with a principled re-purposing of the semantic dimension toward interoperability for the federated case.
Realistic, well-governed evaluation substrate
The CLB cohort (839 patients, consent level ≥5, 1996–2024) is processed through a production ETL to OSIRIS-RWD with MR004/GDPR governance, uniform date-shifting, and a documented transparency portal (ref 14); simulated federated stations mirror real constraints (date obfuscation, ICD-10 vs ICD-O-3 schema heterogeneity).
Actionable side-by-side capability matrix
Open federated toolchain
Use-case-anchored usability demonstration
Weaknesses
Purely demonstrative evaluation with no detection quantification
Near-tautological core conclusion
Weak external validity behind the scalability claim
Limited auditability of the evidence
Privacy of aggregate outputs not analysed
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is an infrastructure/tooling paper from the STRONG AYA federated oncology consortium (AYA = adolescent and young adult cancer). The authors build and compare two data-quality (DQ) dashboards: an **in-house** Power BI/DAX dashboard over a Microsoft SQL Server holding raw OSIRIS-RWD-transformed EHR data, and a **federated** Plotly Dash client executing Vantage6 tasks over Flyover-generated RDF triple stores (SPARQL-queryable), deliberately showing per-station metrics instead of aggregated results. The framework adapts Kahn et al.'s completeness/consistency/plausibility dimensions, re-purposing the semantic axis toward **interoperability** for the federated arm, with ENCR/JRC validation rules integrated as CSV. The central hypothesis (Introduction) is that both levels are essential and complementary. The usability demonstration targets two genuine consortium use cases: 1/3/5-year AYA survival rates and high-cost treatment proportions — explicitly as feasibility assessment, not execution.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What do the two dashboards in this federated learning data quality study actually do?
The in-house dashboard (Power BI, Microsoft SQL Server) lets each institution run patient-level quality checks — completeness, implausible values, temporal logic — on its own raw data and trace errors to individual patients. The federated dashboard (Plotly Dash + Vantage6 over RDF graph databases) lets institutions compare quality metrics without individual records ever leaving their premises.
How did the authors test whether the dashboards detect data errors?
They took 839 real AYA breast-cancer records from one French cancer centre, split them into two simulated federated stations, and deliberately introduced errors such as impossible sex–tumour-site combinations, ICD-10 vs ICD-O-3 coding splits, and implausible BMI values, then checked what each dashboard could flag.
Does the federated dashboard expose patient data?
No. It displays per-station aggregate counts and patterns (e.g. missingness rates, coding-system divergence, distribution differences). Individual-level detail and patient identifiers remain local, which is also why the federated dashboard cannot trace an error to a specific patient.
What are the main limitations of this study?
It is a qualitative demonstration, not a quantitative evaluation: no detection rates over the injected errors are reported, only two simulated stations derived from a single centre are tested, two supplementary figures are available only on request, and the authors concede a temporal-consistency blind spot in the federated setting.
What is the paper's main recommendation?
Neither tool alone is sufficient: the authors recommend combining a schema-agnostic local (in-house) dashboard for patient-level validation with a federated dashboard for cross-institution oversight — a model they argue suits dynamic consortia like STRONG AYA.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 could not be analysed from the extracted text; judgments on visual design rest on captions and Table 1.
- Supplementary Table 1 (inconsistency inventory) and Supplementary Figures 2-3 were not in the extraction and are partly 'available on request, in French', limiting detection analysis.
- The extraction interleaves medRxiv watermark boilerplate throughout, and some passages appear truncated; section boundaries may be imprecise.
- The text contains no dedicated 'Limitations' heading; limitation-related statements were drawn from the Strengths/limitations and Discussion passages.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.