Automated Extraction of Records of Processing Activities (RoPA) Using Hybrid RAG and Locally Deployed Large Language Models
To Duy Hinh, Nguyen Le Quoc Anh, Phan Van Tri, Khuong Nguyen-An
cs.CL · cs.CR · cs.IR
Quick verdict
Proposes first Vietnamese RoPA benchmark with honest 3-level eval; local LLM matches cloud on coverage (~55%), but precision unmeasured and data synthetic.
Summary
This paper presents RoPA Manager, a system that automates extraction of Records of Processing Activities (RoPA) required by Vietnam's Personal Data Protection Law (effective 2026), using hybrid retrieval (PostgreSQL tsvector lexical ranking, pgvector dense search, weighted RRF fusion) feeding locally deployed LLMs. Its main contribution is the first Vietnamese RoPA benchmark: 32 synthetic organizations, 77 processing activities, 12 field groups, and 4,338 manually authored reference values. Evaluation is deliberately split into three levels: (1) scorer-only robustness without LLM calls (F1 = 0.9493 on noisy data, n = 3,965; correct-refusal 0.7874 fails its 0.80 threshold); (2) end-to-end token coverage of 50.04-55.25% across three models, with local Qwen3.5-27B-GPTQ-Int4 statistically indistinguishable from cloud DeepSeek-V4-Flash (difference 0.20 pp, 95% CI [-0.93, 1.32], p = 0.72) and Gemma-4-31B significantly worse; (3) expert review of 1,558 reference values (35.9%) finding no wrong labels and 99.68% inter-expert agreement (PABAK = 0.9936). Key limitations, candidly acknowledged by the authors: fully synthetic data, no value-level precision measurement on real model output, single-run paired tests, no retrieval ablation or Recall@K/MRR/NDCG, and a semantic chunking configuration that exposes only ~19% of each document to the LLM context - an untested competing explanation for the low coverage.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The paper contributes the first Vietnamese RoPA extraction benchmark: 32 synthetic organizations, 77 processing activities, 12 field groups, and 4,338 reference values, of which 35.9% were independently reviewed by two experts with no values rated Wrong.
- A locally deployed Qwen3.5-27B-GPTQ-Int4 on a single 24 GB GPU is statistically indistinguishable from cloud DeepSeek-V4-Flash on token coverage (difference 0.20 pp, 95% CI [-0.93, +1.32], p = 0.72), supporting data-sovereignty-compliant deployment.
- The proposed three-level reporting scheme cleanly separates scorer robustness (F1 = 0.9493 without LLM calls) from end-to-end coverage (50.04-55.25%) and reference-label reliability (PABAK = 0.9936), avoiding common metric conflation in RAG evaluations.
- All conclusions are explicitly limited to technical feasibility and coverage; the paper does not claim operational legal compliance.
Strengths
Three-level evaluation design prevents metric conflation
Section V and Figure 1 explicitly separate scorer robustness (Level 1, F1=0.9493 with no LLM), end-to-end coverage (Level 2, 50.04-55.25%), and label reliability (Level 3, PABAK=0.9936); the paper states outright that Level-1 numbers do not measure extraction performance - a discipline rare in RAG system papers.
Exceptional reproducibility infrastructure
Section IV.D freezes dataset v2.0, scorer v1.3, two git commits (c38e18c, 045547f), fixed seeds (20260705, 20260712), and a SHA-256-checked prompt; Table I publishes HNSW parameters (m=16, ef_construction=64), temperature 0.2, Pydantic 2.5.3; code, benchmark, and expert review records are public at github.com/tdhinhit/RoPA-demo.
Statistically literate reporting
Candid self-identification of a major confound
First-mover benchmark with verification
Weaknesses
End-to-end utility is unproven in both directions
Retrieval component never validated
Fully synthetic, author-authored data
Single-run experiments
Author-set KPI thresholds with an unnoticed-in-text FAIL
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a concrete regulatory pain point: Article 31 of Vietnam's Personal Data Protection Law (91/2025/QH15, effective 1 Jan 2026) and Decree 356/2025/ND-CP require Records of Processing Activities (RoPA), whose manual preparation is slow and expertise-heavy. It proposes RoPA Manager, a four-layer system combining hybrid retrieval (PostgreSQL tsvector lexical ranking, pgvector/HNSW dense search, weighted RRF) with locally deployed LLMs so PII never leaves on-premises infrastructure. Three claims: (1) the first Vietnamese RoPA benchmark (32 synthetic organizations, 77 processing activities, 12 field groups, 4,338 manually authored reference values, 35.9% expert-reviewed); (2) a three-level reporting scheme separating scorer robustness, end-to-end coverage, and label reliability; (3) feasibility of local deployment, with Qwen3.5-27B-GPTQ-Int4 statistically indistinguishable from cloud DeepSeek-V4-Flash. Unusually for a systems paper, claims are well-fenced: the authors repeatedly state conclusions are limited to technical feasibility and coverage.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a RoPA and why does it matter in Vietnam?
A Record of Processing Activities (RoPA) documents how an organization processes personal data. Vietnam's Personal Data Protection Law (91/2025/QH15), effective January 1, 2026, requires organizations to create and maintain one under Article 31, and manual preparation demands legal expertise and significant staff time.
How accurate is the RoPA Manager extraction system?
The paper reports end-to-end token coverage of 50.04-55.25% against reference labels - a recall-like measure. It explicitly does not measure value-level precision (the share of generated values that are correct), so real-world extraction accuracy is not yet established; the automated scorer's F1 of 0.9493 measures scorer robustness only, without any LLM.
Can locally deployed LLMs replace cloud models for this task?
On 32 paired synthetic scenarios, locally deployed Qwen3.5-27B-GPTQ-Int4 on a single 24 GB GPU showed no statistically significant difference from cloud DeepSeek-V4-Flash (0.20 percentage-point difference, 95% CI [-0.93, +1.32], p = 0.72), while Gemma-4-31B performed significantly worse (about 5 pp below the others).
Is the code and benchmark dataset publicly available?
Yes. The paper states that the source code, the Vietnamese RoPA benchmark, and both experts' review records are publicly available at github.com/tdhinhit/RoPA-demo, with frozen versions (dataset v2.0, scorer v1.3), fixed seeds, and SHA-256-checked prompts for reproducibility.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (three evaluation levels) could not be analyzed; only its caption text was extracted.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.