Automated Event Log Generation from Unstructured Text Using Finetuned LLMs
Maximilian Seeth, Gabriel Marques Tavares, Daniel Schuster
cs.AI
Quick verdict
Finetuned open-weight LLMs convert synthetic texts into XES event logs for process mining; results are strong on synthetic data, real-world validation missing.
Summary
The paper addresses the scarcity of structured event logs for process mining by finetuning open-weight LLMs to translate unstructured text into XES event logs. Because no suitable text-to-log dataset exists, the authors synthesize one: they sample 2,887 traces from the Road Traffic Fine Management log and 400 from Sepsis Cases, and use three frontier models (GPT-5.1, Qwen3-max, Mistral-medium) with personas and varied prompts to render traces as synthetic police reports and discharge summaries. They finetune Llama3.1-8B, Ministral-8B, and Qwen2.5-7B with LoRA (rank 16) and compare against zero- and one-shot prompting. SFT dominates: the finetuned Llama3.1-8B emits 430 of 434 well-formed XES test traces, variant counts land near the original 43 (zero-shot yields almost nothing valid), and Levenshtein/Kendall's Tau scores reach ~0.99, while Heuristics-Miner process F1 approaches the original log's 0.90. Generalization experiments cover the medical domain and an unseen text generator, though Inductive-Miner F1 there stays low (0.09 to 0.13-0.20). Main limitations: evaluation is fully synthetic (texts are generated from the very traces the models must recover), no real-world documents are tested, output is XES-only, and no significance testing supports the comparisons.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Supervised finetuning on a synthetic text-to-log corpus substantially outperforms zero- and one-shot prompting for generating well-formed, faithful XES event logs with 7-8B open-weight LLMs.
- Finetuned Llama3.1-8B and Ministral-8B produce traces whose activity order and content nearly match the originals (Levenshtein/Kendall ≈0.99), with variant counts close to the original log's 43.
- Process models mined (Heuristics Miner) from SFT-generated logs approach the original log's F1 of 0.90, though Inductive-Miner precision never reaches the original 0.40.
- The learned text-to-log mapping transfers across domains (police to medical) and to texts from an unseen frontier generator, though with a larger remaining gap to original-log quality.
- No suitable public text-to-log dataset exists, making the authors' synthetic corpus generation pipeline itself a contribution.
Strengths
Clear gap analysis and positioning
Table 1 systematically contrasts six prior LM-for-PM works across nine dimensions (LM type, SFT, RL, log vs model extraction, trace vs process evaluation); the paper is, per this table, the first to combine decoder LLMs + SFT + log extraction + both evaluation levels.
Dual-level evaluation framework
Sections 5.2-5.3 pair trace metrics (well-formedness, variants, hallucinations, Levenshtein, Kendall's Tau, attribute overlap) with process-level mining (Inductive and Heuristics Miner, fitness/precision/F1); the miner-dependent conclusions (e.g., Inductive F1 0.51 vs Heuristics near 0.90) itself demonstrates the framework's value.
Transparent synthetic-data quality control
Practitioner-oriented design and honest negative reporting
Weaknesses
Fully synthetic evaluation loop
Weak cross-domain transfer underplayed in conclusions
Thin statistical reporting
Metric design choices that flatter results
Easy primary task setting
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets process mining's best-known bottleneck: event logs must be structured, while much organizational knowledge (tickets, manuals, reports) is text. The authors position open-weight decoder LLMs as 'data translators' from unstructured text to XES event logs and make four contribution claims (Section 1): an open-weight text-to-log approach; evidence that SFT — not prompting — is the pre-condition for reliable logs; a new synthetic text-to-log dataset; and a dual trace/process evaluation framework. Table 1 positions the work as the first combining decoder LLMs, SFT, prompt variation, log extraction, and both trace- and process-level evaluation. Within its synthetic scope the claims are largely supported, but every evaluation input is model-generated, which bounds what they can prove.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper do?
It finetunes small open-weight language models (Llama3.1-8B, Ministral-8B, Qwen2.5-7B) to translate unstructured text such as reports into structured XES event logs that process mining tools can analyze, and shows the finetuned models beat zero-shot and one-shot prompting by a wide margin.
Is the data real?
No. The training and test texts are synthetic: three frontier LLMs (GPT-5.1, Qwen3-max, Mistral-medium) were used to rewrite traces from two real public event logs (Road Traffic Fine Management, Sepsis Cases) into fictional police reports and hospital discharge summaries. Performance on authentic documents is not yet evaluated.
Does finetuning actually beat prompting?
Yes, decisively in this setup. In zero-shot mode only Qwen2.5-7B produced any valid XES output (2 traces); one-shot prompting fixed syntax but not fidelity, while finetuned models produced 430/434 well-formed traces with ~0.99 trace-similarity scores and process models close to the original log under the Heuristics Miner.
Is code and data available?
The authors share training and evaluation scripts plus dataset samples on GitHub (omseeth/Automated-Event-Log-Generation-from-Unstructured-Text-Using-Finetuned-LLMs). Full re-derivation requires regenerating texts via paid frontier APIs and single-GPU LoRA training; only samples, not the full corpus, are stated as released.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Numeric values plotted in Figures 2, 4, 5, 6, 7 and 8 are only available through the in-text mentions; the full curves, bars and CI bands could not be independently verified from the extraction.
- SFT hyperparameters are deferred to the GitHub repository ('We share hyperparameter details with the repository'), so the training configuration could not be verified from the paper text itself.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.