Escaping Python Dependency Hell: A Hybrid Replay-and-Repair Pipeline for Python Dependency Resolution
Veronica Poweska, Ariana Oyanguren, Jessica Pourleyli, Sourena Khanzadeh, Manar Alalfi
cs.AI · cs.MA · cs.SE
Quick verdict
PLLM+ lifts HG2.9K dependency repair to 51.9% from 40.4%, but 99.7% of fixes replay prior baseline solutions; honest yet thin on novelty and generalization.
Summary
The paper presents PLLM+, a hybrid pipeline for repairing Python dependency specifications so that previously failing code snippets execute in Docker containers. The pipeline is ordered by cost: it first infers a minimum interpreter version via static AST analysis, then replays known-good dependency configurations from a competition-provided solutions database built from earlier PLLM, PyEGo, and ReadPy runs, validates candidate versions against the live PyPI JSON API, and only then invokes a bounded LLM repair loop (Gemma 2 served via Ollama) with typed error classification and a Proposer/Critic debate. On the HG2.9K benchmark of 2,891 dependency-failing snippets, PLLM+ solves 1,500 (51.9%) versus 1,169 (40.4%) for the PLLM baseline, cutting average runtime from 368.7 to 71.8 seconds (median 364.5 to 21.5). Crucially, 1,495 of the 1,500 fixes (99.7%) come from database replay and only 5 from the LLM fallback; the authors honestly note that success rests on solutions mined from the same dataset. Main limitations: no statistical testing or variance reporting; the union of all three baselines' successful configurations is never reported, so replay's gain over trivially pooling prior solutions is unclear; unsolved cases are lumped into a coarse 'failed' status; the exact Gemma 2 variant is unspecified; and code is only promised upon publication.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- PLLM+ solves 1,500 of 2,891 HG2.9K dependency-failing snippets (51.9%) versus 1,169 (40.4%) for the PLLM baseline.
- Average per-snippet runtime drops from 368.7 s to 71.8 s (~5.1×), with the median falling from 364.5 s to 21.5 s.
- 1,495 of 1,500 successful fixes (99.7%) come from replaying the competition-provided solutions database; the LLM fallback adds only 5 fixes.
- Deterministic, cost-ordered reuse of validated configurations is the primary driver of performance, with LLM repair as a secondary fallback.
Strengths
Honest provenance audit (RQ3)
Section IV-B3 / Fig. 3 openly reports that 1,495 of 1,500 fixes (99.7%) come from solutions-database replay and only 5 from the LLM fallback — the paper resists overclaiming its agentic component, and this negative result about LLM utility is itself informative for the FSE-AIWare competition setting.
Cost-ordered deterministic validation explains the speedup
Stages 1-3 (AST interpreter lower bound, archive replay, live PyPI JSON-API validation that eliminates hallucinated versions before any Docker build) mechanically account for the 5.1× runtime reduction (368.7→71.8 s mean; median 364.5→21.5 s, §IV-B2) without invoking Gemma 2 for 1,495 snippets.
Clean head-to-head comparability
Candid threats-to-validity discussion
Weaknesses
Missing natural control: the baselines' union
No statistical inference
Elaborate but unvalidated LLM machinery
Coarse failure reporting for the proposed tool
Reproducibility gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.26952, 22 Sep 2026) introduces **PLLM+**, a cost-ordered hybrid pipeline for automatic Python dependency repair, evaluated on **HG2.9K** (2,891 dependency-failing GitHub snippets) within the FSE-AIWare competition framework. Three research questions structure the evaluation: success rate versus the PLLM baseline (RQ1), runtime efficiency (RQ2), and the provenance of fixes (RQ3). The headline claims, repeated consistently in abstract and §IV-B, are: 1,500/2,891 solved (51.9%) versus 1,169 (40.4%) for PLLM; average runtime cut from 368.7 to 71.8 seconds (≈5.1×; median 364.5→21.5 s); and 1,495/1,500 fixes (99.7%) sourced from solutions-database replay versus 5 from the LLM fallback.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PLLM+ and what problem does it solve?
PLLM+ is a hybrid pipeline that automatically repairs Python dependency specifications so code snippets that previously failed with dependency errors can run inside Docker containers. It tries cheap deterministic steps first — static interpreter inference, replaying known-good dependency configurations from a solutions database, and live PyPI validation — and only falls back to an LLM-based repair loop when those fail.
How much better is PLLM+ than the LLM-only baseline?
On the HG2.9K benchmark (2,891 dependency-failing snippets), PLLM+ solved 1,500 snippets (51.9%) versus 1,169 (40.4%) for the PLLM baseline, cutting average runtime from 368.7 to 71.8 seconds per snippet. However, 1,495 of its 1,500 fixes came from replaying the solutions database; the LLM contributed only 5 fixes.
Does the LLM actually do the repair work in PLLM+?
Almost never in this evaluation. The multi-agent LLM loop (Gemma 2 with Proposer/Critic debate) solved just 5 of 1,500 snippets (0.3%). The paper's key finding is that deterministic replay of previously validated configurations dominates, with the LLM serving as a rarely used fallback.
Is the code and dataset available?
The HG2.9K dataset and baseline artifacts come from the public FSE-AIWare competition. The authors name a GitHub repository (vpoweska/fse-aiware-python-dependencies) but state the code will be released only upon publication; at the time of this review it was not yet available.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 (architecture, success bars, fix-source split, runtime distribution) are not analyzable from the extracted text.
- Table I appears garbled in extraction (caption spacing, possibly misaligned counts); the reported arithmetic mismatch may be extraction noise rather than a paper error.
- Derived baseline solve counts (PyEGo ≈1,302; ReadPy ≈1,365) are computed by subtracting Table I failure counts from 2,891 and inherit that table's uncertainty.
- Review is text-only; the named repository and competition artifacts were not inspected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.