When Agent Metrics Measure Different Things: An Evidence-Grounded Audit of the Praxa AI Pipeline
Stefan G. Creadore, Peyton Woakz
cs.SE · cs.AI
Quick verdict
Shows agent metrics can be numerically correct yet measure the wrong construct; rigorous, candid audit of one private pipeline, but narrow and self-audited.
Summary
This paper is a retrospective measurement audit of Praxa, a private AI agent pipeline, testing whether numerically correct metrics actually measure the constructs their labels imply. Pinned to a specific Git revision, the authors examine three cases. First, a historical 139-case offline routing report shows 112 passes and 27 failures (80.58%) alongside zero gating failures, reconciled by documented known-gap exemptions from the development gate. Second, an identifier-free export of 8,843 tool-attempt rows contains 8,395 recorded durations, of which 121 equal the signed 32-bit maximum and carry abandoned-client labels; inspected database code clamps elapsed lifecycle age. The pooled 99th percentile is 2,147,483,647 ms versus 38,118.31 ms among server-observed completed calls — a stratum contrast, not a treatment effect. Third, a single-trajectory compaction pilot's 94.39% follow-up input reduction becomes 46.54% when the trigger call is included. Verification includes reproduction of all descriptive calculations, 91 timing statistics cross-checked by a separate rational-arithmetic implementation, 13 scoring-function conformance tests and 12 verifier tests, plus finite-completion bounds for missing durations (e.g., 92.83–98.65% below 30 seconds in the server/completed stratum). Main limitations: all evidence comes from one private system audited by its own affiliates; historical provider runs were not independently reproduced; the compaction pilot is one trajectory; the repository remains private. No generalization, superiority, or statistical-significance claims are made.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The historical 139-case routing report's zero gating failures coexist with 27 case failures because known gaps are explicitly exempted from the development gate, so gate status must not be reported as an evaluation score.
- The operational latency field mixes event semantics: 121 abandoned-client calls saturate at the signed 32-bit maximum, so the pooled p99 of 2,147,483,647 ms measures lifecycle age, not active execution latency (server/completed p99: 38,118.31 ms).
- The compaction pilot's 94.39% follow-up-only input reduction falls to 46.54% across the trigger plus follow-up calls, demonstrating that the accounting boundary changes the reported answer.
- All retained descriptive statistics reproduce under a separate weighted rational-arithmetic implementation (91 cells, 1e-12 relative tolerance) and 25 passing software tests, while finite-completion bounds (e.g., 92.83%–98.65% at 30 s in server/completed) delimit what missing durations leave unknowable.
Strengths
Unusual claim-discipline infrastructure
An evidence registry (E01–E15) with Git blob SHAs and PostgreSQL MD5 digests, a six-level claims ledger (VERIFIED through REJECTED, §2.3), and pervasive explicit NOT ESTABLISHED statements (§2.1, §9) keep every conclusion tethered to what the retained artifacts can support.
Dual-path computational verification
91 timing-statistic cells are cross-checked by an independent Python rational-arithmetic implementation at 1e-12 relative tolerance (§4.3); the original computeAgenticMetrics source (blob 01bf2308…) is hash-verified and executed under Node 22.16.0 in 13 conformance tests (Table 3); 12 verifier tests include a same-total/different-distribution corruption rejection (Appendix D); every non-null frequency cell is reconciled payload-to-CSV (§5.2).
Concrete, quantified measurement-boundary findings
Principled missing-data treatment
Active falsification attempts
Weaknesses
Single private system, self-audited
Thin empirical substrate where the findings matter most
Limited novelty, acknowledged
Non-atomic, mutable-source extraction
No independent human review
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a retrospective **measurement audit** of Praxa, the authors' own private AI agent pipeline, pinned to commit `83d29cf…` (E01) with historical artifacts at other revisions. Four RQs (§2.2): what separates case correctness from development gating (RQ1); what event the recorded latency measures (RQ2); how sensitive the compaction comparison is to its accounting boundary (RQ3); which stronger claims remain unresolved (RQ4). The thesis — *numerically correct metrics can measure a different construct than their labels imply* — is instantiated in three cases:
1. **Gate ≠ case denominator (§5.1):** the historical 139-case report shows 112 passes / 27 failures (80.58%) yet **zero gating failures**, reconciled by documented known-gap exemptions. Legacy cases are 63/139 (45.32%) and pass 63/63, versus 49/76 (64.47%) for the rest — a composition identity the paper refuses to read as improvement.
2. **Lifecycle age ≠ execution latency (§5.3, Table 4):** of 8,843 attempt rows, 8,395 have durations and 448 (5.07%) are null. Exactly **121 values equal 2,147,483,647 ms** (the signed 32-bit maximum, 1.44% of non-null), all client-reported/failed/abandoned, with uncapped intervals spanning up to 4,097,646,570 ms (~24.86 days); inspected SQL clamps `finished_time − started_at` to the integer cap. Pooled p99 (2,147,483,647 ms) vs server/completed p99 (38,118.31 ms) is correctly labeled a stratum contrast.
3. **Accounting boundary (§5.4, Table 6):** the compaction pilot's follow-up input falls 10,695 → 600 tokens (−94.39%); including the trigger (10,610 → 10,790) gives 21,305 → 11,390 (−46.54%). Two-call durations *increase* 27.64% (2,902 → 3,704 ms).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
That agent evaluation metrics can be arithmetically correct while measuring a different construct than their label implies. The paper shows three cases in one private AI agent pipeline: a routing gate reporting zero failures while 27 of 139 cases fail (known gaps are exempted from the gate); a latency field whose pooled 99th percentile is a 32-bit integer cap caused by abandoned client calls; and a token-saving figure that shrinks from 94.39% to 46.54% once the trigger call is included in the accounting.
Is Praxa's code or the paper's data publicly available?
No. The audited repository is private (GitHub praxa-labs/Praxa-AI, access owner-controlled). The paper ships a companion archive with the submission containing frequency CSVs, verification scripts, tests, and receipts with checksums, but the authors themselves note that source hashes do not prove public accessibility, and the source database cannot be re-queried by outsiders.
Does the paper claim the Praxa agent is bad or failing?
No. The paper explicitly states that 'not established' describes the limits of the audit, not a finding that a capability is absent or impossible, and it makes no claims of superiority, statistical significance, or generalization to other systems. Its point is that certain metrics, as recorded, cannot support broader interpretations.
Why is the reported 99th-percentile latency 2,147,483,647 milliseconds?
That number is the signed 32-bit integer maximum. Exactly 121 of 8,395 recorded durations (1.44%) hit this cap, all in the client-reported, failed, abandoned-call stratum, and inspected database code clamps elapsed lifecycle age to this value. They represent the age of abandoned calls (up to about 24.86 days of wall-clock span), not active tool execution; among server-observed completed calls the p99 is 38,118.31 ms.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure regions interleave with body text during PDF extraction (e.g., axis ticks merge into the text near Table 2 and Figures 1–4), so chart labels are garbled; all tables, formulas, and body text cited in this review are intact and verbatim from the extraction.
- The provided text ends cleanly at reference [11] with page marker 22; no evidence of end-of-document truncation.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.