Crypto Accounting Bench: Evaluating Frontier and Open-Weight Models on Crypto-Asset Accounting Tasks
Kareem Khattab, Omar Khattab, Mohamed Ibrahem
cs.AI · cs.CL
Quick verdict
CAB benchmark uses real ledger entries: 12 LLMs reach 77.4% Mean Score but only 56.8% best Pass@3; strong design, yet no CIs and an unvalidated LLM judge.
Summary
The paper introduces Crypto Accounting Bench (CAB), a benchmark of 118 tasks from 7 pseudonymized real organizations requiring models to reconstruct the complete journal entry an organization actually posted for a crypto-asset transaction. Tasks supply transaction mechanics, quantities, wallet and legal-entity context, counterparty evidence, related legs, recurrence, tax-lot evidence, and the full chart of accounts; the posted entry is withheld. Twelve frontier and open-weight models run 3 attempts per task (4,248 trajectories) under an identical agentic harness (bash tool, up to 12 turns), graded by frozen task-local 6-criterion rubrics with family-specific weights, a GPT-5.5 judge, 7 required structural gates, and independent deterministic diagnostics. Results: Gemini 3.7 Flash leads Mean Score at 77.43%; Kimi K3 leads Best@3 (82.37%) and Pass@3 (56.78%). Best-of-3 diagnostics show 97.8% base-amount agreement against 56.3% deciding-account accuracy, locating failures in account selection and entry composition rather than arithmetic. Limitations: no confidence intervals, so the 2.10-point spread among the top nine models is unsettled; judge accuracy is not independently validated; the public dataset is a transformed derivative of the evaluation corpus, with mappings withheld; some rubric families have tiny samples (realized gain/loss n=5).
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- CAB is the first benchmark in this domain whose answer keys are complete journal entries that real crypto-holding organizations actually posted, rather than synthetic generations.
- Across 12 models and 4,248 trajectories, models achieve high partial credit (Mean Score 71.92-77.43%) but no model reliably composes the full recorded entry (best Pass@3 56.78%).
- Failures concentrate in organization-specific account selection and complete-entry composition, not arithmetic: base-amount agreement is 97.8% while deciding-account accuracy is 56.3% on best-of-3 diagnostics.
- The same failure pattern holds across every evaluated model family, consistent with a shared limitation rather than a single pipeline defect.
Strengths
Real-ledger answer keys distinguish CAB from synthetic predecessors
Targets are the 246 non-zero reference lines that 7 real organizations actually posted (§3.1, §4.1), vs FinBalance's deterministic generator and APEX-Accounting's synthetic company worlds (§2); median 569 chart accounts per task against only 141 distinct accounts used makes the distractor space genuinely wide (§4).
Layered grading with explicit authority separation and calibration
Frozen per-task 6-criterion rubrics (Table 1 weights), a judge confined to PASS/FAIL verdicts, local weighted summation, and 7 gates of which 6 are deterministic (§5.2). Appendix A quantifies gate discipline: 92 of 1,917 perfect-rubric attempts still fail a gate; the 999.99 corruption test lands at 0.85-0.90 while only the material gate fires; gates fire on no correct answer across all 118 references.
Clean diagnostic decomposition of the failure mode
Honest metric and uncertainty framing
Per-attempt run identity enables exact audit
Weaknesses
No statistical inference despite a 2-decimal leaderboard
Judge validity is asserted, not established
Public release cannot reproduce the reported numbers
Fragile family-level comparisons on tiny strata
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CAB evaluates one deliberately narrow operation: reconstructing the complete journal entry a real organization posted for a crypto-asset transaction. The population is 118 tasks from 7 pseudonymized organizations, 246 non-zero reference lines, median 569 chart accounts per task against only 141 distinct accounts actually used (§4) — a wide distractor space. The paper claims three things: (1) CAB is the first benchmark here whose answer keys are posted-ledger entries rather than generator output (§2 contrasts FinBalance, APEX-Accounting, GDPval); (2) 12 frontier/open models reach 71.92-77.43% Mean Score but at most 56.78% Pass@3 over 3 attempts; (3) failures sit in account selection and entry composition, not arithmetic. The descriptive-target framing — the target is what the organization posted, not a normative ideal (§1, §9) — is clearly scoped and honestly defended, though it means a defensible alternative treatment scores zero.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Crypto Accounting Bench (CAB)?
CAB is a benchmark of 118 tasks drawn from 7 pseudonymized real organizations. Each task gives a language model the transaction evidence and the organization's full chart of accounts, and asks it to reconstruct the exact balanced journal entry the organization actually posted for a crypto-asset transaction; the answer key is withheld from the model.
Which AI models performed best on CAB?
Per the paper, Gemini 3.7 Flash led Mean Score at 77.43%, while Kimi K3 led Best@3 (82.37%) and Pass@3 (56.78%). All 12 evaluated models scored between 71.92% and 77.43% Mean Score; the best full-entry Pass@3 was 56.78%, meaning roughly half of tasks had no fully correct attempt across 3 tries even for the top model.
Why do LLMs struggle with crypto accounting tasks according to the paper?
Diagnostics show models get numbers and direction right — 97.8% base-amount agreement and 98.1% direction accuracy on best-of-3 reads — but select the wrong organization-specific account in a large share of tasks (56.3% deciding-account accuracy). Failures are in accounting judgment (choosing accounts from ~569-account charts using wallet, entity, counterparty, and history evidence) and in composing every required line, not in arithmetic or JSON formatting.
Is the CAB dataset and code publicly available?
Yes, partially. A transformed educational derivative of the same 118 cases (renamed identifiers, rescaled values, shifted dates) is on Hugging Face with reference answers, and the evaluation toolkit (runner, scorer, validation, leaderboard generator) is on GitHub. The exact evaluation corpus, real organization identities, and transformation mappings are not released, so the paper's exact numbers cannot be re-derived from the public files.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are text-extracted labels only; visual encodings (e.g., Figure 4 quadrant plot, Figure 5 scatter) were inferred from captions and surrounding text.
- Model names in the text (e.g., 'Gemini 3.7 Flash', 'GPT-5.6 Sol') are as printed in the extracted text and could not be checked against public release versions.
- The evaluation corpus is private and transformed for public release, so no external verification of the exact reported numbers is possible from this text alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.