Scratchy: Visual-Scratchpad Multimodal Reasoning for Cryptographic Proof Generation in EasyCrypt
Yupeng Ren, Zhaoxuan Li, Rui Zhang
cs.CR · cs.AI
Quick verdict
Scratchy: structured visual proof states lift EasyCrypt compile rates to 21.1% vs 9.4% control; promising, but visual-modality contribution not fully isolated.
Summary
The paper presents Scratchy, a pipeline that helps multimodal LLMs generate machine-checked cryptographic proofs in EasyCrypt. A Structured Natural-Language Proof Sketch is deterministically compiled into a typed proof-relation graph, which a structure-preserving visual compiler renders as a formula-rich image; a multimodal model then writes a proof fragment that EasyCrypt must parse, type-check, and close. The authors also introduce Scratchy-eval, 114 tasks (64 free-form proof-generation targets, 50 multiple-choice diagnostics) drawn from 37 pinned official EasyCrypt files across 26 sources. Under six paired representation conditions and four API models (Claude Opus 5, Gemini 3.7 Flash, GPT-5.6-Sol, Kimi K3), the professional graph compiles 54/256 fragments (21.1%), versus 24/256 (9.4%) for a shuffled-edge graph and 11/256 (4.3%) for coarse relation text; paired source-group bootstrap gains are +16.8 points (95% CI 8.8-24.6) and +11.7 points (3.5-19.8). Gains concentrate on complex cryptography (33.3%) and reduction paths (21.4%), while the shuffled graph wins on foundations and program analysis. Limitations: absolute pass rates remain low; MCQ accuracy sits near ceiling (97.0-99.5%); the structured sketch, including tactic routes, is assumed as input with unexplained provenance; the visual channel is not disentangled from the extra textual content it carries; and no public code link is given.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Representing proof relations as a structured, relation-faithful visual state improves EasyCrypt proof generation across four frontier multimodal models, reaching a pooled 21.1% compile pass rate versus 9.4% for a shuffled-edge control.
- The gain depends on relation fidelity rather than image presence, since generic diagrams and structure-corrupted graphs yield markedly smaller benefits.
- Scratchy's advantages concentrate on protocol-specific reduction paths and relational equiv judgments, where dependencies span multiple programs.
- Scratchy-eval, a 114-task benchmark from 37 pinned official EasyCrypt files, enables controlled representation studies judged by the machine verifier rather than exact string match.
Strengths
Verifier-grounded scoring avoids exact-match bias
Generation is judged solely by EasyCrypt acceptance in a pinned environment; the paper reports that only 10 of 54 professional-graph successes exactly match official proofs, so 44 valid alternative proofs are credited (Section 5.3), and the machine-checked contrast in Section 6.1 (Listing 2) shows a genuinely different accepted loop proof.
Ablation design isolates relation fidelity
Six paired conditions include a shuffled-edge graph and a generic diagram as visual controls; the paired W/L/T analysis with source-group bootstrap over 26 groups (Table 4: PG vs SG 39W/9L/208T, +11.7pp, CI [3.5, 19.8]) respects within-source dependence.
Unusually honest disaggregation
Provenance-preserving benchmark construction
Cross-model consistency plus a mechanistic diagnostic
Weaknesses
Visual channel not separated from information content
SNPS provenance is unaddressed
Statistical breadth is thin
The 50 MCQ diagnostics carry almost no signal
No released artifacts or contamination discussion
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This method-plus-benchmark paper targets machine-checked cryptographic proof generation in EasyCrypt. The authors argue that linear textual interfaces hide proof-theoretic dependencies (game hops, invariants, assumptions, advantage bounds) distributed across programs, and that rendering these dependencies as a structured image can help multimodal LLMs. Scratchy takes a Structured Natural-Language Proof Sketch (SNPS), deterministically compiles it into a typed proof-relation graph (five node types: threat model, program facts, obligations, rules, goal), and renders it via a structure-preserving visual compiler (Algorithms 1-2) into a formula-rich image; a multimodal model then produces a proof fragment that is reconstructed into source and adjudicated by the EasyCrypt verifier (Eqs. 1-2, Section 4). Three central claims: (1) structured visual proof states improve generation beyond text and corrupted-visual controls; (2) relation fidelity, not image presence, drives the gain; (3) Scratchy-eval enables controlled representation studies with verifier-judged scoring.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Scratchy system for EasyCrypt proofs?
Scratchy is a pipeline that converts a structured natural-language proof sketch into a typed proof-relation graph, renders that graph as a formula-rich image, and lets a multimodal LLM write an EasyCrypt proof fragment guided by the image; the EasyCrypt verifier alone decides whether the reconstructed source is accepted.
How much does the visual proof state improve results?
Across four commercial multimodal models and 64 proof-generation tasks, the professional graph compiled 54/256 fragments (21.1%) versus 24/256 (9.4%) for a shuffled-edge graph and 11/256 (4.3%) for coarse relation text; the paired gain over the best visual control was +11.7 percentage points (95% CI 3.5-19.8).
What is in the Scratchy-eval dataset?
114 tasks built from 37 pinned official EasyCrypt files across 26 sources: 64 free-form proof-generation targets (55 lemmas, 7 equivalences, 2 expectation-Hoare judgments) and 50 multiple-choice diagnostics on static validity and semantic recognition of EasyCrypt constructs.
Is Scratchy's code or dataset publicly available?
The paper describes a release record retaining task provenance, graph metadata, and verification artifacts, but the provided text gives no repository URL, so no public code or dataset link is confirmed.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 2(b) layerwise values are conveyed only through embedded axis labels and captions; per-layer numeric expert-utilization values are not recoverable from the text extraction, so those claims were assessed from the authors' description.
- The metadata abstract supplied with the task ends mid-sentence, but the in-paper abstract is complete in the extraction; the body, appendix (Data Card, Listings 3-4), and references through Zhou et al. (2026) show no truncation.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.