mimeo: Compiling Public Expert Corpora into Agent Skills and Testing What Transfers
cs.AI
Quick verdict
Careful, honest study of expert-profile agent skills: knowledge access gain is real, persona not unique, judgment transfer unproven at test ceiling.
Summary
mimeo is an open-source Python tool that compiles a named person's public web record into an agent skill file (SKILL.md/AGENTS.md); a quotation-matching stage rejected 13.2% of extracted spans across eight logged builds (37.8 model calls, 7.1 minutes each). Using one coding-agent harness (claude-sonnet-5), four expert profiles (Karpathy, Sutton, He, Dean), and four LLM judges from different vendors, the paper tests three claims separately. Knowledge access: on 20 obscure single-record questions mimeo scored 20/20 while no closed-book condition exceeded 10/20; however, BM25 over the same cached pages answered 15-17, and the authors concede the probes were drawn from the corpus mimeo was built from, so its ceiling was guaranteed by design. Grounding: personas written from model memory misstated a documented position on 1-4 of 20 answers under every grader; the plain agent and mimeo never did. Persona: every persona is identifiable on short prompts, a from-memory profile is at least as identifiable as mimeo, and attaching task material lowers identification by 18-23 points. Judgment transfer: both tests hit ceiling (94-97% planted-issue coverage; 94-100% on 16 scenarios), leaving transfer unmeasured, and the 'sounds like the expert' score flipped with the judge. Limitations: four experts, one agent model, author-written tasks and keys, no expert human raters, and experiment runs plus analysis code are absent from the public release.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Corpus access via a mimeo file clearly improves recall of obscure documented wording over closed-book answering (20/20 vs at most 10/20 on single-record questions).
- A simple BM25 keyword search over the same cached pages answers 15-17/20, so any benefit of distillation over lookup remains unresolved.
- Personas written from model memory misstate a documented position on 1-4 of 20 answers under every grader, while the plain agent and mimeo never do.
- Personas are recognizable on short open prompts regardless of grounding; a from-memory profile is at least as identifiable as mimeo, and task material lowers identification by 18-23 points.
- Judgment transfer is not demonstrated: both transfer tests hit their ceiling, and the keyless 'sounds like the expert' measure changes with the judge model.
Strengths
Claim separation with matched controls
Section 3 and Table 1 decompose 'give an agent an expert file' into knowledge access, recognizable persona, and judgment transfer, each tested against no-skill, one-line persona, and from-memory-profile controls plus a BM25 arm — avoiding the single 'knowledge not steering' blur common in this literature.
Rare candor about design limits
Appendix E.1 states outright that E4's 20/20 ceiling 'follows from the design' because probes were drawn from mimeo's own build corpus; Appendix D.4 documents 2/36 invented quotations when matching is disabled, one misattributed quote even with it on, and C.2 reports four gallery bundles with dangling source identifiers.
Above-norm statistical discipline
A retrieval control that undercuts the tool's own headline
Judge-instability documented as a result
Weaknesses
The titular question is left unmeasured by construction
Small, single-source evidence base
E4 design circularity caps what the headline shows
Results tree not released
Data-handling blemishes in an attribution paper
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
mimeo (K-Dense, Inc.; arXiv 2609.00453) is both a tool paper and an evaluation study. The tool compiles a named person's public web record into SKILL.md/AGENTS.md agent-skill files via eight cached stages: identity disambiguation, eight-category source discovery (27 query templates, 25-source budget), fetching, per-source distillation, cross-source clustering, fuzzy quotation-span matching (0.82 threshold), authoring, and a critique-refine loop. The paper's central move is decomposing "give an agent a file about an expert" into three separable claims — knowledge access, recognizable persona, judgment transfer — and testing each with matched controls (Table 1). Its stated conclusions: corpus access clearly helps obscure recall; BM25 recovers most of that gain so distillation's value is unresolved; grounding prevents misstatements that from-memory personas make; personas are recognizable but not unique to sourced profiles; and judgment transfer is not demonstrated.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the mimeo tool do?
mimeo is an open-source command-line tool that takes a person's name, searches their public web material (essays, talks, interviews, papers), checks extracted quotations against cached source text, and writes an agent skill file (SKILL.md or AGENTS.md) that a coding agent can load. In the paper's logged builds the quotation check rejected about 13.2% of extracted quotations, and each build averaged 37.8 model calls and 7.1 minutes.
Does loading an expert profile file make an AI agent decide like that expert?
This paper could not demonstrate that. Both judgment-transfer tests hit a ceiling where every condition, including no skill at all, found 94-97% of planted engineering issues and scored 94-100% on 16 application scenarios, so transfer was unmeasured. The one clear grounding benefit: from-memory personas misstated documented positions on 1-4 of 20 answers, while the sourced file and the plain agent never did.
Is a distilled skill file better than just searching the source documents?
Unresolved in this study. On 20 obscure quotation-heavy questions mimeo scored 20/20 while a simple BM25 keyword search over the same cached pages scored 15-17; the difference was not statistically significant (McNemar p=.063-.25), so the sample cannot tell whether the static file beats looking passages up on demand.
Are the generated expert profiles verified and accurate?
No. The paper is explicit that a text match makes a quotation 'attested, not verified' — it cannot prove who said the words or that the surrounding claim follows. The authors report one matched quotation credited to the wrong source and four of 22 released profiles containing source identifiers that point to nothing.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with marked [...sampling gap...] omissions (parts of Sections 4-5 and of Appendices A, D.2, F and L were skipped); all numbers cited in this review come from the visible text, and judgments touching only the skipped portions were avoided or softened.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.