Graph Memory for LLM Agents: At What Cost? A Comparative Evaluation of Query, Ingest, and Update Performance Across Graph Database Engines
Donald Nguyen, Gurbinder Gill, Hadi Ahmadi, Christopher J. Rossbach
cs.DB · cs.AI
Quick verdict
Vendor benchmark of 8 graph engines incl. the authors' own Corvic AI; refreshingly transparent, but internal number inconsistencies and no variance reporting.
Summary
This is a vendor-affiliated benchmark: the authors compare Corvic AI, the columnar query engine behind their own ontology product, against seven graph-capable systems (LoraDB, Ladybug, DuckPGQ, Memgraph, Neo4j, HugeGraph, FalkorDB) on a synthetic biomedical-shaped property graph of 1.02M nodes and 5.34M rows, running twenty query shapes at three scales (1K to 1.02M nodes). At full scale, Corvic AI posts the lowest latency geomean (2.19 ms) among engines completing all 20 shapes correctly; Ladybug wins on bounded-neighborhood shapes, while Memgraph (0.90 ms) and FalkorDB (0.85 ms) beat Corvic AI (2.47 ms) on point updates. The largest effect is ingest throughput, spanning roughly 860x (5.0k-4.3M rows/s); a total-cost-of-ownership crossover model argues ingest dominates below ~10^5 queries per refresh (worked example: q* of about 38,304 vs Memgraph). A DuckPGQ-versus-handwritten-SQL analysis attributes a 2.26x gap on graph shapes to plan choice. Limitations include a single synthetic dataset, unspecified hardware, no repetition counts or variance, undisclosed per-shape correctness failures, internal inconsistencies (Ladybug 3.97 vs 4.08 ms; DuckPGQ's 16.7M rows/s batch throughput exceeding Corvic AI's 14.8M despite the conclusion's claim), and no released code or workload.
Review complete
Assessment at a glance
6
risk flags identified
● high 3
● medium 2
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- No engine is categorically fastest: the query shape, not the 'graph database' label, predicts which system wins (Sections 5.4, 8).
- Bulk-ingest throughput varies roughly 860x (5.0k to 4.3M rows/s) and, per the TCO model, dominates total cost below roughly 10^5 queries per data refresh (Section 5.5).
- DuckPGQ's 2.26x slowdown on graph shapes stems from query-plan choice (a GRAPH_TABLE three-way join), not from SQL/PGQ syntax (Section 5.3).
- Native adjacency storage helps only on bounded local traversals, while columnar scan-and-join execution wins on shapes touching a large fraction of the graph (Section 5.4).
Strengths
Joint reporting plus a TCO model makes the trade-offs decision-ready
Sections 4.5-5.5 report latency, ingest, point-update, batch-update and correctness together; the Eq. 5-6 crossover is arithmetically verifiable from Table 1 (61.4 s vs 1.24 s ingest; 0.90 vs 2.47 ms gives q* of about 38,300), and Section 6 shows how each metric reported alone would support a different, misleading recommendation.
Plan-sensitivity isolation is the paper's best experiment
Section 5.3 explains DuckPGQ's 2.26x gap on graph-family shapes (5.70 vs 2.52 ms) via a concrete GRAPH_TABLE plan that joins vertex tables and scans all 330k protein rows to re-verify a label the edge's foreign key already identifies — a mechanism, not just a ranking, and correctly scoped to the current optimizer rather than SQL/PGQ itself.
Unusual transparency about conflict of interest and negative results
Shape-level wins against the authors' own system are quantified
Weaknesses
No statistical dispersion or repetition protocol
Internal number inconsistencies
Geomeans over non-identical shape sets
Reproducibility gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a vendor benchmark: three authors are from Corvic AI Research, and the subject system is the query engine behind their own ontology product. Eight systems are tested — Corvic AI, LoraDB, Ladybug, DuckPGQ, Memgraph, Neo4j, HugeGraph, FalkorDB — with PuppyGraph evaluated but excluded (Appendix A). The dataset is a synthetic, biomedical-shaped property graph of 1.02M nodes and 5.34M node-plus-edge rows, materialized at 1K, 10K, and 1.02M nodes (Section 3.1); the taxonomy references STaRK-PRIME/PrimeKG, and the authors stress that no real data is sampled (Section 7). Twenty query shapes, each implemented in each engine's native language (Section 3.2), span neighborhood lookups, bounded paths, set operations, anti-joins, aggregation, top-k, temporal filters, scans, and hash/self joins. Claimed contributions: the workload, joint measurement of latency, ingest, update and correctness, a plan-sensitivity decomposition, and a TCO crossover model. The headline claims — no system is categorically fastest, and ingest cost dominates below roughly 10^5 queries per refresh — are deliberately modest, and Section 1.1 explicitly rejects best-database framing. Two headline numbers nonetheless need scrutiny: the 10^5 threshold is generalized from a single engine pair, and Section 8's claim that Corvic AI led on batch-update throughput is contradicted by Table 1 (DuckPGQ 16.7M vs Corvic AI 14.8M rows/s).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Is Corvic AI faster than Neo4j and other graph databases?
In this benchmark, on a synthetic 1.02M-node graph with twenty query shapes, Corvic AI's latency geomean was 2.19 ms versus Neo4j's 6.92 ms, and it loaded data roughly 800x faster (4.3M vs 53k rows/s). But this is a vendor benchmark of the authors' own product: Corvic AI was slower at single-record updates than Memgraph and FalkorDB, and the paper itself stresses that no system wins everywhere.
Do you need a graph database for graph-shaped queries?
The paper's answer is no, not categorically. The native graph engine tested (Ladybug) won only on bounded neighborhood queries, while columnar SQL engines won on scans and joins; DuckPGQ's slower results were traced to a specific query-plan choice, not to SQL/PGQ syntax itself.
When does data-ingest speed matter more than query speed?
The authors model total cost as ingest time plus the number of operations times per-operation latency; two engines' costs cross at q* = (ingest-time difference) / (per-operation latency difference). In their Corvic-versus-Memgraph example the crossover is about 38,000 operations per data refresh — below that volume, the faster-loading engine wins overall. The threshold differs for each engine pair.
Is the benchmark's code or dataset publicly available?
No. The paper releases only its results table (Appendix A). The query workload, data generator, and measurement harness are not published, and the hardware is unspecified, so the reported numbers cannot currently be independently reproduced.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The abstract is truncated mid-sentence in the extracted text; abstract-level claims were cross-read against body sections.
- Figures 1-6 are not analyzable from text; figure-only results (e.g., Figure 2 scaling curves, Figure 6 heatmap values) were assessed only via numbers quoted in prose.
- Engine identities and versions (e.g., Ladybug, LoraDB) could not be externally verified from the provided text.
- Per-shape correctness-failure detail (which shapes failed for 18/20 and 19/20 engines) is absent, limiting correctness assessment.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.