TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations
Ayush Jain, Sreeharsha Paruchuri, Ishita Gupta, Fan Zhang, Tanner Schmidt, Jakob Engel, Katerina Fragkiadaki, Adam W. Harley
cs.CV · cs.AI · cs.RO
Quick verdict
Claims first dense 3D tracker tracking all points over 1000+ frames via voxel de-duplication; solid engineering, but headline margin is over one weak baseline.
Summary
TrackEverything is a 3D point tracker that aims to track every visible scene point across long videos, not just sparse queries or first-frame points. The method processes videos in 16-frame sliding windows, unprojects features into a world-coordinate 3D feature cloud, then refines each point's window-end position with a static/dynamic classification before decoding dense trajectories only for points classified as dynamic. A voxelization-based merge at window boundaries fuses co-located tracks, so representation size scales with unique scene geometry rather than video length, and a 3D variant of WAFT replaces memory-heavy 4D correlation volumes with feature sampling. On TAPVid-3D the system reports over 20% higher APD-P than the open-source all-frame dense baseline VDPM on 48-frame clips, roughly matches sparse trackers (TAPIP-3D, SpatialTracker-v2) and first-frame dense DeltaV2, and is the only tested method tracking all points across 1000+ frame videos within 40 GB of GPU memory. Ablations attribute gains to iterative refinement and WAFT; with sensor geometry accuracy rises sharply (71.7 APD-P on PStudio). Main limitations: performance is coupled to external geometry quality, merging is irreversible via mean-pooling, D4RT still leads on two of three 48-frame datasets, and code and weights are not yet released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- TrackEverything is the first 3D point tracker that tracks all visible points across videos exceeding 1000 frames within 40 GB of GPU memory.
- Voxelization-based de-duplication at sliding-window boundaries makes representation complexity scale with unique scene geometry rather than video duration.
- On TAPVid-3D 48-frame clips it outperforms all open-source all-frame dense 3D trackers by more than 20% APD, driven mainly by iterative refinement and 3D WAFT.
- The static/dynamic decomposition preserves accuracy (31.2 APD-P unchanged) while cutting latency 4.8x and memory 2.5x.
- With sensor-provided geometry it surpasses D4RT on PStudio (71.7 vs 49.6) and ADT (46.5 vs 40.8) despite roughly 20x fewer parameters.
Strengths
Genuine capability milestone with measured efficiency
First all-frame dense 3D tracker at 1000+ frames: 9.1 GB and 0.05 s/frame at default settings (Table 2b), peak memory under 30 GB at 900 frames (Fig. 3c), and 23.1 FPS tracker-only with early merge (Table 4) — while every dense baseline OOMs by roughly 96-200 frames.
Attributable design, not just scale
61M parameters vs D4RT's 1B+; Table 2b-c isolate each component: static/dynamic routing saves 4.8x latency and 2.5x memory at identical APD-P (31.2), iterative refinement is worth +4.8 and 3D WAFT +1.9 APD-P; Table 2d shows voxelization is necessary (OOM without it).
Honest failure-mode analysis
Community-service dataset repairs
Careful baseline re-evaluation
Weaknesses
Headline margin is over a weak single baseline
Strong dependence on external geometry quality
No uncertainty quantification
Training/eval overlap in the scaling study
Dense evaluation is synthetic-only
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
TrackEverything (arXiv 2609.30222, CMU + Meta) is a method paper in 3D point tracking. Its central claim: by representing video as a persistent, de-duplicated 3D scene-track cloud in world coordinates, dense all-point tracking can scale with unique scene geometry instead of video duration — enabling, per the authors, the first tracker to follow all visible points across 1000+ frame videos within 40 GB of GPU memory (Sections 1, 4.2). Three supporting claims: (1) voxelization-based merging at sliding-window boundaries bounds representation growth; (2) an 'endpoints before trajectories' decomposition concentrates dense decoding on dynamic points; (3) 3D WAFT replaces 4D correlation volumes with feature sampling (Sections 2-3). The abstract's competitive framing — more than 20% APD over open-source all-frame dense trackers, parity with sparse trackers on long sequences — is the headline a reader should scrutinize.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is TrackEverything?
A 3D point-tracking model from CMU and Meta (arXiv 2609.30222) that tracks every visible point of a scene across long videos, rather than a sparse set of user-selected points or only points visible in the first frame. It represents video as a de-duplicated 3D point cloud in world coordinates and reports the first all-point dense 3D tracking over videos longer than 1000 frames within 40 GB of GPU memory.
How does TrackEverything handle 1000+ frame videos without running out of memory?
It processes video in 16-frame sliding windows and, at each window boundary, merges tracks that land in the same 3D voxel via mean-pooling, so the representation grows with unique scene geometry instead of frame count. Static points skip trajectory decoding, and a 3D WAFT feature-sampling scheme replaces memory-heavy 4D correlation volumes. Measured peak memory stays under 30 GB at 900 frames on PointOdyssey.
Is the TrackEverything code available?
At the time of this review, no code or model weights had been released. The paper points to a project page (trackeverything.github.io) with visualizations and promises to release the codebase together with corrections to the PointOdyssey dataset, including fixes for off-by-one camera extrinsics affecting 16 sequences.
How does TrackEverything compare to D4RT?
On 48-frame TAPVid-3D clips, the closed-source D4RT reports higher APD-P on DriveTrack and Panoptic Studio but lower on ADT (average 43.8 vs 34.7 for TrackEverything). The authors note D4RT has roughly 20 times more parameters, uses private training data, has no public code, and cannot run beyond 48 frames, whereas TrackEverything scales to 1000+ frames and surpasses D4RT's reported numbers when given sensor geometry.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Text-only extraction: figures (Fig. 1-10) and qualitative visualizations could not be inspected; performance-curve claims in Fig. 3 are taken from captions and surrounding text.
- Several cited baselines (D4RT, VDPM, Any4D, DeltaV2) are 2026 venue papers whose reported numbers could not be independently verified in this review.
- Project-website visualizations are referenced as evidence for real-world dense tracking quality but were not accessible during review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.