Passing: An Endless Journey through Reconstructed Spacetime with AI-Generated Sound
Akira Takahashi, Chihiro Nagashima, Zhi Zhong, Shusuke Takahashi, Yuki Mitsufuji
cs.SD · cs.AI
Quick verdict
Artwork paper: slit-scan-style spacetime resampling plus real-time V2A; compelling integrated system, but no evaluation of the installation or agency claims.
Summary
This artwork paper documents Passing, an interactive audiovisual installation exhibited at NEORT++ (Tokyo, April 2026), created by Sony researchers with visual artist Ryu Furusawa. A single continuous 4K 480-fps HDR recording of a monorail journey is treated as a spatiotemporal volume and resampled along nonlinear cross-sectional trajectories, then pre-rendered into 372 clips in four groups: A (forward), B (reverse plus mirroring), and C/D (distorted branches). A camera-based detector senses only viewer presence and probabilistically enables C/D transitions; viewers influence sound solely through the image. The selected stream conditions SpecMaskFoley, a real-time video-to-audio model on an RTX 4080; the authors omit its CLIP visual encoder, anchor semantics via CLAP conditioning on recorded monorail ambience, and fine-tune with about 35 minutes of custom audiovisual data, outputting 32-kHz stereo. The paper frames the model as a 'speculative listener' and claims creative agency is distributed among artist, AI, and audience. Main limitations: no evaluation of the final installation is reported - the FAD/DeSync comparison in Fig. 5 is inherited from the original SpecMaskFoley paper and does not transfer to this modified configuration; the distributed-agency claim is argued conceptually, with no visitor data; and no code, trajectory mathematics, or dataset is released, so reproducibility is low.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 0 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Treating a single continuous recording as a spatiotemporal volume and resampling it along nonlinear cross-sectional trajectories yields an endless, non-looping-feeling journey (Sec. 3.1, App. B.2-B.3).
- A modified SpecMaskFoley (CLIP encoder removed, CLAP-conditioned on monorail ambience) synthesizes synchronized 32-kHz stereo audio in real time on a single RTX 4080 for the reconfigured video stream (Sec. 4, App. A).
- Viewer presence influences sound only indirectly: presence state alters the visual sequence, and the AI generates sound for whatever it is shown, never operating sound as an instrument (Sec. 3.2, 5.1).
- Creative agency is distributed among artist (rules of reconstruction), AI (auditory interpretation), and audience (presence-mediated path selection), forming a three-way agency loop (Sec. 2.3, 6).
Strengths
End-to-end real-time system integration at exhibition scale
Sec. 4.3/App. A.1: continuous 32-kHz stereo generation on one RTX 4080, with display (4K/120fps) decoupled from model conditioning (precomputed 224x224/25fps) - a clean, documented solution to running a generative audio model inside a live exhibit.
Justified, documented architecture modification
App. A.1 gives a dual rationale for dropping CLIP: artistic (semantics anchored in the audio domain via CLAP conditioning on recorded monorail ambience) and technical (CLIP was a real-time inference bottleneck); Table 1 documents the configuration deltas (32 kHz, 128x800 Mel, 400 tokens, Vocos vs HiFi-GAN).
Concrete sequence and transition design
Sensible adaptation recipe for a tiny custom dataset
Honest conceptual framing and privacy hygiene
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This artwork/demo paper (cs.SD) documents *Passing*, an interactive audiovisual installation exhibited at NEORT++ (Tokyo, April 17-26, 2026), created with visual artist Ryu Furusawa. The system couples three components: (1) spacetime reconstruction of a single continuous 4K 480-fps HDR monorail recording into pre-rendered trajectory clips (Sec. 3.1, App. B); (2) binary camera-based viewer-presence detection that probabilistically enables branch transitions (Sec. 3.2-3.3, App. B.3); (3) real-time video-to-audio synthesis via a modified SpecMaskFoley (Sec. 4, App. A). The central claims are primarily conceptual: that the system sustains an endless journey from finite footage, that the AI functions as a 'speculative listener' for a physically impossible visual world, and that creative agency is distributed tripartitely among artist, AI, and audience (Sec. 6). This is not an algorithmic contribution paper; the technical content is an integration and reconfiguration of previously published components (SpecMaskGIT [21], SynchFormer [22], CLAP [23], SpecMaskFoley [2]).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 0 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Passing installation?
Passing is an interactive audiovisual artwork created by Sony researchers with visual artist Ryu Furusawa, exhibited at NEORT++ in Tokyo in April 2026. It takes a single continuous 4K high-speed recording made through a monorail window, treats the frames as a spatiotemporal volume, and generates an endless journey by slicing through that volume along nonlinear trajectories, while an AI model generates synchronized sound in real time.
How does the AI generate the sound in Passing?
It uses SpecMaskFoley, a real-time video-to-audio model running on an RTX 4080 GPU. The authors removed its CLIP visual encoder for speed, anchored semantics via CLAP conditioning on recorded monorail interior ambience, kept ControlNet-based temporal alignment, fine-tuned with about 35 minutes of custom recordings, and output 32-kHz stereo audio synchronized to the displayed video.
How do viewers interact with Passing?
A camera detects only whether a viewer is present in the viewing zone; no imagery is recorded or stored. Presence probabilistically enables transitions to distorted 'C/D' video branches; otherwise playback follows a default A/B loop. Viewers never control sound directly - their presence changes the video, and the AI generates sound for whatever visual flow it receives.
Was the Passing system evaluated?
No formal evaluation of the final installation is reported. The paper cites FAD and DeSync benchmarks from the original SpecMaskFoley paper on VGGSound, but the configuration used in Passing differs (no CLIP encoder, different tokenizer settings, extra fine-tuning data), and no listening tests, latency measurements, or visitor studies are presented.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-6 could not be analyzed directly: Fig. 5's benchmark positions and Fig. 6's transition graph are reconstructed from captions and prose only
- arXiv preprint (2609.27489, Sep 2026); peer-review status unknown
- Perceptual/aesthetic claims of a live installation cannot be fully assessed from text alone
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.