Disentangling Spatial Reasoning Failures with Composable Video Operators

This introduction is based on the preprint available as arXiv:2610.01999v1, posted on October 1, 2026; neither the paper nor this summary indicates peer review, and its independent peer-review status is therefore not confirmed. The paper asks why vision-language models fail on video spatial questions, since a benchmark’s final score alone cannot reveal whether the issue is scene perception or the computation and interpretation of spatial relations. Its scope includes distance, relative direction, route planning, camera-pose estimation, and measurement tasks in static and dynamic videos.

To separate causes, the authors compare video-only answering with answering augmented by structured spatial context. This context stores a coordinate contract, global scene dimensions, 3D object observations, and a timestamped camera trajectory in a shared schema. Two versions are used: predicted context from masks, tracks, depth geometry, and camera poses; and ground-truth context from source annotations. Their difference indicates a perception penalty, while remaining failures with valid ground-truth context are treated as a reasoning residual.

The controlled diagnosis reports four recurring error sources: inaccurate perception; missing required information in the context; selection of the wrong quantity; and errors in reference frames or spatial state. For example, a model may report center-to-center distance rather than the closest surface gap, or decide left versus right from scene axes rather than an observer’s viewpoint. Across eight subtasks, 52.5–99.1% of ground-truth-context failures matched one of the paper’s diagnostic rules; this consistency does not establish a unique cause for every error.

The proposed solution, CROSS, is a training-free library of typed geometric operators and composable spatial skills. Its operator families cover coordinate frames, geometry/shape, set and decision operations, and temporal/state operations. A surface-distance skill aggregates object geometry and measures the nearest boundary gap; a relative-direction skill builds an observer frame and projects the target into it; and a route skill updates position and heading after each step. A gate adds only valid derived outputs to the model context and otherwise falls back to video-only reasoning.

Diagnostic experiments used Qwen3-VL-32B-Thinking on REVSI-TINY with 823 questions and RESTI-TINY with 775 questions, using 32 and 30 frames respectively. With predicted context, CROSS raised the REVSI-TINY average from 50.0% to 60.4% and the RESTI-TINY average from 34.3% to 43.9%. On ReVSI with 64 frames, direct inference averaged 55.9%, predicted context 52.2%, and predicted context with CROSS 60.2%. In the code-writing interface, SpatialClaw on the full DSI-Bench rose from 62.8% to 66.3% while the atomic operators were kept frozen.

The limitations are consequential. Predicted context still depends on open-vocabulary object recall, monocular metric scale, and rotational drift; the heuristic gate cannot detect every incorrect output; and its room-scale assumptions may not transfer to outdoor scenes. The operator set does not cover acceleration or appearance order, and the indoor RESTI subset contained no acceleration questions. Deterministic operators also precompute part of an answer, so the results show reliable convention enforcement rather than necessarily an internally learned spatial algorithm. Finally, published-system scores are not matched comparisons because backbones and frame budgets differ, and the paper reports no significance tests.

Original arXiv paper