VISTA: Lossless Visual Memory for Multimodal Agents

This overview is based on the preprint (arXiv) version of paper 2610.02200 (October 2026); its independent peer-review status is not established and is not asserted here. The paper introduces VISTA as a “visual harness” for multimodal models, intended to support long-horizon reasoning in interactive environments. Its central idea is that a model should not rely only on compressed image representations retained in the conversation context: it should be able to revisit prior visual evidence whenever needed and reorganize its visual input around the current problem.

VISTA has three components: visual observations of the environment, lossless visual memory, and visual inspection tools. Every frame returned by the environment, including intermediate animation frames, is stored at original resolution and indexed by turn and frame number. The model can inspect several frames from earlier times together or enlarge a selected rectangular region; a pixel-reading tool also returns RGB values for a selected region. These operations do not alter the environment; only a game-action call advances its state.

At each turn, the agent sees the current state and available actions, forms hypotheses and plans in natural language, and inspects visual memory when needed. It also maintains two notes: GUIDE.md for rules and reusable knowledge across levels, and WORKING.md for temporary state and plans in the current level. As the context limit approaches, the agent writes a handoff summary and continues in a fresh context; visual memory, action history, and notes remain available. The harness itself is not a trained model: it mechanically executes tool calls and returns their outputs.

As reported by the authors, on the 25 public ARC-AGI-3 games, VISTA with Claude Opus 5.0 at extra-high reasoning effort achieved an RHAE of 100.00 and completed every game. Its 7,302 total actions were 57.4% fewer than the 17,135-action first-time-human reference. The GPT-5.6 Sol version, at max reasoning effort, also completed all games and reached 99.00 RHAE with 9,126 actions. RHAE measures completion and action efficiency relative to a human reference, assigning zero credit to unfinished levels.

Ablations with GPT-5.6 Sol indicate that replacing text grids with images increased the official baseline score from 13.33 to 47.32; continuous conversation and notes raised it to 70.05. Adding lossless memory and inspection brought RHAE to 94.10, and exact pixel readout raised it to 99.00. In a direct comparison, image observations reached 99.00 RHAE with 9,126 actions, versus 96.75 RHAE with 10,614 actions for text grids; reported average token use per game was 30.7 million versus 71.9 million, respectively.

The paper also evaluates VISTA, with minimal adaptation, on GameWorld, AI GameStore, and 39 BabyVision visual-tracking questions, outperforming minimal-harness baselines with the same model on all three, but these results do not guarantee performance in other environments. A main limitation is dependence on the underlying model’s capabilities. The authors also cannot rule out that public games appeared in training data for models released after the benchmarks were published. Evaluation on novel games, including ARC-AGI-3’s private set, would be a stronger generalization test. Token costs also rise with larger contexts or higher-resolution images, and the paper itself does not report evaluation on real physical tasks.

Original arXiv paper