LiteReality-Agent: Agentic Reconstruction of Interactive 3D Indoor Scenes

LiteReality-Agent is an agentic system that turns RGB-D room scans, including colour, depth, and camera parameters, into interactive and simulation-ready 3D scenes. Its central idea is to formulate reconstruction as coding: an agent reads and edits a Python file, Room.py, and executing that file builds the scene. The file is therefore the agent’s persistent, reproducible, and editable workspace. The source is an arXiv preprint dated 1 October 2026, and its independent peer-review status has not been confirmed.

The system has two stages. During “scene initialisation,” the room shell and detected object boxes are obtained from scan data. A layout agent checks and repairs placements outside the room, wall penetrations, missing supports, and incompatible overlaps through bounded translations or resizing. Structured or articulated objects are then created through procedural generation, whereas visually complex and mostly static objects are reconstructed with a 3D generation model. The result is an initial scene in Room.py.

During “agentic authoring,” the agent runs an observe–edit–verify loop. Its tools select informative views, compare renders with calibrated photographs, perform metric measurement and wall-image stitching, and retrieve PBR textures. The workflow successively refines wall materials, fixtures and lighting, small objects, and finally the geometry, appearance, and articulation of procedural assets. Every change is saved in code and rebuilt, so the comparison reflects the current edit.

Quality control is not only image similarity. Sessions have turn and tool-call limits, and exploration and self-checking are disabled during the final 10% of tool calls so edits can be completed. The system then separately assesses mesh collisions and object support. Bounding-box overlap between a chair and table is not necessarily an error, whereas intersecting mesh surfaces are treated as one. Unanchored objects receive bounded corrections and are rechecked. The scene is exported to MuJoCo with mass, inertia, collision geometry, and joints; some physical properties are authored or estimated.

Evaluation covers 10 indoor scenes and 620 recorded views. Compared with LiteReality and the general-purpose GPT-6-in-Codex and Fable 5.1-in-Claude-Code agents, LiteReality-Agent reports the best mean visual metrics: 0.6105 SSIM, 14.129 dB PSNR, 0.2033 RMSE, and 0.5392 LPIPS. Its depth mean absolute error was 10.50 cm over 611 frames. All 86 identified furniture objects had near contact with support, and no surface intersection between furniture pairs was detected.

The limitations are substantial. Physical properties are largely estimated from language-model inference, object categories, material priors, or geometry, without reference measurements; consequently, MuJoCo output and robot-interaction demonstrations do not establish reproduction of real-world dynamics. Size or pose errors in RoomPlan detections may also propagate into reconstruction, while collision, support, and image-similarity checks do not guarantee true layout accuracy. The perceptual evaluation also has a small sample, uneven scene coverage, and dependent ratings, and therefore does not establish statistical significance or physical validity.

Original arXiv paper