MOLE: Mixture of Latent Experts for Complementary Visual Reasoning

This overview is based on preprint arXiv:2610.01917v1, released on October 1, 2026. The paper does not report peer review or acceptance at a journal or conference, so the independent-review status of this preprint is unconfirmed and its findings should be read accordingly. It argues that latent visual reasoning methods often let multiple latent tokens access the same visual evidence through shared value projections. Adding tokens may therefore create redundant representations rather than new reasoning capacity.

The proposed MOLE, or Mixture of Latent Experts, sparsely routes visual tokens to subsets of latent visual experts at every decoder layer. Each image token selects its top-K experts according to router scores, while other connections are masked. Router scores are also used as an additive attention prior. Consequently, experts need not access identical sets of visual evidence.

MOLE also uses an Expert-Specific Value Adapter, or ESVA. It normalizes the shared value-projection output, transforms it through an expert-specific two-layer MLP, and combines it with a scaled residual path. The default configuration has four visual experts, four summary experts, Top-K routing of 2, and an ESVA hidden dimension of 32. Visual experts do not attend to one another during evidence extraction; summary experts aggregate their representations, and answer tokens access both groups.

Training has two stages to prevent the model from bypassing the latent pathway. In Stage 1, prompt and answer tokens cannot directly attend to image tokens, so visual information must pass through the latent route. In Stage 2, direct access is restored. The objective combines answer-prediction loss with MoE losses for load balancing and router stability, plus a diversity loss for ESVA transformations. The main experiment uses Qwen2.5-VL-7B, LoRA, and a balanced 63K-example subset from CoVT-Dataset, while freezing the vision encoder and projector.

As reported by the authors, across five benchmarks and six metrics—CVBench-2D/3D, BLINK, RWQA, V*, and POPE F1—MOLE achieves an average of 78.6: 78.2, 87.8, 63.0, 71.5, 80.6, and 90.2. Per the text, this average is 4.9 points above data-matched SFT and 3.6 above the strongest compared latent baseline, LIVR, at a latent budget of 8. The best compared 16-token configuration is reported as 3.1 points lower. Masking the latent pathway while preserving direct visual access reduced the average by 9.2 points.

An important limitation is that the evidence is mainly limited to these five benchmarks, the 63K-example training set, and the Qwen2.5-VL and Qwen3-VL model families; generalization to other models, data, or applications is not established. GPT-4o and Claude-4-Sonnet results come from external reports using different evaluation configurations and are included only for reference. MOLE also retains inference overhead: on RWQA, the paper reports about 23% higher time, 0.8% higher FLOPs, and 1.6% higher peak memory than the backbone. Because these findings come from an unreviewed preprint, they should be interpreted cautiously pending independent confirmation.

Original arXiv paper