When the Same Reward Teaches Different Visual Skills

This summary draws on the arXiv preprint “Same Reward, Different Skills: When Multimodal RL Learns to Look” (posted October 2026). The paper does not report peer review or acceptance at a journal or conference; the absence of such a statement does not by itself indicate the work is fraudulent or invalid, but its independent peer-review status remains unconfirmed. Its central question is whether reinforcement learning with verifiable rewards genuinely strengthens visual skills in vision-language models, or mainly makes nonvisual answer-production strategies easier to elicit.

On Geometry3K, 100 GRPO steps for Qwen2.5-VL-3B-Instruct with real images improved accuracy by 0.238 over the base model. Yet training without images and training with gray canvases, each then tested with real images, improved accuracy by 0.137 and 0.125 respectively — 57% and 52% of the real-image gain. For the 7B model, gray-canvas training recovered 0.78 of the real-image gain. Removing images at test time, however, eliminated nearly all of these improvements.

The authors distinguish reading a value from a marked location, locating a named target, and “discovery” of a target defined only by a relation such as the lowest point. Under longer 400-step training with real images and a trainable vision encoder, benchmark accuracy rose from 0.175 to 0.436 and 0.453 across two runs. But grounding pair accuracy, after a small early rise, fell by 0.070 and 0.052 from step 100 to step 400, ending below the base model’s level — a pattern the authors call "grounding corrosion," even as benchmark gains persisted.

To test their proposed design rule, the researchers built counterfactual coordinate scenes: the question stays fixed across two images, but changing one visual fact changes the answer. The target is never named; the model must first identify, for example, the point with the smallest y-coordinate before reading the requested coordinate. In a base-model audit, 92.9% of this corpus’s rows both benefited positively from the image and fell in a learnable range (neither trivially easy nor unreachable). The 7B model was trained with standard GRPO, five responses per prompt, equal correctness-and-format rewards, and frozen vision encoder and projector.

On a separate confirmatory set of 20-point scenes, denser than any training scene, discovery accuracy rose from 0.425 to 0.740 after 30 steps and to 0.875 after 100. Replacing test images with gray canvases reduced discovery and identification-probe accuracy to exactly zero. A control trained on gray canvases and then evaluated with real images showed almost no gain at matched step 30; the gain attributable to training images was 0.360, with a 95% interval of [0.275, 0.445]. The learned skill also transferred to an independently built grounding task, raising its pair accuracy from 0.768 to 0.948.

Within the scope of these experiments, the paper’s conclusion is a design rule: a task should make visual evidence necessary for a correct answer while still providing learnable correctness feedback. Adding a caption that made the training answer extractable from text, with the same images, reward, and budget, cut the discovery gain by nearly two thirds. An important limitation is that the central evidence rests on programmatically generated coordinate scenes, mainly two Qwen2.5-VL scales, and a limited number of runs; generalization to natural images and practical applications has not been tested.

Original arXiv paper