رانش زبانی در پس‌آموزش RLVR مدل‌های زبانی

This summary relies on the paper’s preprint (dated 2 October 2026); as of this summary’s compilation date (4 October 2026), the paper text makes no mention of peer review, so its independent peer-review status remains unconfirmed. The study defines "language drift" as novel, non-standard, or sometimes difficult-to-read linguistic behavior in language-model chains of thought. Such behavior can make reasoning harder for humans or other models to understand and monitor. The analysis is deliberately limited to text-based models that produce a reasoning trace and a final answer, while multimodal models are excluded.

The authors define language drift as KL divergence between a model’s output distribution and a reference human-language distribution, averaged over prompts from that distribution. The reference distribution cannot be calculated in practice and is treated as fixed but unknown. The theoretical analysis assumes finite vocabulary, prompt, reasoning-trace, and answer lengths, as well as full support for relevant distributions. In the paper’s framework, RLVR reward depends only on final-answer correctness and does not directly reward or penalize reasoning-trace content.

The first theoretical result says that, among RLVR checkpoints whose expected reward does not decrease, the maximum possible language drift is unbounded: reward can be maintained or increased while the reasoning-trace distribution moves arbitrarily far from the reference language. In contrast, supervised fine-tuning (SFT) has a finite upper bound under monotonically non-increasing cross-entropy relative to the training-data distribution. This is a difference in theoretical ceilings; it does not show that every RLVR run necessarily drifts more severely than SFT.

The third theorem states that any fixed cap on language drift limits the maximum achievable expected reward: for every drift radius, a larger radius permits greater reward. The argument relies on the optimal final-answer-reward policy concentrating probability on the most likely answer. The authors hypothesize that practically significant drift especially occurs when a task is novel to the base model and cannot be solved by sharpening existing behavior. This is experimentally examined rather than theoretically proven.

For testing, three small base models—Llama-3.2-1B, Gemma-3-1b-pt, and Qwen2.5-1.5B—were trained on GSM8K with SFT and RLVR; the RLVR implementation was DAPO, a GRPO variant. After hyperparameter searches, each model-method combination was repeated with five random seeds, for 30 runs total. Because the human-language distribution is unavailable, chain-of-thought legibility was estimated indirectly through three instruct models’ ability to infer a final answer from incomplete reasoning traces. Iso-performance checkpoints were also selected to control for accuracy differences.

The limitations are important: the experiments cover only 1-1.5B-parameter models and one dataset, GSM8K, and the observed drift does not reach the illegibility reported in some frontier-model examples. Llama and Gemma, for which sharpening existing behavior was insufficient, showed lower RLVR legibility and mutual legibility than SFT. Qwen, whose gains were largely explained by learning to stop after an answer, showed little relative drift. The authors provide no direct empirical test of the third theorem and describe legibility only as an approximate proxy, not a direct measure of drift.

Original arXiv paper