وقتی ترجمه‌ها باید به یک مسیرشناس برسند

A multilingual language model may be stronger in one language without transferring that capability to another. The issue is not only a lack of data; two sentences with the same meaning may follow different internal processing paths. arXiv:2610.01921 proposes a way to bring these paths closer together in mixture-of-experts, or MoE, models.

This article was prepared by Hossein Ramezani’s AI collaborator, an AI assistant based on the site’s recorded knowledge. Its primary source is the arXiv paper “Cross-Lingual Alignment for Decoder-Only Models Using MoE Routers.” The paper describes itself as published as an ICLR 2027 conference paper, but independent peer-review status has not been verified here. Its findings should therefore be read within the reported experimental scope, not as settled scientific certainty.

To understand the idea, the architectural difference comes first. In encoder-based models, it is relatively natural to construct one vector for each sentence and bring translations closer in representation space. Decoder-only large language models do not provide such a ready-made sentence representation. Differences in token counts, tokenization, and language structure also make token-by-token matching across translations difficult.

Common alternatives, such as using the final token’s hidden state or averaging hidden states, do not necessarily represent the meaning of an entire sentence. Averaging can weaken or cancel features encoded in different directions. As a result, directly applying an alignment loss to these representations in decoder-only models is not a simple or unambiguous task.

This research changes the target: instead of aligning hidden states directly, it aligns the outputs of MoE routers. In an MoE layer, a router determines which experts are activated for each token and with what weights. Router output can therefore be viewed as a distribution of how the model uses experts for that token.

The proposed method separately averages routing weights across all tokens in an English sentence and its target-language translation. It then minimizes KL divergence between those average distributions in selected middle layers. This auxiliary loss is added to the standard language-modeling loss, so the main next-token prediction objective is not removed.

A key design choice is not to select every layer. Using cross-lingual routing-divergence curves, the authors selected middle layers that already showed signs of expert sharing across languages. The rationale is that alignment may be useful in some layers and potentially harmful in others, especially where language-specific information is retained.

In the loss computation, gradients flow back only through the target-language sentence. The English sentence is used to construct a reference routing distribution and receives no direct update. To reduce cost, the authors also pack both sequences into one shared pass without padding and stop processing English tokens after the final alignment layer.

The experiments use a short stage of continual pretraining rather than full pretraining from scratch. Four open-source MoE models were studied: Qwen3-30B-A3B, GPT-OSS-20B, Granite-4.0-H-Tiny, and Marco-Nano. Target languages were Vietnamese, Sinhala, Hungarian, Telugu, Kannada, Thai, and Kyrgyz.

For each language, a parallel dataset of 200,000 samples was constructed, and each run covered one training epoch. Languages were limited to those in which the base models already had non-trivial proficiency. This work therefore does not claim to build ability from zero in low-resource languages. The parallel data was mainly sentence-aligned or aligned conversational material.

The first finding was that the auxiliary loss reduced routing divergence from English across all model-language combinations, while ordinary continual pretraining made only small changes to routing behavior. This alignment was measured on FLORES evaluation data, described as out of distribution relative to the training sets.

The second finding concerns the model internals. Router alignment does not automatically equal hidden-state alignment, yet the authors reported with the SoftCKA metric that hidden representations also became more similar to English. This observation was shown for Qwen3 and several languages, suggesting that the loss may affect more than router outputs alone.

The reported downstream result is cautious but positive. The proposed method outperformed the continual-pretraining baseline in 13 of 14 model-language pairs and tied in the remaining pair. The largest average gain was 2.1 points, while the mean improvement across all pairs was reported as 0.9 points.

These figures do not mean every task improved uniformly. The paper itself notes task-level noise, and its detailed tables show declines in some scores, including some translation or comprehension evaluations. The defensible claim is therefore not that this loss improves every multilingual output, but that a positive trend appeared in the average across this experimental evaluation suite.

The limited comparison with hidden-state alignment also matters. For Marco-Nano in Sinhala and Hungarian, the routing method outperformed a cosine-similarity method over mean-pooled hidden states. Yet that comparison covers only one model and two languages. The authors themselves present it as targeted support for the design choice, not as a comprehensive comparison of all alignment methods.

Another experiment updated only routers, changing fewer than 0.1% of parameters. It sometimes matched or exceeded the baseline, but its results varied substantially across settings. The paper’s cautious conclusion is that router-only tuning may be a parameter-efficient option, but it is not a reliable replacement for full-model training.

The central limitation is complete reliance on parallel data. Much pretraining, instruction tuning, and reinforcement-learning data has no parallel translation, while available parallel corpora are often sentence-level and limited in distribution. Even the baseline with 200,000 samples produced modest improvements. Parallel data alone is therefore not a complete training curriculum.

A second limitation concerns intervention timing. The experiments were run on already post-trained models, an environment more suitable for evaluation but sensitive and inefficient for continual pretraining. The authors expect the method could have a larger effect during pretraining, but that is a future-work hypothesis rather than a result established by these experiments.

There is also a conceptual risk: bringing language processing closer together may strengthen knowledge transfer, but indiscriminate alignment could weaken language- and culture-specific signals. The paper selects middle layers specifically to reduce this risk, yet it provides no direct experiment on preserving cultural or language-specific distinctions.

My proposed interpretation for teams training multilingual MoE models is this: do not treat the method as a ready-made capability. Evaluate the auxiliary loss against a clear accept/reject rule. For example, before running it, define a threshold in advance: keep the loss only if routing divergence across languages drops by a meaningful margin (comparable to what was observed in this research) while the average downstream metric stays at or above the baseline; otherwise drop it, or continue testing it only on a small data subset.

As a fully hypothetical example, a team developing an MoE model for Persian and English could set aside permitted translated pairs, measure router divergence across a few middle layers, and define a small test before scaling training: accept the change only if, on an internal evaluation set, Persian generation quality does not fall below a pre-set threshold (for instance, no drop in sample-based human judgment) while the routing-alignment metric improves. If either condition fails, the auxiliary loss should be rejected or retested with a smaller weight.

The paper’s main point is neither that routers fully represent sentence meaning nor that alignment is always desirable. Its narrower claim is that, across four MoE models, seven languages, and a short continual-pretraining stage, bringing expert-use distributions closer for translations was associated with greater internal alignment and average downstream gains. The practical value of the idea remains dependent on testing it with each team’s own data, languages, and training stage, under accept criteria defined in advance.