مسیرگذاری UTF-8 و UTF-16 برای کاهش شکاف توکن میان خط‌ها

This summary relies on preprint arXiv:2610.01984v1, posted on October 1, 2026, which is after the current review date of this assessment (October 4, 2026). The paper is framed as a NeurIPS 2026 submission, but no evidence of completed independent peer review was found in the available material; its review status should therefore be described as unconfirmed, neither necessarily peer-reviewed nor fabricated. The paper studies token-budget disparity in UTF-8 byte-level BPE tokenizers: equivalent text in different scripts can consume very different token counts, affecting usable context, prompt-processing time, and cost under a fixed context window.

The method rests on the pre-merge ‘encoding floor,’ not a guarantee of final token counts. In standard BBPE, ASCII has a one-symbol floor; many Cyrillic, Arabic, and Hebrew characters have a floor of two; and CJK, Devanagari, and Thai characters have a floor of three. Universal Byte-Level Encoding, or UBE, routes each Unicode scalar value solely by its UTF-8 byte length: 1- or 2-byte characters stay on the UTF-8 path, while 3- or 4-byte characters are routed through UTF-16LE.

This lowers the floor for 3-byte Basic Multilingual Plane characters from 3 to 2, while ASCII remains at 1; 4-byte non-BMP characters still have a floor of 4. UBE uses two disjoint 256-symbol alphabets, one for UTF-8 and one in a Unicode Private Use Area block for UTF-16 bytes. BPE merge rules and selection, the absence of any language or script identification, and the Transformer architecture remain unchanged. The cost is 256 fewer merge slots than BBPE, roughly 0.8% overhead at a 32K vocabulary.

On FLORES-200, comprising 1,012 aligned sentences across 204 language-script configurations, UBE reduced the Gini coefficient of English-normalized token counts in every core 16K, 32K, and 64K comparison; relative reductions across the nine core comparisons ranged from 6% to 14%. In the main 101-language, 32K-vocabulary setting, Gini fell from 0.2183 to 0.1979, a 9.4% reduction, and UBE produced fewer tokens for 189 of 204 languages. Santali dropped 34.7% and Central Atlas Tamazight 31.9% in token count. At a 256K vocabulary, the Gini reduction narrowed to about 1.9%.

In additional robustness checks under the same 101-language/32K setting, Gini changed by about -11.1%, -8.5%, and +0.1% for UDHR, SIB-200, and MGSM, respectively. Because of public-text overlap, the authors treat UDHR as a stress test rather than a fully held-out corpus. In 182M-parameter, 32K-vocabulary language models, quality results were small and mixed: across three independent random seeds, held-out BPB Gini was effectively tied, at 0.1819 plus or minus 0.0052 for BBPE and 0.1822 plus or minus 0.0074 for UBE. The paper explicitly does not claim a systematic model-level advantage for UBE on this metric.

UBE targets only this substrate-level source of disparity; it does not resolve data balance, vocabulary-size allocation, pre-tokenizer boundary choices, or merge-rule design. Under the English-only/16K training setting, its extra 256 symbols meant UBE produced fewer tokens in only 37 of 204 languages, and small regressions appeared in some high-coverage languages, including Georgian and Amharic. Non-BMP characters and many emoji receive no encoding-floor reduction. The languages with the largest intrinsic token savings were absent from the downstream language-model training corpus, and UBE is not a drop-in replacement for existing pretrained checkpoints.

Original arXiv paper