How 13 LLM Tokenizers Split Text

Published on November 19, 2026

How 13 LLM Tokenizers Split Text

A tokenizer breaks text into units called tokens and represents each one with an integer ID that a large language model (LLM) can process. The two OpenAI encodings I tested split 1234567 as 123|456|7, where each bar marks a token boundary. Command A+ also uses three tokens, grouping the digits as 1|234|567. Gemma 4 and Qwen3.8 use seven, one per digit.
I compared 13 publicly available tokenizers on 269 short inputs and 7,850 documents. The different digit groupings arise because a tokenizer can impose boundaries before it applies its vocabulary. Within those boundaries, the available tokens and the algorithm determine the final split. I also tested whether decoding returned the original text, including characters that a tokenizer normalizes or uses as internal markers.
  • In the English and code samples, seven tokenizers almost always kept an ASCII symbol and the following newline together. Five always separated them. PLaMo 2 kept 31% of these pairs together.
  • Across the vocabularies, the share of entries classified as kana (hiragana or katakana, alone or mixed with kanji) ranged from about 0.2% to 40% of ordinary tokens. Tokenizers with a larger share generally kept longer stretches of Japanese text together in the sample.
  • Six vocabularies had entries named for images, audio, or video. Four of the model configurations accepted images. With their default settings, one 1920 × 1080 image occupied between 266 and 3,343 token positions.
  • GLM-5.1 contained 99% of the distinct byte sequences in OpenAI's ordinary cl100k_base vocabulary, and Qwen3.8 contained 97%. Even with that overlap, agreement on token boundaries depended on the language.

1. Tokenizers and method

I used fixed revisions of all 13 tokenizers, including the OpenAI encodings o200k_harmony and cl100k_base (o200k and cl100k in the tables). Four focus on Japanese (LLM-jp-4, PLaMo 2, PLaMo 3, and Sarashina2.2). I use “PLaMo” for results that apply to both versions.
I examined these revisions on November 17, 2026. For Grok 2, I loaded the published vocabulary into tiktoken and used SGLang's regular expression to divide the input before applying the vocabulary.
Table S1. Pinned sources for the 13 tokenizers. The OpenAI rows identify the tiktoken encoding. The other rows link to the model repository. Commit IDs are abbreviated to 12 characters.
TokenizerSourceCommit
o200kopenai/tiktoken (o200k_harmony)4e71bbe0c078
cl100kopenai/tiktoken (cl100k_base)4e71bbe0c078
Gemma 4google/gemma-4-E4B-it4ecc2ba63bca
Qwen3.8Qwen/Qwen3.8-27B1d4bf0f2ff60
Command A+CohereLabs/command-a-plus-05-2026-bf165fb6fde5fd12
DeepSeek V4deepseek-ai/DeepSeek-V4-Prob5968e9190ef
Mistral Small 4mistralai/Mistral-Small-4-119B-2603a11f36bebf70
GLM-5.1zai-org/GLM-5.126e1bd6e011f
Grok 2xai-org/grok-2daf4395a80ad
LLM-jp-4llm-jp/llm-jp-4-8b-instruct098f2b2cf330
PLaMo 3pfnet/plamo-3-610m-fin-instructa000b59209f1
PLaMo 2pfnet/plamo-2-1b92c75fd6eea9
Sarashina2.2sbintuitions/sarashina2.2-3b-instruct-v0.14f3626fb1b64
Grok 2 splits are Powered by xAI under the xAI Community License. All rights reserved. Copyright © xAI 2025. PLaMo 3 splits are Built with PLaMo under the PLaMo Community License.
A tokenizer assigns an ID to each entry in its vocabulary. I call entries marked as special by the implementation “special tokens” and the rest “ordinary tokens.” Nine of the tokenizers use byte pair encoding (BPE), which repeatedly merges adjacent units according to learned rules. Eight of those nine first use regular expressions to divide the input into pieces, a step called pre-tokenization. BPE can merge only within a piece. The four Japanese-focused tokenizers use the Unigram algorithm, which selects the segmentation with the highest sum of token scores[1]. My introduction to BPE traces a set of merges step by step.
The eight byte-based tokenizers start from UTF-8 bytes, so a token can end partway through a character. Gemma 4 and the four Unigram tokenizers start from characters and fall back to individual UTF-8 bytes for a character that their vocabulary cannot represent. Spaces can also be part of a token. With ␣ for a space, most tokenizers have forms such as ␣the. PLaMo also has ␣of␣the, with an internal space, and a comma followed by a space (,␣).
Table 1. Vocabulary and segmentation settings. Vocabulary size counts registered IDs, including added and special tokens. The digit column describes standalone ASCII digits. NFC is Unicode Normalization Form C.
TokenizerVocabulary sizeAlgorithmBase unitUsual space positionStandalone ASCII digitsUnicode normalization
o200k201,088BPEBytesLeadingUp to 3, from the leftNone
cl100k100,261BPEBytesLeadingUp to 3, from the leftNone
Gemma 4262,144BPECharactersLeadingSingle digitsNone
Qwen3.8248,077BPEBytesLeadingSingle digitsNFC
Command A+255,032BPEBytesLeadingUp to 3, from the rightNone
DeepSeek V4129,280BPEBytesLeadingUp to 3, from the leftNone
Mistral Small 4131,072BPEBytesLeadingSingle digitsNone
GLM-5.1154,856BPEBytesLeadingUp to 3 from the left, then smaller tokens as neededNone
Grok 2131,072BPEBytesLeadingSingle digitsNone
LLM-jp-4196,608UnigramCharactersLeading, with an extra initial space markerSingle digitsNone
PLaMo 3107,520UnigramCharactersLeading, internal, or trailingSingle digitsNone
PLaMo 2100,000UnigramCharactersLeading, internal, or trailingSingle digitsNone
Sarashina2.2102,400UnigramCharactersLeadingSingle digitsNone
I used the short inputs to trace individual splits. For longer texts and code samples, I measured token counts and boundary positions by running all 13 tokenizers on the same 7,850 documents (7.5 MB in UTF-8). The Japanese and English portions each contain 200 texts and 50 code samples, all written for this study. Of the 7,350 documents in other languages and scripts, 6,600 are web excerpts. I measured compression efficiency by dividing the total UTF-8 bytes in each group by its total token count. For the same documents, a higher value means fewer tokens. The measurement data (JSON) include the input groups, counts, and tokenizer revisions.
Each of the 149 language and script categories contributed 50 documents. The 6,600 web excerpts from FineWeb2 cover 132 of those categories (36 in T4 and 96 in T5). The other 17 contain texts written for the study. They cover Japanese and English, the three T2 categories (Simplified Chinese, Traditional Chinese, and Korean), and 12 T3 categories including German and Arabic.
News, social media, code, and academic writing each contribute another 50 samples in Japanese and 50 in English. These 400 samples bring the total written for the study to 1,250.
For each ordered pair of character types, I measured the fraction of adjacent pairs with a token boundary between the two characters. To compare two tokenizers, I counted every internal byte position where either one split, even positions inside a character. Agreement is the fraction of those positions where both split, with counts pooled across documents. Both measurements require the token bytes to line up with the original text. After I removed LLM-jp-4's extra leading space, all 7,850 documents aligned for every tokenizer.
I ran the text measurements with Python 3.14, Transformers 4.57.6, Tokenizers 0.22.2, tiktoken 0.14.0, and SentencePiece 0.2.2. I generated the random strings with Python's random.Random(20261117). The measurement data also give the SHA-256 checksum for the ordered document IDs and texts, vocabulary statistics, and the settings and token counts for the image, audio, and video tests.
In the examples, ␣ marks an ASCII space and ⏎ a line feed (LF). Angle brackets such as <e3> enclose the hexadecimal bytes of a token that cannot be decoded as UTF-8 on its own. I encoded plain text without a chat template or automatic start or end tokens. LLM-jp-4 prepends a space marker of its own. It contributes an extra token when it stays separate from the first character.

2. Digits

The two OpenAI encodings, DeepSeek V4, and GLM-5.1 all use \p{N}{1,3} to group digits from the left before BPE. For 1234567, that gives three pieces (123, 456, and 7). The OpenAI and DeepSeek vocabularies have a token for each piece. GLM-5.1 lacks 456, so its final split is 123|45|6|7.
Table 2. ASCII digit splits and vocabulary coverage. A bar marks a token boundary and ␣ marks a space. Digits per token is the total number of digits in 300 random 64-digit strings divided by their total token count.
Tokenizer12345673.141592026-11-17Two- and three-digit ASCII entriesDigits per token (64-digit inputs)
o200k123|456|73|.|141|59202|6|-|11|-|171,1002.91
cl100k123|456|73|.|141|59202|6|-|11|-|171,1002.91
Gemma 41|2|3|4|5|6|73|.|1|4|1|5|92|0|2|6|-|1|1|-|1|701.00
Qwen3.81|2|3|4|5|6|73|.|1|4|1|5|92|0|2|6|-|1|1|-|1|701.00
Command A+1|234|5673|.|14|1592|026|-|11|-|171,1002.91
DeepSeek V4123|456|73|.|141|59202|6|-|11|-|171,1002.91
Mistral Small 41|2|3|4|5|6|73|.|1|4|1|5|92|0|2|6|-|1|1|-|1|701.00
GLM-5.1123|45|6|73|.|141|59202|6|-|11|-|172771.63
Grok 21|2|3|4|5|6|73|.|1|4|1|5|92|0|2|6|-|1|1|-|1|701.00
LLM-jp-4␣|1|2|3|4|5|6|7␣|3|.|1|4|1|5|9␣|2|0|2|6|-|1|1|-|1|700.98
PLaMo 31|2|3|4|5|6|73|.1|4|1|5|92|0|2|6|-|1|1|-|1|701.00
PLaMo 21|2|3|4|5|6|73|.1|4|1|5|92|0|2|6|-|1|1|-|1|701.00
Sarashina2.21|2|3|4|5|6|73|.|1|4|1|5|92|0|2|6|-|1|1|-|1|701.00
Command A+ first applies \d{1,3}(?=(?:\d{3})*\b), which groups digits in threes from the right at a word boundary. The result, 1|234|567, lines up with the commas in 1,234,567. The same rule gives 14|159 after the decimal point in 3.14159 and 2|026 at the start of the date 2026-11-17. Before a letter or underscore, the right-grouping rule no longer matches. A second rule then groups from the left, giving 123|456|7|kg for 1234567kg.
Qwen3.8, Mistral Small 4, and Grok 2 use \p{N} to isolate each digit before BPE. Gemma 4 and the four Japanese-focused tokenizers reach the same result on a standalone run of ASCII digits because their vocabularies have no entries made up of two or more ASCII digits. PLaMo can still combine one digit with a neighboring character. For instance, it splits 3.5kg as 3|.5|kg.
The two OpenAI encodings, Command A+, and DeepSeek V4 have entries for all 100 two-digit strings and all 1,000 three-digit strings, including leading zeros. Each of the 300 random 64-digit inputs therefore took 22 tokens, or 2.91 digits per token. GLM-5.1 averaged 1.63 because its vocabulary covers just 177 of the three-digit strings. It also lists longer digit strings that its own rules prevent it from producing.
GLM-5.1 has 163 entries consisting of four or more ASCII digits with an optional leading space. Of these, 39 have the space. It encodes 1985 as 198|5 and ␣2002 as ␣|200|2. Another 91 entries, including ␣41, combine a space with one to three digits. Its pre-tokenizer separates the space from the digits in these cases too. The ID ordering offers a clue to where these entries came from.
The same number can split differently when its written form changes. In the full-width string 1234567, Command A+ and Grok 2 split each digit from 3 onward into byte fragments. The other eleven kept each character intact, with o200k and DeepSeek V4 also joining 12. Writing 123,456,789 in kanji (一億二千三百四十五万六千七百八十九) took 11 tokens with DeepSeek V4 and 27 with cl100k, the extremes in this comparison.

3. Whitespace and newlines

A line break can share a token with the punctuation before it. In end.⏎⏎Next, for instance, cl100k makes the period and both line feeds one token, .⏎⏎. Qwen3.8 keeps the two line feeds together but puts the period in a separate token. In both regular expressions, [\r\n]* at the end of the punctuation pattern lets newlines stay in the same piece. The entry .⏎⏎ is available to cl100k and missing from Qwen3.8's vocabulary.
Table 3. Token boundaries around punctuation and line feeds. A bar marks a boundary, ␣ a space, and ⏎ a line feed. The first input contains a Japanese sentence ending (です。), two line feeds, and the character 次 (“next”).
Tokenizerです。⏎⏎次end.⏎⏎Nextf(x)⏎y
o200kです|。⏎⏎|次end|.⏎⏎|Nextf|(x|)⏎|y
cl100kです|。⏎⏎|次end|.⏎⏎|Nextf|(x|)⏎|y
Gemma 4です|。|⏎⏎|次end|.|⏎⏎|Nextf|(|x|)|⏎|y
Qwen3.8です|。|⏎⏎|次end|.|⏎⏎|Nextf|(x|)|⏎|y
Command A+です|。⏎⏎|次end|.⏎⏎|Nextf|(x|)⏎|y
DeepSeek V4です|。⏎⏎|次end|.⏎⏎|Nextf|(x|)⏎|y
Mistral Small 4です|。⏎⏎|次end|.⏎⏎|Nextf|(x|)⏎|y
GLM-5.1です|。⏎⏎|次end|.⏎⏎|Nextf|(x|)⏎|y
Grok 2です|。⏎⏎|次end|.⏎⏎|Nextf|(x|)⏎|y
LLM-jp-4␣|です|。|⏎|⏎|次␣end|.|⏎|⏎|Next␣f|(|x|)|⏎|y
PLaMo 3です。|⏎⏎|次end|.|⏎⏎|Nextf|(|x|)|⏎|y
PLaMo 2です。⏎|⏎|次end|.⏎⏎|Nextf|(|x|)|⏎|y
Sarashina2.2です|。|⏎|⏎|次end|.|⏎|⏎|Nextf|(|x|)|⏎|y
The 200 English texts and 100 code samples contained 2,238 places where an ASCII symbol was immediately followed by a newline. Six tokenizers kept every one of those pairs together (cl100k, Command A+, DeepSeek V4, Mistral Small 4, GLM-5.1, and Grok 2). A seventh, o200k, kept all but one together. A program that detects line breaks or enforces an output grammar by token ID has to recognize tokens such as )⏎ as well as tokens consisting entirely of whitespace.
Qwen3.8, Gemma 4, LLM-jp-4, PLaMo 3, and Sarashina2.2 have no ordinary token that combines an LF with a non-whitespace character. Among ordinary entries that decode as UTF-8 on their own, o200k has 2,034 distinct strings combining an LF with non-whitespace, and Command A+ has 2,280. PLaMo 2 has 39, including 。⏎ with a Japanese full stop. It kept the symbol and newline together at 31% of those 2,238 positions.
Table 4. Whitespace in distinct ordinary token strings that decode as UTF-8 on their own. Run lengths count characters in tokens made entirely of spaces, line feeds (LFs), or tabs. The last two columns count strings containing at least one LF.
TokenizerWhitespace-only entriesLongest space runLongest LF runLongest tab runEntries with LFEntries with LF and non-whitespace
o200k43412816202,2252,034
cl100k44112832202,1781,977
Gemma 4100313131310
Qwen3.844012832202000
Command A+50751264322,5442,280
DeepSeek V41571281611849799
Mistral Small 4129755161,0691,064
GLM-5.145451232202,1781,977
Grok 237525632171,8111,637
LLM-jp-468161110
PLaMo 339324860
PLaMo 24216246139
Sarashina2.22681110
Even among tokens made up entirely of whitespace, the length of a run matters. LLM-jp-4 and Sarashina2.2 need one token per LF, while PLaMo 2 has tokens for up to two consecutive LFs, PLaMo 3 for four, and Mistral Small 4 for five. In each of the other eight vocabularies, a single token can hold at least 16 LFs. Seven tokenizers also have entries that include the carriage-return/line-feed pair \r\n (o200k, cl100k, Qwen3.8, Command A+, GLM-5.1, Grok 2, and PLaMo 3). The other six split the pair.
Spaces between letters sometimes share a token with a letter. With 16 spaces between a and b, the whole string took four tokens in Sarashina2.2 and three in every other tokenizer. Twelve of the 13 vocabularies have a token for the full-width space (U+3000). Gemma 4 encodes it as three tokens, one per UTF-8 byte.
Table S2. Token counts for whitespace inserted between a and b. Each cell gives the whole string's token count minus two. The letters can share tokens with adjacent whitespace. LF is a line feed, and \r\n is a carriage return followed by a line feed.
Tokenizer4 spaces16 spaces64 spaces200 spaces1 LF2 LFs4 LFs8 LFs\r\n\r\n
o200k111211111
cl100k111211111
Gemma 4113711114
Qwen3.8111211111
Command A+111211111
DeepSeek V4111311114
Mistral Small 4111311124
GLM-5.1111211111
Grok 2111211111
LLM-jp-41141312484
PLaMo 3112711121
PLaMo 21141311244
Sarashina2.21282512484
In the code sample with a four-space indent before if, all eight byte-based BPE tokenizers and LLM-jp-4 left the last space attached to the word, giving ␣␣␣|␣if. Gemma 4, PLaMo 3, and Sarashina2.2 produced ␣␣␣␣|if. PLaMo 2 can combine the preceding LF and all four spaces into one token. Its vocabulary has 20 entries made up of newlines followed by indentation. PLaMo 3 has none.

4. Code and structured formats

In get_user_profile_by_id, the four underscores account for a four-token difference. The eight byte-based BPE tokenizers attach each underscore to the word that follows it, giving get|_user|_profile|_by|_id. Gemma 4, LLM-jp-4, and Sarashina2.2 give each underscore its own token and need nine. PLaMo also needs five, pairing underscores on either side of a word in get|_user_|profile|_by_|id.
Table 5. Token counts for the code and data samples. Each column holds the input fixed, including its whitespace. Counts exclude automatic start and end tokens.
TokenizerPython functionget_user_profile_by_idHTML fragmentJSON without whitespaceYAMLUUIDSHA-1 (40 hex digits)
o200k3551717221823
cl100k3551716221823
Gemma 44491916283634
Qwen3.83851716283434
Command A+3551716211823
DeepSeek V43551718231824
Mistral Small 43851718243435
GLM-5.13551716222327
Grok 23851716233434
LLM-jp-44791921293735
PLaMo 3445929293635
PLaMo 2415929253635
Sarashina2.25492016293635
Python function
def fibonacci(n: int) -> int:
    if n < 2:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)
HTML fragment
<div class="container"><a href="/index.html">Home</a></div>
JSON without whitespace
{"a":1,"b":[{"c":null},{"d":"e"}]}
YAML
services:
  web:
    image: nginx:latest
    ports:
      - "80:80"
The UUID was 550e8400-e29b-41d4-a716-446655440000.
The SHA-1 digest was da39a3ee5e6b4b0d3255bfef95601890afd80709.
For the eight byte-based BPE tokenizers, that choice starts with the regular expression. In cl100k, [^\r\n\p{L}\p{N}]?\p{L}+ admits a single leading symbol, so a piece can start with _user, (n, or .html. Even a boundary inside a word can depend on the character immediately before it. DeepSeek V4 splits tokenization as token|ization. Add a slash and the first token takes the initial t with it, giving /t|oken|ization.
Table S3. Splits of tokenization after four different characters, on its own, and with an initial capital. A bar marks a token boundary and ␣ a space.
TokenizerAloneAfter a spaceAfter (After _After /Initial capital
o200ktoken|ization␣token|ization(token|ization_token|ization/token|izationToken|ization
cl100ktoken|ization␣token|ization(token|ization_token|ization/token|izationToken|ization
Gemma 4token|ization␣token|ization(|token|ization_|token|ization/|token|izationToken|ization
Qwen3.8token|ization␣token|ization(token|ization_token|ization/token|izationToken|ization
Command A+token|ization␣tokenization(token|ization_token|ization/token|izationToken|ization
DeepSeek V4token|ization␣token|ization(token|ization_token|ization/t|oken|izationToken|ization
Mistral Small 4token|ization␣token|ization(token|ization_token|ization/|token|izationToken|ization
GLM-5.1token|ization␣token|ization(token|ization_token|ization/token|izationToken|ization
Grok 2token|ization␣token|ization(token|ization_token|ization/token|izationToken|ization
LLM-jp-4␣token|ization␣|␣token|ization␣(|token|ization␣_|token|ization␣/|token|ization␣|Token|ization
PLaMo 3token|ization␣token|ization(|token|ization_|token|ization/|token|izationToken|ization
PLaMo 2token|ization␣token|ization(|token|ization_|token|ization/|token|izationToken|ization
Sarashina2.2token|ization␣token|ization(|token|ization_|token|ization/|token|izationToken|ization
Of the 13 tokenizers, PLaMo used the fewest tokens for the sample HTML fragment and the most for the compact JSON sample. Both versions needed nine tokens for the HTML, compared with 17–20 for the other tokenizers. Their vocabularies include <div␣class=" and the complete closing tag </a>. In the JSON, PLaMo separated most quotation marks and brackets and used 29 tokens. The others used 16–21. In o200k, for example, ": and },{" are each a single token.
Digit grouping also shows up in code. The UUID and SHA-1 digest took 18–27 tokens with the five tokenizers that can group ASCII digits, and 34–37 with the eight that handle digits individually. On 300 random hexadecimal strings of 64 characters each, the first group averaged 1.5–1.8 UTF-8 bytes per token and the second about 1.1.
I generated 300 strings of each kind. Each Base64 string encodes 48 random bytes. I sampled hiragana uniformly from U+3041–U+3093 and kanji from U+4E00–U+9FA5, giving rare characters the same chance of appearing as common ones. On random lowercase and alphanumeric strings, the averages across tokenizers ranged from 1.3 to 2.0 bytes per token. On random hiragana, the four Japanese-focused tokenizers averaged 3.3–3.5 bytes per token, a little over one character. The average for cl100k was 2.1.
Table S4. UTF-8 bytes per token for random strings. Each column pools bytes and tokens across 300 strings of the indicated length. The Base64 strings are 64 characters long.
Tokenizer64 hex digitsBase6432 lowercase letters32 alphanumeric characters32 hiragana32 kanji
o200k1.741.461.911.462.721.57
cl100k1.751.391.841.392.111.28
Gemma 41.131.421.951.453.191.44
Qwen3.81.131.361.891.353.051.73
Command A+1.781.541.961.542.931.64
DeepSeek V41.721.421.841.422.621.72
Mistral Small 41.121.311.771.302.861.49
GLM-5.11.491.391.861.393.081.64
Grok 21.141.441.911.442.671.60
LLM-jp-41.111.301.821.313.411.68
PLaMo 31.121.341.911.373.511.19
PLaMo 21.121.331.881.363.491.16
Sarashina2.21.121.351.951.383.291.09

5. Vocabulary by script

Of the ordinary tokens in cl100k, 89.6% are Latin-script entries. In o200k the Latin share is 67.4%. Cyrillic accounts for 7.1%, Arabic for 4.0%, and four Indic scripts together for 4.1%. To compare vocabularies this way, I classified entries by the scripts of their letters and marks. For example, a token consisting of Han characters (kanji in Japanese) and a full stop counts in the Han-only column. The kana column includes hiragana, katakana, mixtures of the two, and kana mixed with kanji.
Table 6. Percentage of each ordinary vocabulary in each script category. Digits, symbols, and whitespace can accompany letters without changing their script classification. Entries that cannot be decoded as UTF-8 on their own have a separate column. Percentages are rounded.
TokenizerLatinHan onlyKana, alone or with kanjiHangulCyrillicArabicIndicDigits, symbols, whitespaceOther scriptsInvalid UTF-8 in isolation
o200k67.43.70.41.27.14.04.13.87.50.8
cl100k89.60.80.20.30.70.10.07.40.10.8
Gemma 460.67.31.91.85.13.211.13.55.40.0
Qwen3.855.721.52.22.77.53.50.82.03.60.4
Command A+73.04.92.03.13.43.02.13.92.32.4
DeepSeek V456.127.50.70.94.12.41.03.82.41.1
Mistral Small 467.92.71.23.45.97.22.93.14.51.1
GLM-5.165.218.40.60.46.41.40.15.90.90.7
Grok 283.55.10.81.03.00.10.35.00.40.8
LLM-jp-440.825.520.27.00.20.20.24.51.40.1
PLaMo 350.516.129.50.90.60.40.11.10.60.1
PLaMo 249.816.131.00.70.50.20.11.20.30.1
Sarashina2.240.416.240.20.11.10.00.01.70.10.1
I used the Unicode Script property and ignored digits, symbols, and whitespace when assigning scripts. A mixture of kana and Latin letters goes under “Other scripts.” The four Indic scripts counted here are Devanagari, Bengali, Tamil, and Telugu. The digits, symbols, and whitespace category includes emoji. Each ordinary token appears in one column.
Gemma 4 has the largest share of entries in the four Indic scripts (11.1%), and Mistral Small 4 has the largest share in Arabic (7.2%). Han-only entries make up 18–28% of the Qwen3.8, DeepSeek V4, and GLM-5.1 vocabularies. In the four Japanese-focused vocabularies, 20–40% of entries contain kana alone or in combination with kanji.
A Han-only entry such as 東京 (“Tokyo”) can occur in both Japanese and Chinese. Some character forms are more distinctive. I compared the character sets of three older encodings, Japan's Shift_JIS and the Chinese encodings GB2312 and Big5. About 25,000 of Qwen3.8's Han-only entries contain a character found only on the Chinese side of that comparison. DeepSeek V4 has about 15,000 and GLM-5.1 about 13,000. LLM-jp-4 also has about 18,000. Together with its large kana vocabulary, that suggests substantial coverage of both Chinese and Japanese strings. PLaMo and Sarashina2.2 each have fewer than 1,000 such entries and more entries with a character specific to Shift_JIS.
I tested each Han character with Python's shift_jis, gb2312, and big5 codecs. A character is specific to the Japanese side if only shift_jis can encode it. Examples include 図, 働, and 込. A character is specific to the Chinese side if either Chinese codec can encode it and shift_jis cannot. Each of the two side-specific columns counts tokens with at least one such character and none specific to the opposite side. “Neither” includes shared characters, as in 東京 and 社会, as well as characters missing from all three encodings. These tests identify differences in repertoire even when a token's language is ambiguous.
Table S5. Ordinary vocabulary entries with kana or Han characters, grouped by script and encoding repertoire. Digits, symbols, and whitespace are ignored. The three Han-only columns omit one LLM-jp-4 entry containing characters specific to both sides.
TokenizerHiragana onlyKatakana onlyMixed kana, or kana with kanjiHan only (neither)Han only (Shift_JIS side)Han only (GB2312/Big5 side)
o200k317403794,4551272,815
cl100k876915467286
Gemma 41,5442,0281,35312,0517646,445
Qwen3.81,8631,6321,96428,11752324,794
Command A+1,8091,4131,8318,0525053,850
DeepSeek V448323915419,69815615,332
Mistral Small 47725892592,484133866
GLM-5.147034611715,18611613,172
Grok 24375001564,4961781,967
LLM-jp-48,6167,53623,52929,1652,97017,953
PLaMo 310,8207,79213,12913,9082,644748
PLaMo 211,0387,37712,56512,9662,615540
Sarashina2.28,2328,42324,54413,3343,163113
These vocabularies give quite different token counts on Japanese text. Across the 200 Japanese documents, UTF-8 bytes per token ranged from 2.62 with cl100k to 6.48 with PLaMo 3. The same text therefore took about 2.5 times as many tokens with cl100k as with PLaMo 3. On the 200 English documents, Sarashina2.2 averaged 4.4 bytes per token and the other 12 averaged 5.3–6.5.
I divided the total number of UTF-8 bytes in each group by the total number of tokens for that group. T2 covers Simplified Chinese, Traditional Chinese, and Korean (150 documents). T3 covers 12 language and script categories (600 documents), T4 covers 36 (1,800), and T5 covers 96 (4,800). The four Japanese-focused tokenizers averaged 1.6–2.4 bytes per token on the T4 and T5 web excerpts. In these groups they often fell back to bytes for characters missing from their vocabularies.
Table S6. UTF-8 bytes per token, calculated from totals within each group. All 13 tokenizers processed the same documents in a column. The English and Japanese columns each cover 200 texts, code covers 100 samples, and T2–T5 group the other languages and scripts.
TokenizerEnglishJapaneseCodeChinese and Korean (T2)T3 (12 categories)T4 (36 web categories)T5 (96 web categories)
o200k5.523.613.703.834.904.682.96
cl100k5.472.623.502.403.292.152.22
Gemma 45.404.903.414.425.034.503.23
Qwen3.85.395.093.604.975.203.762.73
Command A+5.585.203.904.644.943.232.69
DeepSeek V45.474.193.644.324.143.312.60
Mistral Small 45.403.623.563.584.613.472.47
GLM-5.15.473.953.723.684.052.532.46
Grok 25.523.983.713.993.592.492.39
LLM-jp-45.286.353.304.992.582.432.36
PLaMo 36.296.483.413.153.241.962.13
PLaMo 26.486.403.553.102.981.702.07
Sarashina2.24.436.343.102.072.451.621.93

6. Japanese text

In the 200 Japanese documents, the four Japanese-focused tokenizers split 20–23% of adjacent kanji pairs. Gemma 4, Qwen3.8, and Command A+ split 38–43%, five others split 51–80%, and cl100k split 94%. The Japanese-focused tokenizers also kept katakana together more often. They split just 5–6% of adjacent katakana pairs, compared with 18–91% for the other nine.
Table 7. Percentage of adjacent character pairs split by a token boundary in the 200 Japanese documents. Read each pair from left to right. The Japanese punctuation category includes brackets, the middle dot, and the prolonged sound mark.
TokenizerKanji, kanjiHiragana, hiraganaKatakana, katakanaKanji to hiraganaHiragana to kanjiHiragana to Japanese punctuation
o200k7256529899100
cl100k948491100100100
Gemma 44226186989100
Qwen3.83824206787100
Command A+4324256687100
DeepSeek V45137639096100
Mistral Small 48043579198100
GLM-5.16140559296100
Grok 26640529197100
LLM-jp-4211954669100
PLaMo 323205547633
PLaMo 223215577832
Sarashina2.2201964570100
A Japanese word written with kanji can have a hiragana ending called okurigana. For instance, 食べ, the stem of “eat,” combines the kanji 食 with the hiragana べ. A switch to hiragana can also mark the start of a grammatical particle. The two OpenAI encodings split at almost every kanji-to-hiragana transition in the sample. The four Japanese-focused tokenizers kept the characters together about half the time.
When Japanese punctuation followed hiragana, both PLaMo versions kept the two in one token about two-thirds of the time. Both vocabularies include sentence endings such as です。 and ました。, complete with the full stop. In the greeting below, PLaMo used just two tokens, one per sentence. Qwen3.8 recognized both phrases but gave each full stop its own token, for a total of four.
Table 8. Tokenizations of ありがとうございます。よろしくお願いいたします。 (“Thank you. I look forward to working with you.”). A bar marks a boundary and ␣ a space. Angle brackets enclose the hexadecimal bytes of tokens that cannot be decoded independently.
TokenizerSplitTokens
o200kありがとうございます|。|よろしく|お願|いい|た|します|。8
cl100kありがとうござ|います|。|よ|ろ|<e38197e381>|<8f>|お|<e9a1>|<98>|い|い|た|します|。15
Gemma 4ありがとうございます|。|よろしく|お願いいたします|。5
Qwen3.8ありがとうございます|。|よろしくお願いいたします|。4
Command A+ありがとうございます|。|よろしくお願い|いたします|。5
DeepSeek V4あり|が|とう|ござ|います|。|よ|ろ|しく|お|願|い|いた|します|。15
Mistral Small 4ありがとうございます|。|よろしく|お願|いい|たし|ます|。8
GLM-5.1ありがとうござ|います|。|よ|ろ|しく|お|願い|いたします|。10
Grok 2あり|が|とう|ご|ざ|い|ます|。|よ|ろ|しく|お|願|い|いた|します|。17
LLM-jp-4␣ありがとうございます|。|よろしくお願いいたします|。4
PLaMo 3ありがとうございます。|よろしくお願いいたします。2
PLaMo 2ありがとうございます。|よろしくお願いいたします。2
Sarashina2.2ありがとうござい|ます|。|よろしくお願い|いたします|。6
Among the longest entries containing kana are greetings and phrases familiar from Japanese websites. Both PLaMo 3 and LLM-jp-4 have a token for ご覧いただきありがとうございます (“Thank you for reading”). Given that phrase on its own, PLaMo 3 returns just that token. LLM-jp-4 adds a separate space marker before it. PLaMo 2 splits the phrase itself in two, ご覧|いただきありがとうございます.
Table S7. Three of the longest ordinary entries in each vocabulary's kana category (kana alone or mixed with kanji). Length counts Unicode code points, including any leading space, shown as ␣.
TokenizerLongest kana entries
o200kありがとうございました, ありがとうございます, ␣風吹けば名無し
cl100kありがとうござ, ありがとう, こんにちは
Gemma 4ありがとうございました, のではないでしょうか, ありがとうございます
Qwen3.8よろしくお願いいたします, ␣ありがとうございました, ありがとうございました
Command A+ありがとうございました, のではないでしょうか, ありがとうございます
DeepSeek V4かもしれません, ではありません, してください
Mistral Small 4ありがとうございました, ありがとうございます, ラクターデザイン
GLM-5.1かもしれません, ありがとうござ, してください
Grok 2␣外部リンク, キャラクター, してください
LLM-jp-4ご覧いただきありがとうございます, ブログランキングに参加しています, このレビューは参考になりましたか
PLaMo 3ご覧いただきありがとうございます, いただきありがとうございました。, あけましておめでとうございます。
PLaMo 2いただきありがとうございました。, あけましておめでとうございます。, 明けましておめでとうございます
Sarashina2.2だろうと思います, お願いいたします, にお越しください
Personal names illustrate another difference. LLM-jp-4 and Sarashina2.2 have single tokens for 徳川家康 (Tokugawa Ieyasu) and 大谷翔平 (Shohei Ohtani). PLaMo split Tokugawa Ieyasu's name into two tokens, 徳川|家康, one for the family name and one for the given name. Most of the other nine tokenizers used one token per character for that name.

7. English words and spaces

A space before an English word often belongs to the same token as the word. In 22,195 word samples from the 200 English documents, a word with its leading space took one token about 95–99% of the time for 12 tokenizers. For Sarashina2.2 the share was 73.4%. It sometimes encoded the space separately.
Table 9. Case and space variants in the ordinary vocabulary, and single-token coverage of 22,195 English word samples. Variant counts cover sequences of at least two ASCII letters with an optional leading space. For the last column, I tested each distinct spelling once per document, with one leading space.
TokenizerASCII letter entriesDistinct forms ignoring case and leading spaceExtra variants / ordinary vocabularyEntries spanning spaces between ASCII lettersWords encoded with their leading space in one token
o200k105,79467,81619.0%098.3%
cl100k68,59140,59227.9%097.8%
Gemma 4139,42985,97520.4%099.2%
Qwen3.8104,06067,55714.7%097.9%
Command A+143,69490,35920.9%099.3%
DeepSeek V463,82141,69817.3%098.0%
Mistral Small 470,68049,57316.2%096.6%
GLM-5.175,74446,28819.0%097.8%
Grok 289,12756,38725.0%098.9%
LLM-jp-478,32951,02613.9%098.5%
PLaMo 343,82824,52018.0%8,19195.0%
PLaMo 237,62121,08116.5%10,05095.7%
Sarashina2.240,32123,95616.0%073.4%
I extracted runs of ASCII letters preceded by a space and followed by a space, comma, or period. I counted each distinct spelling once per document and encoded it with one leading space. To account for LLM-jp-4's automatic space marker, I encoded a followed by a space and the word, then subtracted the token count for a alone.
A vocabulary can store several forms of the same English word. For example, the, ␣the, The, ␣The, THE, and ␣THE all reduce to the if the leading space is removed and the letters are lowercased. I applied the same rule to all entries consisting of two or more ASCII letters with an optional leading space. A group of six entries contributes five extra variants. Across all groups, the extra variants made up 27.9% of cl100k's ordinary vocabulary and 13.9% of LLM-jp-4's, the highest and lowest shares.
Only PLaMo had ordinary entries that span the space between English words. Counting entries with an ASCII letter on either side of one or more spaces gave 8,191 for PLaMo 3 and 10,050 for PLaMo 2. They include ␣of␣the, ␣one␣of␣the, and ␣as␣well␣as. A tokenizer that splits at each word before applying its vocabulary cannot use such a phrase as one token. These phrases likely help explain why PLaMo covered 6.3–6.5 bytes per token in the English sample.
The English and code samples also show which neighboring characters stayed together. Both PLaMo versions joined the space to the digit at all 623 space-to-digit positions, as in ␣1. Where an ASCII symbol preceded a lowercase letter, the eight byte-based BPE tokenizers put a boundary between them 36–42% of the time. The rate for the other five was 86–100%.
Table S8. Percentage of adjacent character pairs split by a token boundary, pooled across 200 English texts and 100 code samples. Each column counts only pairs of the indicated types, in the order shown. “Symbol” covers ASCII punctuation and symbols. Percentages are rounded to whole numbers.
TokenizerLowercase to spaceSpace to lowercaseSpace to uppercaseSymbol to lowercaseSymbol to spaceSpace to digitDigit, digitSymbol to newline
o200k1000036100100100
cl100k1000036100100100
Gemma 410032100100100100100
Qwen3.81000036100100100100
Command A+1000036100100100
DeepSeek V41000042100100100
Mistral Small 410000391001001000
GLM-5.11000036100100160
Grok 210000381001001000
LLM-jp-410015100100100100100
PLaMo 38531786650100100
PLaMo 2844158665010069
Sarashina2.21001126100100100100100
In the contraction don't, the apostrophe can stand alone, join the t, or stay inside the whole word.
  • o200k and Command A+ produced don't as one token.
  • cl100k, Qwen3.8, DeepSeek V4, Mistral Small 4, GLM-5.1, and Grok 2 produced don|'t.
  • Gemma 4, PLaMo, and Sarashina2.2 produced don|'|t. LLM-jp-4 also used three tokens, with its initial space marker attached to the first, ␣don|'|t.
Capitalization affected every tokenizer in one test. The sentence The quick brown fox jumps over the lazy dog. took 9–10 tokens as written and 12–18 in all capitals.

8. Other languages, emoji, and Unicode normalization

For each of seven more languages, I ran one short sentence through all 13 tokenizers. Gemma 4, which had the largest share of Indic-script entries, used 14 tokens for the Hindi sentence. The same sentence took at least 45 tokens with cl100k and the four Japanese-focused tokenizers, reaching 71 with Sarashina2.2. LLM-jp-4 tied for the fewest tokens on the Chinese sentence (10) and used the fewest on the Korean one (12). Across all seven examples, o200k, Gemma 4, Qwen3.8, and Command A+ used 10–23 tokens per sentence.
Table 10. Token counts for one sentence per language. Each column compares the 13 tokenizers on one input. The sentences vary in length and wording across languages.
TokenizerChinese (Simplified)KoreanRussianArabicHindiThaiGerman
o200k13171415151820
cl100k22302630494126
Gemma 412151416141520
Qwen3.810151414231223
Command A+11131614171519
DeepSeek V410201717272025
Mistral Small 416171513202424
GLM-5.110231519423925
Grok 213182030374222
LLM-jp-410123437484527
PLaMo 314293127452929
PLaMo 215293230633829
Sarashina2.225462340717228
Six sentences describe good weather and a walk in the park. The German sentence uses two long compound nouns. I encoded each sentence separately.
Chinese (Simplified)
今天天气很好,我们一起去公园散步吧。
Korean
오늘은 날씨가 좋아서 공원에 산책하러 갑니다.
Russian
Сегодня хорошая погода, пойдём гулять в парк.
Arabic
الطقس جميل اليوم، لنذهب للتنزه في الحديقة.
Hindi
आज मौसम अच्छा है, चलो पार्क में टहलने चलते हैं।
Thai
วันนี้อากาศดี ไปเดินเล่นที่สวนสาธารณะกันเถอะ
German
Die Donaudampfschifffahrtsgesellschaft überprüft die Geschwindigkeitsbegrenzung.
The number of visible characters can be a poor guide to the token count. The woman scientist emoji 👩‍🔬 consists of three Unicode code points (woman, zero-width joiner, microscope). Gemma 4 and GLM-5.1 used one token for each. Six other tokenizers split inside some of those code points and needed eight or nine tokens. Mistral Small 4 has just ten ordinary entries containing a pictographic character, compared with 1,219 in Gemma 4. It split even the single code point 😂 into four tokens, one per UTF-8 byte.
Table 11. Pictographic vocabulary entries and token counts for Unicode inputs. The vocabulary count covers distinct ordinary token strings that decode as UTF-8 on their own and contain a character with the Unicode property Extended_Pictographic. The remaining columns count tokens for the inputs shown. The two spellings of café end in U+00E9 and U+0065 U+0301, respectively.
TokenizerPictographic entries😂👩‍🔬🇯🇵🇺🇸🇫🇷One full-width spaceハンカクカタカナガギグcafé, precomposed / decomposed
o200k7415121252 / 3
cl100k2128181282 / 3
Gemma 41,2191363142 / 2
Qwen3.83628181152 / 2
Command A+1161491282 / 2
DeepSeek V45115121283 / 3
Mistral Small 41049241353 / 4
GLM-5.182313181282 / 3
Grok 23015121282 / 3
LLM-jp-41,0562772151 / 2
PLaMo 32221891142 / 2
PLaMo 22201891142 / 2
Sarashina2.27719241143 / 2
The tokenizers also differed on less common forms of Japanese characters. The half-width katakana string ハンカクカタカナガギグ contains 14 code points, counting each voicing mark separately. Gemma 4, Qwen3.8, and the four Japanese-focused tokenizers used 14–15 tokens. The other seven split some characters into byte fragments and used 25–35.
Two spellings can look identical and still produce different IDs. The last column shows café with a single é and with e followed by a combining acute accent (U+0301). Qwen3.8 alone applies Unicode normalization. It uses Normalization Form C (NFC), which gives canonically equivalent strings the same representation[2]. The two spellings produce identical token IDs in Qwen3.8 and different ID sequences in all 12 other tokenizers, even when the token counts match.
I also checked whether encoding and then decoding each of 32 inputs restored the original string. Three kinds of change appeared.
  • Qwen3.8 returned the NFC forms of decomposed accents, Japanese voicing marks, and Hangul jamo (the letters that make up a syllable).
  • Gemma 4, LLM-jp-4, and Sarashina2.2 use ▁ (U+2581) internally for a space. A literal a▁b in the input came back as a b.
  • PLaMo 3 uses U+EE00 as an internal boundary marker. When the input contained that character between a and b, decoding returned ab.

9. Repeated characters

A separator line can be one token. Seven of the eight byte-based BPE tokenizers encoded 64 equals signs as one token and did the same for 64 periods. Mistral Small 4 needed two tokens for the equals signs and four for the periods. One o200k token consists of 112 hyphens. Command A+ has a token for 128 hyphens.
Table 12. Token counts for runs of one character. In the Japanese examples, あ is hiragana, ー lengthens a vowel, 。 ends a sentence, and 笑 indicates laughter. The last column gives the length of the longest ordinary token made of one repeated non-whitespace character.
Tokenizera × 64= × 64. × 64あ × 32ー × 16。 × 16笑 × 16Longest non-whitespace run (characters)
o200k811324816112
cl100k811321681696
Gemma 484416441616
Qwen3.88111628896
Command A+411164816128
DeepSeek V4811321648128
Mistral Small 43224161681676
GLM-5.181132164896
Grok 241116161616128
LLM-jp-41755924916
PLaMo 3101322736810
PLaMo 2161322736810
Sarashina2.2148842488
In either PLaMo vocabulary, the longest run of a single non-whitespace character stored as one token is ten characters. Both versions took 22 tokens for the 64 periods. PLaMo 3's preprocessing also puts boundaries on either side of a run of four or more copies of the same non-newline character. For 32 copies of あ, Sarashina2.2 used four tokens and both PLaMo versions used seven. The two OpenAI encodings, DeepSeek V4, and GLM-5.1 needed 32. For 16 copies of the Japanese prolonged sound mark ー, tokenizers either needed two to four tokens or split all 16 characters.

10. Special tokens

A special token can mark the end of a turn or the position of an image. What happens if the user types the marker itself? Given the literal text <|im_end|>, Qwen3.8 returns the same ID that its chat template uses to end a turn. Whether that ID actually ends the turn depends on the model and the software that receives it.
Table 13. Registered special tokens and literal end markers. Apparent reserved slots are identified by name and can be ordinary or special entries. The last column gives the library's response to the marker shown, using the test's tiktoken adapter for Grok 2.
TokenizerRegistered special tokensApparent reserved slotsLiteral end markerDefault encoding result
o200k1,0901,081<|endoftext|>Error (tiktoken default)
cl100k50<|endoftext|>Error (tiktoken default)
Gemma 4246,227<eos>One special token
Qwen3.8210<|im_end|>One special token
Command A+150<|END_OF_TURN_TOKEN|>One special token
DeepSeek V41,2301,217<|end▁of▁sentence|>One special token
Mistral Small 41,000972</s>One special token
GLM-5.1180<|endoftext|>One special token
Grok 2128125<|eos|>Error (tiktoken adapter used here)
LLM-jp-4352338<|return|>One special token
PLaMo 35927<|plamo:eos|>One special token
PLaMo 22923<|plamo:eos|>One special token
Sarashina2.270</s>One special token
For each of the ten Hugging Face tokenizers I tested, the end marker listed in the table encoded as one special token. With tiktoken's default encode, the two OpenAI encodings and the Grok 2 adapter raised an error for their respective markers. In the ordinary-text comparisons I used tiktoken's encode_ordinary, which treats those spellings as text.
A high special-token count can reflect reserved IDs. Sequentially named slots accounted for 1,081 of o200k's 1,090 registered special tokens, 1,217 of DeepSeek V4's 1,230, and 972 of Mistral Small 4's 1,000. Gemma 4 registers just 24 tokens as special and has 6,227 entries named <unused…> in its ordinary vocabulary.
An ordinary entry can also look like a control marker. Qwen3.8 and GLM-5.1 have ordinary tokens for <think> and <tool_call>. Each PLaMo version has 76 ordinary entries spelled like HTML tags, including </div>.

11. Image, audio, and video tokens

Six of the 13 vocabularies contain special or added tokens named for images, audio, or video. Of the model configurations I examined, four accept an image as input. Gemma 4 also accepts audio and video, and Qwen3.8 accepts video. The two OpenAI entries in this comparison are encodings distributed independently of a model's input processor.
Table S9. Special and added token names for images, audio, video, and regions within an image. The input column describes the pinned model's processor. A dash marks the two standalone encodings, and “None” in a marker column means no matching name was found.
TokenizerInputs besides textImage and videoAudioImage regions and coordinates
o200k—NoneNoneNone
cl100k—NoneNoneNone
Gemma 4Images, audio, video<|image>, <|image|>, <image|>, <|video|><|audio>, <|audio|>, <audio|>None
Qwen3.8Images, video<|vision_start|>, <|vision_end|>, <|vision_pad|>, <|image_pad|>, <|video_pad|><|audio_start|>, <|audio_end|>, <|audio_pad|>, <tts_pad>, <tts_text_bos>, <tts_text_eod>, <tts_text_bos_single><|object_ref_start|>, <|object_ref_end|>, <|box_start|>, <|box_end|>, <|quad_start|>, <|quad_end|>
Command A+Images<|START_OF_IMG|>, <|END_OF_IMG|>, <|IMG_LINE_BREAK|>, <|IMG_PATCH|>NoneNone
DeepSeek V4None<|image|>, <|image2|>, <|rl_image_start|>, <|rl_image_pad|>None<|ref|>, <|/ref|>, <|box|>, <|/box|>, <|point|>, <|/point|>, <|polygon|>, <|/polygon|>
Mistral Small 4Images[IMG], [IMG_BREAK], [IMG_END][AUDIO], [BEGIN_AUDIO]None
GLM-5.1None<|begin_of_image|>, <|end_of_image|>, <|begin_of_video|>, <|end_of_video|>, <|image|>, <|video|><|begin_of_audio|>, <|end_of_audio|>, <|begin_of_transcription|>, <|end_of_transcription|><|begin_of_box|>, <|end_of_box|>
Grok 2NoneNoneNoneNone
LLM-jp-4NoneNoneNoneNone
PLaMo 3NoneNoneNoneNone
PLaMo 2NoneNoneNoneNone
Sarashina2.2NoneNoneNoneNone
In all four image models, a separate encoder turns pixels into vectors. The input processor repeats the same placeholder ID once per vector and adds delimiters. The model replaces each placeholder's embedding with the corresponding image vector. Each placeholder and delimiter takes up one position in the context window.
An image ID can also carry information about the pixels. Chameleon quantizes an image into discrete codes, each with an ID in the vocabulary shared with text[3]. The model can generate those IDs one by one, and an image decoder reconstructs an image from them.
  1. [3] Chameleon Team, "Chameleon: Mixed-Modal Early-Fusion Foundation Models," arXiv:2405.09818, 2024.
With the default image-processing settings, a 1920 × 1080 image occupied 266 token positions in Gemma 4, 1,736 in Mistral Small 4, 2,042 in Qwen3.8, and 3,343 in Command A+. In Gemma 4, 264 positions held image vectors and the other two marked the start and end of the image.
Table 14. Token positions per image with each processor's default settings. Dimensions are width × height in pixels. Counts include image-vector placeholders and any start, end, or separator tokens.
Image processor224 × 224512 × 5121024 × 7681920 × 10804032 × 3024
Gemma 4258258268266268
Qwen3.8662587702,04211,846
Command A+2592591,2873,3433,343
Mistral Small 4723801,0641,7362,352
I loaded each pinned processor configuration and chat template with AutoProcessor in Transformers 5.23.0, using PyTorch 2.15.0, torchvision 0.29.2, and Pillow 13.0.0. This includes the Transformers processor distributed with Mistral Small 4. I encoded a conversation containing one all-black image and counted the image placeholders and delimiters. I also checked where each model's implementation replaces the placeholder embeddings with image vectors.
Table S10. Placeholders and delimiters for one 1920 × 1080 image with the pinned processors' default settings. Each placeholder reserves one position for an image vector.
Image processorPlaceholders for image vectorsStart, end, and separator tokens
Gemma 4264 × <|image|>2 (start and end)
Qwen3.82,040 × <|image_pad|>2 (start and end)
Command A+3,328 × <|IMG_PATCH|>2 (start and end), 13 × <|IMG_LINE_BREAK|>
Mistral Small 41,705 × [IMG]30 × [IMG_BREAK], 1 × [IMG_END]
The processors turn image dimensions into token counts in four ways.
  • Gemma 4 resizes the image to fit a vector budget. The default limit is 280 vectors. With the same 1024 × 768 image, the five available limits (70 through 1,120) gave between 63 and 1,064 vectors.
  • Qwen3.8 uses one vector per 32 × 32 pixels after resizing. Its configured minimum and maximum pixel counts correspond to 64 and 16,384 vectors.
  • Command A+ uses 256 vectors per 512 × 512 tile. It divides an image into up to 12 tiles and adds a downscaled overview when it uses more than one. Both of the largest test images used 12 tiles plus the overview, or 3,328 vectors.
  • Mistral Small 4 scales the longer side down if it exceeds 1,540 pixels, adjusts both dimensions to multiples of 28, and uses one vector per 28 × 28 pixels.
The delimiters explain the rest of the count. Mistral Small 4 puts [IMG_BREAK] between rows of vectors and closes the last row with [IMG_END]. Command A+ puts <|IMG_LINE_BREAK|> after each tile. Gemma 4 and Qwen3.8 mark the start and end of the whole image and pass coordinates or grid dimensions to the model in separate arrays.
Audio and video use placeholders too. At its default settings, Gemma 4 uses about one audio placeholder per 40 milliseconds, up to 750 for 30 seconds of audio. For video it samples 32 frames and inserts a timestamp and up to 70 video placeholders per frame. Qwen3.8 samples at a target of two frames per second, subject to its frame-count limits. It groups frames in pairs and inserts a timestamp and placeholders for each pair. On the same ten-second clip, Gemma 4 produced 2,112 video placeholders and Qwen3.8 produced 2,200.
I used silence sampled at 16 kHz for audio and black 640 × 360 frames at 30 frames per second for video. In Gemma 4, one second of audio produced 25 <|audio|> placeholders, ten seconds produced 250, and both 30 and 45 seconds produced 750. For the ten-second video, Gemma 4 inserted 66 <|video|> placeholders for each of 32 frames. Qwen3.8 inserted 220 <|video_pad|> placeholders for each of ten frame pairs. These counts exclude delimiters and timestamps.
The pinned DeepSeek V4 and GLM-5.1 models accept only text despite having image markers. GLM-5.1 also has audio markers. Qwen3.8's seven audio entries and Mistral Small 4's [AUDIO] and [BEGIN_AUDIO] likewise go unused by these processors. One possible reason to keep these entries is to share a vocabulary with other models in the same family. In o200k, cl100k, Grok 2, and the four Japanese-focused vocabularies, I found no token names identifying images, audio, or video. The numbered reserved slots in o200k and Grok 2 give no indication of their purpose.
DeepSeek V4 also registers 415 special tokens with names from <|place_holder_mm_span_0021|> through <|place_holder_mm_span_0435|>. I counted these as apparent reserved slots in the special-token comparison. Six other added tokens delimit tables, rows, and cells. The opening tags are <|table|>, <|tr|>, and <|td|>.

12. Vocabulary overlap and boundary agreement

How much of cl100k can be found in another vocabulary? I compared each ordinary token's byte sequence, ignoring its numeric ID. GLM-5.1 contains 99% of cl100k's distinct byte sequences, and Qwen3.8 contains 97%. Reversing the comparison changes the denominator. For the two OpenAI vocabularies, o200k contains 85% of cl100k's sequences, and cl100k contains 43% of o200k's.
Table 15. Percentage of each OpenAI vocabulary's distinct ordinary byte sequences present in the row vocabulary. The denominator is the vocabulary named in the column. Matching byte sequences may have different IDs.
TokenizerShare of cl100k presentShare of o200k present
o200k85%100%
cl100k100%43%
Gemma 467%66%
Qwen3.897%68%
Command A+88%74%
DeepSeek V459%41%
Mistral Small 456%50%
GLM-5.199%54%
Grok 277%50%
LLM-jp-442%27%
PLaMo 336%23%
PLaMo 235%20%
Sarashina2.229%17%
The order of the token IDs offers a clue to the history of these vocabularies. I divided each ID range into ten equal bins. In GLM-5.1, Latin-script tokens make up over 90% of each of the first six bins. Han characters become frequent in the next two. In Qwen3.8, Latin entries make up about 94% of each of the first three bins, while kanji or kana account for 98–100% of the fifth and sixth bins. This ordering, together with the overlap in byte sequences, suggests that the two vocabularies grew out of a base similar to cl100k, with Chinese and other entries added later. The unused GLM-5.1 digit strings could be among those additions. They are absent from cl100k.
I divided the range from the smallest to the largest ID of an ordinary, independently decodable UTF-8 token into ten equal intervals and counted scripts within each interval. In both PLaMo versions, Latin entries predominate in the first half and kanji or kana in much of the second. Script shares change comparatively little across the ID range in o200k, Mistral Small 4, Command A+, and Grok 2.
Table S11. Script shares by position in the ID range. Each cell gives two percentages, Latin followed by kanji or kana, among ordinary tokens in the interval that decode independently as UTF-8. Column headings give the interval's lower bound.
Tokenizer0%10%20%30%40%50%60%70%80%90%
o200k72/469/469/468/468/468/467/466/466/566/4
cl100k88/088/087/191/192/192/191/191/292/191/1
Gemma 475/268/465/664/662/662/762/761/861/828/38
Qwen3.894/094/094/082/130/1000/9835/654/555/552/10
Command A+78/578/677/775/775/774/774/773/873/872/8
DeepSeek V461/2560/2559/2559/2758/2856/2954/3154/3154/3252/33
Mistral Small 472/371/470/469/469/468/468/468/467/466/4
GLM-5.191/092/092/092/092/093/032/630/9728/2644/4
Grok 286/385/585/584/684/684/684/683/683/783/7
LLM-jp-458/3451/4247/4544/4741/5039/5240/5039/5146/454/43
PLaMo 392/0100/0100/0100/065/333/974/963/974/9636/38
PLaMo 292/0100/0100/0100/069/282/983/973/974/9627/56
Sarashina2.255/3949/4846/5143/5441/5636/6135/6333/6532/6533/64
For a comparison that treats both vocabularies equally, I used the Jaccard index. It counts the distinct byte sequences they share and divides by the number of distinct sequences in their union. On that measure, cl100k and GLM-5.1 overlap by 64%, even though GLM-5.1 contains nearly all of cl100k. PLaMo 2 and PLaMo 3 overlap by 84%. For pairs consisting of one Japanese-focused tokenizer and one of the other nine, the index ranges from 13% to 28%.
PLaMo 3 contains 95% of the ordinary byte sequences in PLaMo 2. All 13 vocabularies share 18,201 distinct byte sequences.
Table S12. Vocabulary Jaccard index (%). Each cell counts the distinct ordinary byte sequences shared by the row and column vocabularies and divides by the number in their union. Values are rounded to whole percentages.
Tokenizero200kcl100kGemma 4Qwen3.8Command A+DeepSeek V4Mistral Small 4GLM-5.1Grok 2LLM-jp-4PLaMo 3PLaMo 2Sarashina2.2
o200k—404043483344444316171613
cl100k40—2339333533645017212116
Gemma 44023—38443133313424181614
Qwen3.8433938—454135553524181613
Command A+48334445—3337394322171614
DeepSeek V43335314133—40494428232216
Mistral Small 4443333353740—354417201814
GLM-5.144643155394935—4724211915
Grok 24350343543444447—23232218
LLM-jp-4161724242228172423—323133
PLaMo 317211818172320212332—8444
PLaMo 21621161616221819223184—44
Sarashina2.2131614131416141518334444—
Vocabulary overlap and agreement on token boundaries answer different questions. I counted the positions where both tokenizers split a document as a share of the positions where either one did, pooling the counts across documents. On the Japanese texts, cl100k and GLM-5.1 agreed at 66% of those positions. The closest match to cl100k was o200k at 72%. The two PLaMo versions reached 96%, and LLM-jp-4 and Sarashina2.2 reached 90%.
Gemma 4, Qwen3.8, and Command A+ agreed at 88–90% for every pair in that group. DeepSeek V4, GLM-5.1, and Grok 2 also fell in the 88–90% range for every pair. I compared byte positions, including those inside characters, and excluded the start and end of each document.
Table S13. Shared token boundaries on 200 Japanese texts (%). Each cell gives shared boundaries as a percentage of all positions split by either tokenizer. Counts are pooled across documents, including byte offsets inside characters and excluding document endpoints. Values are rounded.
Tokenizero200kcl100kGemma 4Qwen3.8Command A+DeepSeek V4Mistral Small 4GLM-5.1Grok 2LLM-jp-4PLaMo 3PLaMo 2Sarashina2.2
o200k—727068688283868552525452
cl100k72—5250506271666539394039
Gemma 47052—90888070767667646367
Qwen3.8685090—887868757369666468
Command A+68508888—7768747564626264
DeepSeek V48262807877—80908859585959
Mistral Small 4837170686880—828452525352
GLM-5.186667675749082—8856555656
Grok 28565767375888488—55555555
LLM-jp-4523967696459525655—757490
PLaMo 352396466625852555575—9677
PLaMo 25440636462595356557496—76
Sarashina2.2523967686459525655907776—
The English and code results show how much the input matters. On the English texts, cl100k and GLM-5.1 agreed at 99.99% of boundary positions. Across all pairs of byte-based BPE tokenizers, agreement was about 96–100%. Their similar rules for dividing words and spaces likely explain much of that agreement. Code also contains many symbols next to identifiers. On the code samples, agreement between those eight tokenizers and the three that separated the underscores in get_user_profile_by_id was 66–84%.
Both PLaMo versions can span spaces between English words. On the English texts, their agreement with the nine BPE tokenizers and with LLM-jp-4 was 76–78%.
Table S14. Shared token boundaries on 200 English texts (%). Each cell gives shared boundaries as a percentage of all positions split by either tokenizer. Counts are pooled across documents, including byte offsets inside characters and excluding document endpoints. Values are rounded.
Tokenizero200kcl100kGemma 4Qwen3.8Command A+DeepSeek V4Mistral Small 4GLM-5.1Grok 2LLM-jp-4PLaMo 3PLaMo 2Sarashina2.2
o200k—989697999897989893777879
cl100k98—96989898981009993777779
Gemma 49696—97969695969797787681
Qwen3.8979897—969796989795787680
Command A+99989696—9897989994777879
DeepSeek V49898969798—98989993777779
Mistral Small 4979895969798—989893767779
GLM-5.1981009698989898—9993777779
Grok 29899979799999899—94777879
LLM-jp-4939397959493939394—777683
PLaMo 377777878777776777777—9567
PLaMo 27877767678777777787695—66
Sarashina2.2797981807979797979836766—
On the code samples, Gemma 4, LLM-jp-4, and Sarashina2.2 agreed with one another at 85–87% of boundary positions.
Table S15. Shared token boundaries on 100 code samples (%). Each cell gives shared boundaries as a percentage of all positions split by either tokenizer. Counts are pooled across documents, including byte offsets inside characters and excluding document endpoints. Values are rounded.
Tokenizero200kcl100kGemma 4Qwen3.8Command A+DeepSeek V4Mistral Small 4GLM-5.1Grok 2LLM-jp-4PLaMo 3PLaMo 2Sarashina2.2
o200k—937688939393979575646069
cl100k93—7385888891949072615866
Gemma 47673—84787675767887827785
Qwen3.8888584—908786908983716676
Command A+93887890—9290939477656170
DeepSeek V49388768792—93939476656270
Mistral Small 4939175869093—939475646169
GLM-5.197947690939393—9575646069
Grok 29590788994949495—77656270
LLM-jp-4757287837776757577—797486
PLaMo 364618271656564646579—9379
PLaMo 26058776661626160627493—74
Sarashina2.2696685767070696970867974—

13. Vocabulary usage and token length

A few frequent tokens can account for much of a text even when the vocabulary is large. On the 7,850 documents, the 80 most frequent Sarashina2.2 tokens accounted for half of all its token occurrences. It took 1,289 distinct Gemma 4 tokens to reach the same share. These frequencies reflect the mix of languages in the corpus, where 7,350 documents come from outside the Japanese, English, and code groups.
Table 16. Vocabulary usage across all 7,850 documents. The share used is relative to the ordinary vocabulary. The 50% and 90% columns count distinct tokens in descending order of frequency. The last column gives the share of all token occurrences that end inside a UTF-8 character.
TokenizerTotal token occurrencesShare of ordinary vocabulary usedDistinct tokens covering 50% of occurrencesDistinct tokens covering 90% of occurrencesOccurrences ending inside a character
o200k2,189,11751.2%95928,67210.0%
cl100k3,244,03836.1%2093,54619.5%
Gemma 42,053,72540.3%1,28927,5540.2%
Qwen3.82,378,24135.7%69516,6298.0%
Command A+2,485,56737.1%45816,35613.3%
DeepSeek V42,567,84945.7%5349,7969.4%
Mistral Small 42,634,00854.7%36813,38715.5%
GLM-5.12,836,13637.4%3327,65414.6%
Grok 22,919,10641.5%2936,68715.5%
LLM-jp-43,013,27024.5%2864,1762.3%
PLaMo 33,344,64841.8%1474,42519.2%
PLaMo 23,566,52438.3%1163,14922.1%
Sarashina2.23,856,08633.2%802,35723.3%
Byte fallback helps explain that concentration. PLaMo and Sarashina2.2 often broke unfamiliar characters into their UTF-8 bytes, reusing the same single-byte tokens. About 19–23% of their token occurrences ended inside a character. Among the other character-based tokenizers, the figure was 0.2% for Gemma 4 and 2.3% for LLM-jp-4. The eight byte-based tokenizers start from bytes even for a character that has its own vocabulary entry. For them the percentage counts any boundary left inside a character after BPE merging, so it cannot directly measure how often a character is missing from the vocabulary.
In the five character-based vocabularies, the longest ordinary entries range from 16 Unicode code points in LLM-jp-4 to 32 in PLaMo 3. All eight byte-based vocabularies have entries at least 76 characters long. Command A+ and GLM-5.1 each have a token for 512 spaces, and Grok 2 has one for 256.
Table S16. Lengths of ordinary vocabulary entries, including whitespace. Character statistics count Unicode code points in independently decodable UTF-8 tokens. Byte lengths include every ordinary token.
TokenizerMean length (characters)Mean length (bytes)Share with ≥8 charactersMaximum length (characters)
o200k5.556.9920.9%128
cl100k6.366.4229.6%128
Gemma 45.637.7825.2%31
Qwen3.85.227.4320.2%128
Command A+5.937.0126.7%512
DeepSeek V44.816.5318.3%128
Mistral Small 45.486.7521.0%76
GLM-5.15.366.6421.7%512
Grok 25.916.3226.8%256
LLM-jp-44.567.7618.2%16
PLaMo 34.917.9917.7%32
PLaMo 25.188.3721.3%27
Sarashina2.24.418.5511.5%19
Long tokens can also preserve recognizable strings. For instance, cl100k, Qwen3.8, and GLM-5.1 include the identifier .translatesAutoresizingMaskIntoConstraints. Qwen3.8 and GLM-5.1 contain Chinese website notices about disclaimers and reposting. DeepSeek V4, GLM-5.1, and LLM-jp-4 include Chinese political slogans. Both PLaMo versions have a token for ␣pic.twitter.com/.

Conclusion

GLM-5.1's pre-tokenizer splits the year 1985 as 198|5, so BPE never gets to use the vocabulary entry for the whole year. In 1234567, the same rule leaves 456 intact. BPE produces 45|6 because the vocabulary lacks 456. Qwen3.8 separates the period from the two line feeds in end.⏎⏎Next for the same reason, even though its pre-tokenizer groups them together.
The eight byte-based BPE tokenizers usually split the English sample in the same places, likely because they use similar rules for grouping words with spaces. In Japanese, tokenizers with a higher proportion of kana entries generally kept longer stretches of text together, sometimes an entire greeting with its full stop. The four models that accept images reserve one token position per image vector, and Gemma 4 does the same for audio. Image dimensions or audio duration, together with the processor settings, determine how much of the context window one input uses.

References

  1. [1] Taku Kudo, "Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates," ACL, 2018. See also the Hugging Face descriptions of BPE and Unigram.
  2. [2] Ken Whistler, "Unicode Normalization Forms (Unicode 18.0.0, Revision 58)," Unicode Standard Annex #15, 2026.