How 13 LLM Tokenizers Split Text
1234567 as 123|456|7, where each bar marks a token boundary. Command A+ also uses three tokens, grouping the digits as 1|234|567. Gemma 4 and Qwen3.8 use seven, one per digit.
- In the English and code samples, seven tokenizers almost always kept an ASCII symbol and the following newline together. Five always separated them. PLaMo 2 kept 31% of these pairs together.
- Across the vocabularies, the share of entries classified as kana (hiragana or katakana, alone or mixed with kanji) ranged from about 0.2% to 40% of ordinary tokens. Tokenizers with a larger share generally kept longer stretches of Japanese text together in the sample.
- Six vocabularies had entries named for images, audio, or video. Four of the model configurations accepted images. With their default settings, one 1920 × 1080 image occupied between 266 and 3,343 token positions.
- GLM-5.1 contained 99% of the distinct byte sequences in OpenAI's ordinary
cl100k_basevocabulary, and Qwen3.8 contained 97%. Even with that overlap, agreement on token boundaries depended on the language.
1. Tokenizers and method
o200k_harmony and cl100k_base (o200k and cl100k in the tables). Four focus on Japanese (LLM-jp-4, PLaMo 2, PLaMo 3, and Sarashina2.2). I use “PLaMo” for results that apply to both versions.
| Tokenizer | Source | Commit |
|---|---|---|
| o200k | openai/tiktoken (o200k_harmony) | 4e71bbe0c078 |
| cl100k | openai/tiktoken (cl100k_base) | 4e71bbe0c078 |
| Gemma 4 | google/gemma-4-E4B-it | 4ecc2ba63bca |
| Qwen3.8 | Qwen/Qwen3.8-27B | 1d4bf0f2ff60 |
| Command A+ | CohereLabs/command-a-plus-05-2026-bf16 | 5fb6fde5fd12 |
| DeepSeek V4 | deepseek-ai/DeepSeek-V4-Pro | b5968e9190ef |
| Mistral Small 4 | mistralai/Mistral-Small-4-119B-2603 | a11f36bebf70 |
| GLM-5.1 | zai-org/GLM-5.1 | 26e1bd6e011f |
| Grok 2 | xai-org/grok-2 | daf4395a80ad |
| LLM-jp-4 | llm-jp/llm-jp-4-8b-instruct | 098f2b2cf330 |
| PLaMo 3 | pfnet/plamo-3-610m-fin-instruct | a000b59209f1 |
| PLaMo 2 | pfnet/plamo-2-1b | 92c75fd6eea9 |
| Sarashina2.2 | sbintuitions/sarashina2.2-3b-instruct-v0.1 | 4f3626fb1b64 |
␣ for a space, most tokenizers have forms such as ␣the. PLaMo also has ␣of␣the, with an internal space, and a comma followed by a space (,␣).
| Tokenizer | Vocabulary size | Algorithm | Base unit | Usual space position | Standalone ASCII digits | Unicode normalization |
|---|---|---|---|---|---|---|
| o200k | 201,088 | BPE | Bytes | Leading | Up to 3, from the left | None |
| cl100k | 100,261 | BPE | Bytes | Leading | Up to 3, from the left | None |
| Gemma 4 | 262,144 | BPE | Characters | Leading | Single digits | None |
| Qwen3.8 | 248,077 | BPE | Bytes | Leading | Single digits | NFC |
| Command A+ | 255,032 | BPE | Bytes | Leading | Up to 3, from the right | None |
| DeepSeek V4 | 129,280 | BPE | Bytes | Leading | Up to 3, from the left | None |
| Mistral Small 4 | 131,072 | BPE | Bytes | Leading | Single digits | None |
| GLM-5.1 | 154,856 | BPE | Bytes | Leading | Up to 3 from the left, then smaller tokens as needed | None |
| Grok 2 | 131,072 | BPE | Bytes | Leading | Single digits | None |
| LLM-jp-4 | 196,608 | Unigram | Characters | Leading, with an extra initial space marker | Single digits | None |
| PLaMo 3 | 107,520 | Unigram | Characters | Leading, internal, or trailing | Single digits | None |
| PLaMo 2 | 100,000 | Unigram | Characters | Leading, internal, or trailing | Single digits | None |
| Sarashina2.2 | 102,400 | Unigram | Characters | Leading | Single digits | None |
random.Random(20261117). The measurement data also give the SHA-256 checksum for the ordered document IDs and texts, vocabulary statistics, and the settings and token counts for the image, audio, and video tests.
␣ marks an ASCII space and ⏎ a line feed (LF). Angle brackets such as <e3> enclose the hexadecimal bytes of a token that cannot be decoded as UTF-8 on its own. I encoded plain text without a chat template or automatic start or end tokens. LLM-jp-4 prepends a space marker of its own. It contributes an extra token when it stays separate from the first character.
2. Digits
\p{N}{1,3} to group digits from the left before BPE. For 1234567, that gives three pieces (123, 456, and 7). The OpenAI and DeepSeek vocabularies have a token for each piece. GLM-5.1 lacks 456, so its final split is 123|45|6|7.
| Tokenizer | 1234567 | 3.14159 | 2026-11-17 | Two- and three-digit ASCII entries | Digits per token (64-digit inputs) |
|---|---|---|---|---|---|
| o200k | 123|456|7 | 3|.|141|59 | 202|6|-|11|-|17 | 1,100 | 2.91 |
| cl100k | 123|456|7 | 3|.|141|59 | 202|6|-|11|-|17 | 1,100 | 2.91 |
| Gemma 4 | 1|2|3|4|5|6|7 | 3|.|1|4|1|5|9 | 2|0|2|6|-|1|1|-|1|7 | 0 | 1.00 |
| Qwen3.8 | 1|2|3|4|5|6|7 | 3|.|1|4|1|5|9 | 2|0|2|6|-|1|1|-|1|7 | 0 | 1.00 |
| Command A+ | 1|234|567 | 3|.|14|159 | 2|026|-|11|-|17 | 1,100 | 2.91 |
| DeepSeek V4 | 123|456|7 | 3|.|141|59 | 202|6|-|11|-|17 | 1,100 | 2.91 |
| Mistral Small 4 | 1|2|3|4|5|6|7 | 3|.|1|4|1|5|9 | 2|0|2|6|-|1|1|-|1|7 | 0 | 1.00 |
| GLM-5.1 | 123|45|6|7 | 3|.|141|59 | 202|6|-|11|-|17 | 277 | 1.63 |
| Grok 2 | 1|2|3|4|5|6|7 | 3|.|1|4|1|5|9 | 2|0|2|6|-|1|1|-|1|7 | 0 | 1.00 |
| LLM-jp-4 | ␣|1|2|3|4|5|6|7 | ␣|3|.|1|4|1|5|9 | ␣|2|0|2|6|-|1|1|-|1|7 | 0 | 0.98 |
| PLaMo 3 | 1|2|3|4|5|6|7 | 3|.1|4|1|5|9 | 2|0|2|6|-|1|1|-|1|7 | 0 | 1.00 |
| PLaMo 2 | 1|2|3|4|5|6|7 | 3|.1|4|1|5|9 | 2|0|2|6|-|1|1|-|1|7 | 0 | 1.00 |
| Sarashina2.2 | 1|2|3|4|5|6|7 | 3|.|1|4|1|5|9 | 2|0|2|6|-|1|1|-|1|7 | 0 | 1.00 |
\d{1,3}(?=(?:\d{3})*\b), which groups digits in threes from the right at a word boundary. The result, 1|234|567, lines up with the commas in 1,234,567. The same rule gives 14|159 after the decimal point in 3.14159 and 2|026 at the start of the date 2026-11-17. Before a letter or underscore, the right-grouping rule no longer matches. A second rule then groups from the left, giving 123|456|7|kg for 1234567kg.
\p{N} to isolate each digit before BPE. Gemma 4 and the four Japanese-focused tokenizers reach the same result on a standalone run of ASCII digits because their vocabularies have no entries made up of two or more ASCII digits. PLaMo can still combine one digit with a neighboring character. For instance, it splits 3.5kg as 3|.5|kg.
1985 as 198|5 and ␣2002 as ␣|200|2. Another 91 entries, including ␣41, combine a space with one to three digits. Its pre-tokenizer separates the space from the digits in these cases too. The ID ordering offers a clue to where these entries came from.
1234567, Command A+ and Grok 2 split each digit from 3 onward into byte fragments. The other eleven kept each character intact, with o200k and DeepSeek V4 also joining 12. Writing 123,456,789 in kanji (一億二千三百四十五万六千七百八十九) took 11 tokens with DeepSeek V4 and 27 with cl100k, the extremes in this comparison.
3. Whitespace and newlines
end.⏎⏎Next, for instance, cl100k makes the period and both line feeds one token, .⏎⏎. Qwen3.8 keeps the two line feeds together but puts the period in a separate token. In both regular expressions, [\r\n]* at the end of the punctuation pattern lets newlines stay in the same piece. The entry .⏎⏎ is available to cl100k and missing from Qwen3.8's vocabulary.
| Tokenizer | です。⏎⏎次 | end.⏎⏎Next | f(x)⏎y |
|---|---|---|---|
| o200k | です|。⏎⏎|次 | end|.⏎⏎|Next | f|(x|)⏎|y |
| cl100k | です|。⏎⏎|次 | end|.⏎⏎|Next | f|(x|)⏎|y |
| Gemma 4 | です|。|⏎⏎|次 | end|.|⏎⏎|Next | f|(|x|)|⏎|y |
| Qwen3.8 | です|。|⏎⏎|次 | end|.|⏎⏎|Next | f|(x|)|⏎|y |
| Command A+ | です|。⏎⏎|次 | end|.⏎⏎|Next | f|(x|)⏎|y |
| DeepSeek V4 | です|。⏎⏎|次 | end|.⏎⏎|Next | f|(x|)⏎|y |
| Mistral Small 4 | です|。⏎⏎|次 | end|.⏎⏎|Next | f|(x|)⏎|y |
| GLM-5.1 | です|。⏎⏎|次 | end|.⏎⏎|Next | f|(x|)⏎|y |
| Grok 2 | です|。⏎⏎|次 | end|.⏎⏎|Next | f|(x|)⏎|y |
| LLM-jp-4 | ␣|です|。|⏎|⏎|次 | ␣end|.|⏎|⏎|Next | ␣f|(|x|)|⏎|y |
| PLaMo 3 | です。|⏎⏎|次 | end|.|⏎⏎|Next | f|(|x|)|⏎|y |
| PLaMo 2 | です。⏎|⏎|次 | end|.⏎⏎|Next | f|(|x|)|⏎|y |
| Sarashina2.2 | です|。|⏎|⏎|次 | end|.|⏎|⏎|Next | f|(|x|)|⏎|y |
)⏎ as well as tokens consisting entirely of whitespace.
。⏎ with a Japanese full stop. It kept the symbol and newline together at 31% of those 2,238 positions.
| Tokenizer | Whitespace-only entries | Longest space run | Longest LF run | Longest tab run | Entries with LF | Entries with LF and non-whitespace |
|---|---|---|---|---|---|---|
| o200k | 434 | 128 | 16 | 20 | 2,225 | 2,034 |
| cl100k | 441 | 128 | 32 | 20 | 2,178 | 1,977 |
| Gemma 4 | 100 | 31 | 31 | 31 | 31 | 0 |
| Qwen3.8 | 440 | 128 | 32 | 20 | 200 | 0 |
| Command A+ | 507 | 512 | 64 | 32 | 2,544 | 2,280 |
| DeepSeek V4 | 157 | 128 | 16 | 11 | 849 | 799 |
| Mistral Small 4 | 129 | 75 | 5 | 16 | 1,069 | 1,064 |
| GLM-5.1 | 454 | 512 | 32 | 20 | 2,178 | 1,977 |
| Grok 2 | 375 | 256 | 32 | 17 | 1,811 | 1,637 |
| LLM-jp-4 | 68 | 16 | 1 | 1 | 1 | 0 |
| PLaMo 3 | 39 | 32 | 4 | 8 | 6 | 0 |
| PLaMo 2 | 42 | 16 | 2 | 4 | 61 | 39 |
| Sarashina2.2 | 26 | 8 | 1 | 1 | 1 | 0 |
\r\n (o200k, cl100k, Qwen3.8, Command A+, GLM-5.1, Grok 2, and PLaMo 3). The other six split the pair.
a and b, the whole string took four tokens in Sarashina2.2 and three in every other tokenizer. Twelve of the 13 vocabularies have a token for the full-width space (U+3000). Gemma 4 encodes it as three tokens, one per UTF-8 byte.
| Tokenizer | 4 spaces | 16 spaces | 64 spaces | 200 spaces | 1 LF | 2 LFs | 4 LFs | 8 LFs | \r\n\r\n |
|---|---|---|---|---|---|---|---|---|---|
| o200k | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 1 | 1 |
| cl100k | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 1 | 1 |
| Gemma 4 | 1 | 1 | 3 | 7 | 1 | 1 | 1 | 1 | 4 |
| Qwen3.8 | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 1 | 1 |
| Command A+ | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 1 | 1 |
| DeepSeek V4 | 1 | 1 | 1 | 3 | 1 | 1 | 1 | 1 | 4 |
| Mistral Small 4 | 1 | 1 | 1 | 3 | 1 | 1 | 1 | 2 | 4 |
| GLM-5.1 | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 1 | 1 |
| Grok 2 | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 1 | 1 |
| LLM-jp-4 | 1 | 1 | 4 | 13 | 1 | 2 | 4 | 8 | 4 |
| PLaMo 3 | 1 | 1 | 2 | 7 | 1 | 1 | 1 | 2 | 1 |
| PLaMo 2 | 1 | 1 | 4 | 13 | 1 | 1 | 2 | 4 | 4 |
| Sarashina2.2 | 1 | 2 | 8 | 25 | 1 | 2 | 4 | 8 | 4 |
if, all eight byte-based BPE tokenizers and LLM-jp-4 left the last space attached to the word, giving ␣␣␣|␣if. Gemma 4, PLaMo 3, and Sarashina2.2 produced ␣␣␣␣|if. PLaMo 2 can combine the preceding LF and all four spaces into one token. Its vocabulary has 20 entries made up of newlines followed by indentation. PLaMo 3 has none.
4. Code and structured formats
get_user_profile_by_id, the four underscores account for a four-token difference. The eight byte-based BPE tokenizers attach each underscore to the word that follows it, giving get|_user|_profile|_by|_id. Gemma 4, LLM-jp-4, and Sarashina2.2 give each underscore its own token and need nine. PLaMo also needs five, pairing underscores on either side of a word in get|_user_|profile|_by_|id.
| Tokenizer | Python function | get_user_profile_by_id | HTML fragment | JSON without whitespace | YAML | UUID | SHA-1 (40 hex digits) |
|---|---|---|---|---|---|---|---|
| o200k | 35 | 5 | 17 | 17 | 22 | 18 | 23 |
| cl100k | 35 | 5 | 17 | 16 | 22 | 18 | 23 |
| Gemma 4 | 44 | 9 | 19 | 16 | 28 | 36 | 34 |
| Qwen3.8 | 38 | 5 | 17 | 16 | 28 | 34 | 34 |
| Command A+ | 35 | 5 | 17 | 16 | 21 | 18 | 23 |
| DeepSeek V4 | 35 | 5 | 17 | 18 | 23 | 18 | 24 |
| Mistral Small 4 | 38 | 5 | 17 | 18 | 24 | 34 | 35 |
| GLM-5.1 | 35 | 5 | 17 | 16 | 22 | 23 | 27 |
| Grok 2 | 38 | 5 | 17 | 16 | 23 | 34 | 34 |
| LLM-jp-4 | 47 | 9 | 19 | 21 | 29 | 37 | 35 |
| PLaMo 3 | 44 | 5 | 9 | 29 | 29 | 36 | 35 |
| PLaMo 2 | 41 | 5 | 9 | 29 | 25 | 36 | 35 |
| Sarashina2.2 | 54 | 9 | 20 | 16 | 29 | 36 | 35 |
def fibonacci(n: int) -> int:
if n < 2:
return n
return fibonacci(n - 1) + fibonacci(n - 2)
<div class="container"><a href="/index.html">Home</a></div>
{"a":1,"b":[{"c":null},{"d":"e"}]}services:
web:
image: nginx:latest
ports:
- "80:80"
550e8400-e29b-41d4-a716-446655440000.da39a3ee5e6b4b0d3255bfef95601890afd80709.[^\r\n\p{L}\p{N}]?\p{L}+ admits a single leading symbol, so a piece can start with _user, (n, or .html. Even a boundary inside a word can depend on the character immediately before it. DeepSeek V4 splits tokenization as token|ization. Add a slash and the first token takes the initial t with it, giving /t|oken|ization.
| Tokenizer | Alone | After a space | After ( | After _ | After / | Initial capital |
|---|---|---|---|---|---|---|
| o200k | token|ization | ␣token|ization | (token|ization | _token|ization | /token|ization | Token|ization |
| cl100k | token|ization | ␣token|ization | (token|ization | _token|ization | /token|ization | Token|ization |
| Gemma 4 | token|ization | ␣token|ization | (|token|ization | _|token|ization | /|token|ization | Token|ization |
| Qwen3.8 | token|ization | ␣token|ization | (token|ization | _token|ization | /token|ization | Token|ization |
| Command A+ | token|ization | ␣tokenization | (token|ization | _token|ization | /token|ization | Token|ization |
| DeepSeek V4 | token|ization | ␣token|ization | (token|ization | _token|ization | /t|oken|ization | Token|ization |
| Mistral Small 4 | token|ization | ␣token|ization | (token|ization | _token|ization | /|token|ization | Token|ization |
| GLM-5.1 | token|ization | ␣token|ization | (token|ization | _token|ization | /token|ization | Token|ization |
| Grok 2 | token|ization | ␣token|ization | (token|ization | _token|ization | /token|ization | Token|ization |
| LLM-jp-4 | ␣token|ization | ␣|␣token|ization | ␣(|token|ization | ␣_|token|ization | ␣/|token|ization | ␣|Token|ization |
| PLaMo 3 | token|ization | ␣token|ization | (|token|ization | _|token|ization | /|token|ization | Token|ization |
| PLaMo 2 | token|ization | ␣token|ization | (|token|ization | _|token|ization | /|token|ization | Token|ization |
| Sarashina2.2 | token|ization | ␣token|ization | (|token|ization | _|token|ization | /|token|ization | Token|ization |
<div␣class=" and the complete closing tag </a>. In the JSON, PLaMo separated most quotation marks and brackets and used 29 tokens. The others used 16–21. In o200k, for example, ": and },{" are each a single token.
| Tokenizer | 64 hex digits | Base64 | 32 lowercase letters | 32 alphanumeric characters | 32 hiragana | 32 kanji |
|---|---|---|---|---|---|---|
| o200k | 1.74 | 1.46 | 1.91 | 1.46 | 2.72 | 1.57 |
| cl100k | 1.75 | 1.39 | 1.84 | 1.39 | 2.11 | 1.28 |
| Gemma 4 | 1.13 | 1.42 | 1.95 | 1.45 | 3.19 | 1.44 |
| Qwen3.8 | 1.13 | 1.36 | 1.89 | 1.35 | 3.05 | 1.73 |
| Command A+ | 1.78 | 1.54 | 1.96 | 1.54 | 2.93 | 1.64 |
| DeepSeek V4 | 1.72 | 1.42 | 1.84 | 1.42 | 2.62 | 1.72 |
| Mistral Small 4 | 1.12 | 1.31 | 1.77 | 1.30 | 2.86 | 1.49 |
| GLM-5.1 | 1.49 | 1.39 | 1.86 | 1.39 | 3.08 | 1.64 |
| Grok 2 | 1.14 | 1.44 | 1.91 | 1.44 | 2.67 | 1.60 |
| LLM-jp-4 | 1.11 | 1.30 | 1.82 | 1.31 | 3.41 | 1.68 |
| PLaMo 3 | 1.12 | 1.34 | 1.91 | 1.37 | 3.51 | 1.19 |
| PLaMo 2 | 1.12 | 1.33 | 1.88 | 1.36 | 3.49 | 1.16 |
| Sarashina2.2 | 1.12 | 1.35 | 1.95 | 1.38 | 3.29 | 1.09 |
5. Vocabulary by script
| Tokenizer | Latin | Han only | Kana, alone or with kanji | Hangul | Cyrillic | Arabic | Indic | Digits, symbols, whitespace | Other scripts | Invalid UTF-8 in isolation |
|---|---|---|---|---|---|---|---|---|---|---|
| o200k | 67.4 | 3.7 | 0.4 | 1.2 | 7.1 | 4.0 | 4.1 | 3.8 | 7.5 | 0.8 |
| cl100k | 89.6 | 0.8 | 0.2 | 0.3 | 0.7 | 0.1 | 0.0 | 7.4 | 0.1 | 0.8 |
| Gemma 4 | 60.6 | 7.3 | 1.9 | 1.8 | 5.1 | 3.2 | 11.1 | 3.5 | 5.4 | 0.0 |
| Qwen3.8 | 55.7 | 21.5 | 2.2 | 2.7 | 7.5 | 3.5 | 0.8 | 2.0 | 3.6 | 0.4 |
| Command A+ | 73.0 | 4.9 | 2.0 | 3.1 | 3.4 | 3.0 | 2.1 | 3.9 | 2.3 | 2.4 |
| DeepSeek V4 | 56.1 | 27.5 | 0.7 | 0.9 | 4.1 | 2.4 | 1.0 | 3.8 | 2.4 | 1.1 |
| Mistral Small 4 | 67.9 | 2.7 | 1.2 | 3.4 | 5.9 | 7.2 | 2.9 | 3.1 | 4.5 | 1.1 |
| GLM-5.1 | 65.2 | 18.4 | 0.6 | 0.4 | 6.4 | 1.4 | 0.1 | 5.9 | 0.9 | 0.7 |
| Grok 2 | 83.5 | 5.1 | 0.8 | 1.0 | 3.0 | 0.1 | 0.3 | 5.0 | 0.4 | 0.8 |
| LLM-jp-4 | 40.8 | 25.5 | 20.2 | 7.0 | 0.2 | 0.2 | 0.2 | 4.5 | 1.4 | 0.1 |
| PLaMo 3 | 50.5 | 16.1 | 29.5 | 0.9 | 0.6 | 0.4 | 0.1 | 1.1 | 0.6 | 0.1 |
| PLaMo 2 | 49.8 | 16.1 | 31.0 | 0.7 | 0.5 | 0.2 | 0.1 | 1.2 | 0.3 | 0.1 |
| Sarashina2.2 | 40.4 | 16.2 | 40.2 | 0.1 | 1.1 | 0.0 | 0.0 | 1.7 | 0.1 | 0.1 |
shift_jis, gb2312, and big5 codecs. A character is specific to the Japanese side if only shift_jis can encode it. Examples include 図, 働, and 込. A character is specific to the Chinese side if either Chinese codec can encode it and shift_jis cannot. Each of the two side-specific columns counts tokens with at least one such character and none specific to the opposite side. “Neither” includes shared characters, as in 東京 and 社会, as well as characters missing from all three encodings. These tests identify differences in repertoire even when a token's language is ambiguous.
| Tokenizer | Hiragana only | Katakana only | Mixed kana, or kana with kanji | Han only (neither) | Han only (Shift_JIS side) | Han only (GB2312/Big5 side) |
|---|---|---|---|---|---|---|
| o200k | 317 | 403 | 79 | 4,455 | 127 | 2,815 |
| cl100k | 87 | 69 | 1 | 546 | 7 | 286 |
| Gemma 4 | 1,544 | 2,028 | 1,353 | 12,051 | 764 | 6,445 |
| Qwen3.8 | 1,863 | 1,632 | 1,964 | 28,117 | 523 | 24,794 |
| Command A+ | 1,809 | 1,413 | 1,831 | 8,052 | 505 | 3,850 |
| DeepSeek V4 | 483 | 239 | 154 | 19,698 | 156 | 15,332 |
| Mistral Small 4 | 772 | 589 | 259 | 2,484 | 133 | 866 |
| GLM-5.1 | 470 | 346 | 117 | 15,186 | 116 | 13,172 |
| Grok 2 | 437 | 500 | 156 | 4,496 | 178 | 1,967 |
| LLM-jp-4 | 8,616 | 7,536 | 23,529 | 29,165 | 2,970 | 17,953 |
| PLaMo 3 | 10,820 | 7,792 | 13,129 | 13,908 | 2,644 | 748 |
| PLaMo 2 | 11,038 | 7,377 | 12,565 | 12,966 | 2,615 | 540 |
| Sarashina2.2 | 8,232 | 8,423 | 24,544 | 13,334 | 3,163 | 113 |
| Tokenizer | English | Japanese | Code | Chinese and Korean (T2) | T3 (12 categories) | T4 (36 web categories) | T5 (96 web categories) |
|---|---|---|---|---|---|---|---|
| o200k | 5.52 | 3.61 | 3.70 | 3.83 | 4.90 | 4.68 | 2.96 |
| cl100k | 5.47 | 2.62 | 3.50 | 2.40 | 3.29 | 2.15 | 2.22 |
| Gemma 4 | 5.40 | 4.90 | 3.41 | 4.42 | 5.03 | 4.50 | 3.23 |
| Qwen3.8 | 5.39 | 5.09 | 3.60 | 4.97 | 5.20 | 3.76 | 2.73 |
| Command A+ | 5.58 | 5.20 | 3.90 | 4.64 | 4.94 | 3.23 | 2.69 |
| DeepSeek V4 | 5.47 | 4.19 | 3.64 | 4.32 | 4.14 | 3.31 | 2.60 |
| Mistral Small 4 | 5.40 | 3.62 | 3.56 | 3.58 | 4.61 | 3.47 | 2.47 |
| GLM-5.1 | 5.47 | 3.95 | 3.72 | 3.68 | 4.05 | 2.53 | 2.46 |
| Grok 2 | 5.52 | 3.98 | 3.71 | 3.99 | 3.59 | 2.49 | 2.39 |
| LLM-jp-4 | 5.28 | 6.35 | 3.30 | 4.99 | 2.58 | 2.43 | 2.36 |
| PLaMo 3 | 6.29 | 6.48 | 3.41 | 3.15 | 3.24 | 1.96 | 2.13 |
| PLaMo 2 | 6.48 | 6.40 | 3.55 | 3.10 | 2.98 | 1.70 | 2.07 |
| Sarashina2.2 | 4.43 | 6.34 | 3.10 | 2.07 | 2.45 | 1.62 | 1.93 |
6. Japanese text
| Tokenizer | Kanji, kanji | Hiragana, hiragana | Katakana, katakana | Kanji to hiragana | Hiragana to kanji | Hiragana to Japanese punctuation |
|---|---|---|---|---|---|---|
| o200k | 72 | 56 | 52 | 98 | 99 | 100 |
| cl100k | 94 | 84 | 91 | 100 | 100 | 100 |
| Gemma 4 | 42 | 26 | 18 | 69 | 89 | 100 |
| Qwen3.8 | 38 | 24 | 20 | 67 | 87 | 100 |
| Command A+ | 43 | 24 | 25 | 66 | 87 | 100 |
| DeepSeek V4 | 51 | 37 | 63 | 90 | 96 | 100 |
| Mistral Small 4 | 80 | 43 | 57 | 91 | 98 | 100 |
| GLM-5.1 | 61 | 40 | 55 | 92 | 96 | 100 |
| Grok 2 | 66 | 40 | 52 | 91 | 97 | 100 |
| LLM-jp-4 | 21 | 19 | 5 | 46 | 69 | 100 |
| PLaMo 3 | 23 | 20 | 5 | 54 | 76 | 33 |
| PLaMo 2 | 23 | 21 | 5 | 57 | 78 | 32 |
| Sarashina2.2 | 20 | 19 | 6 | 45 | 70 | 100 |
| Tokenizer | Split | Tokens |
|---|---|---|
| o200k | ありがとうございます|。|よろしく|お願|いい|た|します|。 | 8 |
| cl100k | ありがとうござ|います|。|よ|ろ|<e38197e381>|<8f>|お|<e9a1>|<98>|い|い|た|します|。 | 15 |
| Gemma 4 | ありがとうございます|。|よろしく|お願いいたします|。 | 5 |
| Qwen3.8 | ありがとうございます|。|よろしくお願いいたします|。 | 4 |
| Command A+ | ありがとうございます|。|よろしくお願い|いたします|。 | 5 |
| DeepSeek V4 | あり|が|とう|ござ|います|。|よ|ろ|しく|お|願|い|いた|します|。 | 15 |
| Mistral Small 4 | ありがとうございます|。|よろしく|お願|いい|たし|ます|。 | 8 |
| GLM-5.1 | ありがとうござ|います|。|よ|ろ|しく|お|願い|いたします|。 | 10 |
| Grok 2 | あり|が|とう|ご|ざ|い|ます|。|よ|ろ|しく|お|願|い|いた|します|。 | 17 |
| LLM-jp-4 | ␣ありがとうございます|。|よろしくお願いいたします|。 | 4 |
| PLaMo 3 | ありがとうございます。|よろしくお願いいたします。 | 2 |
| PLaMo 2 | ありがとうございます。|よろしくお願いいたします。 | 2 |
| Sarashina2.2 | ありがとうござい|ます|。|よろしくお願い|いたします|。 | 6 |
ご覧|いただきありがとうございます.
| Tokenizer | Longest kana entries |
|---|---|
| o200k | ありがとうございました, ありがとうございます, ␣風吹けば名無し |
| cl100k | ありがとうござ, ありがとう, こんにちは |
| Gemma 4 | ありがとうございました, のではないでしょうか, ありがとうございます |
| Qwen3.8 | よろしくお願いいたします, ␣ありがとうございました, ありがとうございました |
| Command A+ | ありがとうございました, のではないでしょうか, ありがとうございます |
| DeepSeek V4 | かもしれません, ではありません, してください |
| Mistral Small 4 | ありがとうございました, ありがとうございます, ラクターデザイン |
| GLM-5.1 | かもしれません, ありがとうござ, してください |
| Grok 2 | ␣外部リンク, キャラクター, してください |
| LLM-jp-4 | ご覧いただきありがとうございます, ブログランキングに参加しています, このレビューは参考になりましたか |
| PLaMo 3 | ご覧いただきありがとうございます, いただきありがとうございました。, あけましておめでとうございます。 |
| PLaMo 2 | いただきありがとうございました。, あけましておめでとうございます。, 明けましておめでとうございます |
| Sarashina2.2 | だろうと思います, お願いいたします, にお越しください |
徳川|家康, one for the family name and one for the given name. Most of the other nine tokenizers used one token per character for that name.
7. English words and spaces
| Tokenizer | ASCII letter entries | Distinct forms ignoring case and leading space | Extra variants / ordinary vocabulary | Entries spanning spaces between ASCII letters | Words encoded with their leading space in one token |
|---|---|---|---|---|---|
| o200k | 105,794 | 67,816 | 19.0% | 0 | 98.3% |
| cl100k | 68,591 | 40,592 | 27.9% | 0 | 97.8% |
| Gemma 4 | 139,429 | 85,975 | 20.4% | 0 | 99.2% |
| Qwen3.8 | 104,060 | 67,557 | 14.7% | 0 | 97.9% |
| Command A+ | 143,694 | 90,359 | 20.9% | 0 | 99.3% |
| DeepSeek V4 | 63,821 | 41,698 | 17.3% | 0 | 98.0% |
| Mistral Small 4 | 70,680 | 49,573 | 16.2% | 0 | 96.6% |
| GLM-5.1 | 75,744 | 46,288 | 19.0% | 0 | 97.8% |
| Grok 2 | 89,127 | 56,387 | 25.0% | 0 | 98.9% |
| LLM-jp-4 | 78,329 | 51,026 | 13.9% | 0 | 98.5% |
| PLaMo 3 | 43,828 | 24,520 | 18.0% | 8,191 | 95.0% |
| PLaMo 2 | 37,621 | 21,081 | 16.5% | 10,050 | 95.7% |
| Sarashina2.2 | 40,321 | 23,956 | 16.0% | 0 | 73.4% |
a followed by a space and the word, then subtracted the token count for a alone.
the, ␣the, The, ␣The, THE, and ␣THE all reduce to the if the leading space is removed and the letters are lowercased. I applied the same rule to all entries consisting of two or more ASCII letters with an optional leading space. A group of six entries contributes five extra variants. Across all groups, the extra variants made up 27.9% of cl100k's ordinary vocabulary and 13.9% of LLM-jp-4's, the highest and lowest shares.
␣of␣the, ␣one␣of␣the, and ␣as␣well␣as. A tokenizer that splits at each word before applying its vocabulary cannot use such a phrase as one token. These phrases likely help explain why PLaMo covered 6.3–6.5 bytes per token in the English sample.
␣1. Where an ASCII symbol preceded a lowercase letter, the eight byte-based BPE tokenizers put a boundary between them 36–42% of the time. The rate for the other five was 86–100%.
| Tokenizer | Lowercase to space | Space to lowercase | Space to uppercase | Symbol to lowercase | Symbol to space | Space to digit | Digit, digit | Symbol to newline |
|---|---|---|---|---|---|---|---|---|
| o200k | 100 | 0 | 0 | 36 | 100 | 100 | 10 | 0 |
| cl100k | 100 | 0 | 0 | 36 | 100 | 100 | 10 | 0 |
| Gemma 4 | 100 | 3 | 2 | 100 | 100 | 100 | 100 | 100 |
| Qwen3.8 | 100 | 0 | 0 | 36 | 100 | 100 | 100 | 100 |
| Command A+ | 100 | 0 | 0 | 36 | 100 | 100 | 10 | 0 |
| DeepSeek V4 | 100 | 0 | 0 | 42 | 100 | 100 | 10 | 0 |
| Mistral Small 4 | 100 | 0 | 0 | 39 | 100 | 100 | 100 | 0 |
| GLM-5.1 | 100 | 0 | 0 | 36 | 100 | 100 | 16 | 0 |
| Grok 2 | 100 | 0 | 0 | 38 | 100 | 100 | 100 | 0 |
| LLM-jp-4 | 100 | 1 | 5 | 100 | 100 | 100 | 100 | 100 |
| PLaMo 3 | 85 | 3 | 17 | 86 | 65 | 0 | 100 | 100 |
| PLaMo 2 | 84 | 4 | 15 | 86 | 65 | 0 | 100 | 69 |
| Sarashina2.2 | 100 | 11 | 26 | 100 | 100 | 100 | 100 | 100 |
don't, the apostrophe can stand alone, join the t, or stay inside the whole word.
- o200k and Command A+ produced
don'tas one token. - cl100k, Qwen3.8, DeepSeek V4, Mistral Small 4, GLM-5.1, and Grok 2 produced
don|'t. - Gemma 4, PLaMo, and Sarashina2.2 produced
don|'|t. LLM-jp-4 also used three tokens, with its initial space marker attached to the first,␣don|'|t.
The quick brown fox jumps over the lazy dog. took 9–10 tokens as written and 12–18 in all capitals.
8. Other languages, emoji, and Unicode normalization
| Tokenizer | Chinese (Simplified) | Korean | Russian | Arabic | Hindi | Thai | German |
|---|---|---|---|---|---|---|---|
| o200k | 13 | 17 | 14 | 15 | 15 | 18 | 20 |
| cl100k | 22 | 30 | 26 | 30 | 49 | 41 | 26 |
| Gemma 4 | 12 | 15 | 14 | 16 | 14 | 15 | 20 |
| Qwen3.8 | 10 | 15 | 14 | 14 | 23 | 12 | 23 |
| Command A+ | 11 | 13 | 16 | 14 | 17 | 15 | 19 |
| DeepSeek V4 | 10 | 20 | 17 | 17 | 27 | 20 | 25 |
| Mistral Small 4 | 16 | 17 | 15 | 13 | 20 | 24 | 24 |
| GLM-5.1 | 10 | 23 | 15 | 19 | 42 | 39 | 25 |
| Grok 2 | 13 | 18 | 20 | 30 | 37 | 42 | 22 |
| LLM-jp-4 | 10 | 12 | 34 | 37 | 48 | 45 | 27 |
| PLaMo 3 | 14 | 29 | 31 | 27 | 45 | 29 | 29 |
| PLaMo 2 | 15 | 29 | 32 | 30 | 63 | 38 | 29 |
| Sarashina2.2 | 25 | 46 | 23 | 40 | 71 | 72 | 28 |
- Chinese (Simplified)
- 今天天气很好,我们一起去公园散步吧。
- Korean
- 오늘은 날씨가 좋아서 공원에 산책하러 갑니다.
- Russian
- Сегодня хорошая погода, пойдём гулять в парк.
- Arabic
- الطقس جميل اليوم، لنذهب للتنزه في الحديقة.
- Hindi
- आज मौसम अच्छा है, चलो पार्क में टहलने चलते हैं।
- Thai
- วันนี้อากาศดี ไปเดินเล่นที่สวนสาธารณะกันเถอะ
- German
- Die Donaudampfschifffahrtsgesellschaft überprüft die Geschwindigkeitsbegrenzung.
| Tokenizer | Pictographic entries | 😂 | 👩🔬 | 🇯🇵🇺🇸🇫🇷 | One full-width space | ハンカクカタカナガギグ | café, precomposed / decomposed |
|---|---|---|---|---|---|---|---|
| o200k | 74 | 1 | 5 | 12 | 1 | 25 | 2 / 3 |
| cl100k | 21 | 2 | 8 | 18 | 1 | 28 | 2 / 3 |
| Gemma 4 | 1,219 | 1 | 3 | 6 | 3 | 14 | 2 / 2 |
| Qwen3.8 | 36 | 2 | 8 | 18 | 1 | 15 | 2 / 2 |
| Command A+ | 116 | 1 | 4 | 9 | 1 | 28 | 2 / 2 |
| DeepSeek V4 | 51 | 1 | 5 | 12 | 1 | 28 | 3 / 3 |
| Mistral Small 4 | 10 | 4 | 9 | 24 | 1 | 35 | 3 / 4 |
| GLM-5.1 | 823 | 1 | 3 | 18 | 1 | 28 | 2 / 3 |
| Grok 2 | 30 | 1 | 5 | 12 | 1 | 28 | 2 / 3 |
| LLM-jp-4 | 1,056 | 2 | 7 | 7 | 2 | 15 | 1 / 2 |
| PLaMo 3 | 222 | 1 | 8 | 9 | 1 | 14 | 2 / 2 |
| PLaMo 2 | 220 | 1 | 8 | 9 | 1 | 14 | 2 / 2 |
| Sarashina2.2 | 77 | 1 | 9 | 24 | 1 | 14 | 3 / 2 |
ハンカクカタカナガギグ contains 14 code points, counting each voicing mark separately. Gemma 4, Qwen3.8, and the four Japanese-focused tokenizers used 14–15 tokens. The other seven split some characters into byte fragments and used 25–35.
café with a single é and with e followed by a combining acute accent (U+0301). Qwen3.8 alone applies Unicode normalization. It uses Normalization Form C (NFC), which gives canonically equivalent strings the same representation[2]. The two spellings produce identical token IDs in Qwen3.8 and different ID sequences in all 12 other tokenizers, even when the token counts match.
- Qwen3.8 returned the NFC forms of decomposed accents, Japanese voicing marks, and Hangul jamo (the letters that make up a syllable).
- Gemma 4, LLM-jp-4, and Sarashina2.2 use
▁(U+2581) internally for a space. A literala▁bin the input came back asa b. - PLaMo 3 uses U+EE00 as an internal boundary marker. When the input contained that character between
aandb, decoding returnedab.
9. Repeated characters
| Tokenizer | a × 64 | = × 64 | . × 64 | あ × 32 | ー × 16 | 。 × 16 | 笑 × 16 | Longest non-whitespace run (characters) |
|---|---|---|---|---|---|---|---|---|
| o200k | 8 | 1 | 1 | 32 | 4 | 8 | 16 | 112 |
| cl100k | 8 | 1 | 1 | 32 | 16 | 8 | 16 | 96 |
| Gemma 4 | 8 | 4 | 4 | 16 | 4 | 4 | 16 | 16 |
| Qwen3.8 | 8 | 1 | 1 | 16 | 2 | 8 | 8 | 96 |
| Command A+ | 4 | 1 | 1 | 16 | 4 | 8 | 16 | 128 |
| DeepSeek V4 | 8 | 1 | 1 | 32 | 16 | 4 | 8 | 128 |
| Mistral Small 4 | 32 | 2 | 4 | 16 | 16 | 8 | 16 | 76 |
| GLM-5.1 | 8 | 1 | 1 | 32 | 16 | 4 | 8 | 96 |
| Grok 2 | 4 | 1 | 1 | 16 | 16 | 16 | 16 | 128 |
| LLM-jp-4 | 17 | 5 | 5 | 9 | 2 | 4 | 9 | 16 |
| PLaMo 3 | 10 | 13 | 22 | 7 | 3 | 6 | 8 | 10 |
| PLaMo 2 | 16 | 13 | 22 | 7 | 3 | 6 | 8 | 10 |
| Sarashina2.2 | 14 | 8 | 8 | 4 | 2 | 4 | 8 | 8 |
10. Special tokens
<|im_end|>, Qwen3.8 returns the same ID that its chat template uses to end a turn. Whether that ID actually ends the turn depends on the model and the software that receives it.
| Tokenizer | Registered special tokens | Apparent reserved slots | Literal end marker | Default encoding result |
|---|---|---|---|---|
| o200k | 1,090 | 1,081 | <|endoftext|> | Error (tiktoken default) |
| cl100k | 5 | 0 | <|endoftext|> | Error (tiktoken default) |
| Gemma 4 | 24 | 6,227 | <eos> | One special token |
| Qwen3.8 | 21 | 0 | <|im_end|> | One special token |
| Command A+ | 15 | 0 | <|END_OF_TURN_TOKEN|> | One special token |
| DeepSeek V4 | 1,230 | 1,217 | <|end▁of▁sentence|> | One special token |
| Mistral Small 4 | 1,000 | 972 | </s> | One special token |
| GLM-5.1 | 18 | 0 | <|endoftext|> | One special token |
| Grok 2 | 128 | 125 | <|eos|> | Error (tiktoken adapter used here) |
| LLM-jp-4 | 352 | 338 | <|return|> | One special token |
| PLaMo 3 | 59 | 27 | <|plamo:eos|> | One special token |
| PLaMo 2 | 29 | 23 | <|plamo:eos|> | One special token |
| Sarashina2.2 | 7 | 0 | </s> | One special token |
encode, the two OpenAI encodings and the Grok 2 adapter raised an error for their respective markers. In the ordinary-text comparisons I used tiktoken's encode_ordinary, which treats those spellings as text.
<unused…> in its ordinary vocabulary.
<think> and <tool_call>. Each PLaMo version has 76 ordinary entries spelled like HTML tags, including </div>.
11. Image, audio, and video tokens
| Tokenizer | Inputs besides text | Image and video | Audio | Image regions and coordinates |
|---|---|---|---|---|
| o200k | — | None | None | None |
| cl100k | — | None | None | None |
| Gemma 4 | Images, audio, video | <|image>, <|image|>, <image|>, <|video|> | <|audio>, <|audio|>, <audio|> | None |
| Qwen3.8 | Images, video | <|vision_start|>, <|vision_end|>, <|vision_pad|>, <|image_pad|>, <|video_pad|> | <|audio_start|>, <|audio_end|>, <|audio_pad|>, <tts_pad>, <tts_text_bos>, <tts_text_eod>, <tts_text_bos_single> | <|object_ref_start|>, <|object_ref_end|>, <|box_start|>, <|box_end|>, <|quad_start|>, <|quad_end|> |
| Command A+ | Images | <|START_OF_IMG|>, <|END_OF_IMG|>, <|IMG_LINE_BREAK|>, <|IMG_PATCH|> | None | None |
| DeepSeek V4 | None | <|image|>, <|image2|>, <|rl_image_start|>, <|rl_image_pad|> | None | <|ref|>, <|/ref|>, <|box|>, <|/box|>, <|point|>, <|/point|>, <|polygon|>, <|/polygon|> |
| Mistral Small 4 | Images | [IMG], [IMG_BREAK], [IMG_END] | [AUDIO], [BEGIN_AUDIO] | None |
| GLM-5.1 | None | <|begin_of_image|>, <|end_of_image|>, <|begin_of_video|>, <|end_of_video|>, <|image|>, <|video|> | <|begin_of_audio|>, <|end_of_audio|>, <|begin_of_transcription|>, <|end_of_transcription|> | <|begin_of_box|>, <|end_of_box|> |
| Grok 2 | None | None | None | None |
| LLM-jp-4 | None | None | None | None |
| PLaMo 3 | None | None | None | None |
| PLaMo 2 | None | None | None | None |
| Sarashina2.2 | None | None | None | None |
- [3] Chameleon Team, "Chameleon: Mixed-Modal Early-Fusion Foundation Models," arXiv:2405.09818, 2024.
| Image processor | 224 × 224 | 512 × 512 | 1024 × 768 | 1920 × 1080 | 4032 × 3024 |
|---|---|---|---|---|---|
| Gemma 4 | 258 | 258 | 268 | 266 | 268 |
| Qwen3.8 | 66 | 258 | 770 | 2,042 | 11,846 |
| Command A+ | 259 | 259 | 1,287 | 3,343 | 3,343 |
| Mistral Small 4 | 72 | 380 | 1,064 | 1,736 | 2,352 |
AutoProcessor in Transformers 5.23.0, using PyTorch 2.15.0, torchvision 0.29.2, and Pillow 13.0.0. This includes the Transformers processor distributed with Mistral Small 4. I encoded a conversation containing one all-black image and counted the image placeholders and delimiters. I also checked where each model's implementation replaces the placeholder embeddings with image vectors.
| Image processor | Placeholders for image vectors | Start, end, and separator tokens |
|---|---|---|
| Gemma 4 | 264 × <|image|> | 2 (start and end) |
| Qwen3.8 | 2,040 × <|image_pad|> | 2 (start and end) |
| Command A+ | 3,328 × <|IMG_PATCH|> | 2 (start and end), 13 × <|IMG_LINE_BREAK|> |
| Mistral Small 4 | 1,705 × [IMG] | 30 × [IMG_BREAK], 1 × [IMG_END] |
- Gemma 4 resizes the image to fit a vector budget. The default limit is 280 vectors. With the same 1024 × 768 image, the five available limits (70 through 1,120) gave between 63 and 1,064 vectors.
- Qwen3.8 uses one vector per 32 × 32 pixels after resizing. Its configured minimum and maximum pixel counts correspond to 64 and 16,384 vectors.
- Command A+ uses 256 vectors per 512 × 512 tile. It divides an image into up to 12 tiles and adds a downscaled overview when it uses more than one. Both of the largest test images used 12 tiles plus the overview, or 3,328 vectors.
- Mistral Small 4 scales the longer side down if it exceeds 1,540 pixels, adjusts both dimensions to multiples of 28, and uses one vector per 28 × 28 pixels.
[IMG_BREAK] between rows of vectors and closes the last row with [IMG_END]. Command A+ puts <|IMG_LINE_BREAK|> after each tile. Gemma 4 and Qwen3.8 mark the start and end of the whole image and pass coordinates or grid dimensions to the model in separate arrays.
<|audio|> placeholders, ten seconds produced 250, and both 30 and 45 seconds produced 750. For the ten-second video, Gemma 4 inserted 66 <|video|> placeholders for each of 32 frames. Qwen3.8 inserted 220 <|video_pad|> placeholders for each of ten frame pairs. These counts exclude delimiters and timestamps.
[AUDIO] and [BEGIN_AUDIO] likewise go unused by these processors. One possible reason to keep these entries is to share a vocabulary with other models in the same family. In o200k, cl100k, Grok 2, and the four Japanese-focused vocabularies, I found no token names identifying images, audio, or video. The numbered reserved slots in o200k and Grok 2 give no indication of their purpose.
<|place_holder_mm_span_0021|> through <|place_holder_mm_span_0435|>. I counted these as apparent reserved slots in the special-token comparison. Six other added tokens delimit tables, rows, and cells. The opening tags are <|table|>, <|tr|>, and <|td|>.
12. Vocabulary overlap and boundary agreement
| Tokenizer | Share of cl100k present | Share of o200k present |
|---|---|---|
| o200k | 85% | 100% |
| cl100k | 100% | 43% |
| Gemma 4 | 67% | 66% |
| Qwen3.8 | 97% | 68% |
| Command A+ | 88% | 74% |
| DeepSeek V4 | 59% | 41% |
| Mistral Small 4 | 56% | 50% |
| GLM-5.1 | 99% | 54% |
| Grok 2 | 77% | 50% |
| LLM-jp-4 | 42% | 27% |
| PLaMo 3 | 36% | 23% |
| PLaMo 2 | 35% | 20% |
| Sarashina2.2 | 29% | 17% |
| Tokenizer | 0% | 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% |
|---|---|---|---|---|---|---|---|---|---|---|
| o200k | 72/4 | 69/4 | 69/4 | 68/4 | 68/4 | 68/4 | 67/4 | 66/4 | 66/5 | 66/4 |
| cl100k | 88/0 | 88/0 | 87/1 | 91/1 | 92/1 | 92/1 | 91/1 | 91/2 | 92/1 | 91/1 |
| Gemma 4 | 75/2 | 68/4 | 65/6 | 64/6 | 62/6 | 62/7 | 62/7 | 61/8 | 61/8 | 28/38 |
| Qwen3.8 | 94/0 | 94/0 | 94/0 | 82/13 | 0/100 | 0/98 | 35/6 | 54/5 | 55/5 | 52/10 |
| Command A+ | 78/5 | 78/6 | 77/7 | 75/7 | 75/7 | 74/7 | 74/7 | 73/8 | 73/8 | 72/8 |
| DeepSeek V4 | 61/25 | 60/25 | 59/25 | 59/27 | 58/28 | 56/29 | 54/31 | 54/31 | 54/32 | 52/33 |
| Mistral Small 4 | 72/3 | 71/4 | 70/4 | 69/4 | 69/4 | 68/4 | 68/4 | 68/4 | 67/4 | 66/4 |
| GLM-5.1 | 91/0 | 92/0 | 92/0 | 92/0 | 92/0 | 93/0 | 32/63 | 0/97 | 28/26 | 44/4 |
| Grok 2 | 86/3 | 85/5 | 85/5 | 84/6 | 84/6 | 84/6 | 84/6 | 83/6 | 83/7 | 83/7 |
| LLM-jp-4 | 58/34 | 51/42 | 47/45 | 44/47 | 41/50 | 39/52 | 40/50 | 39/51 | 46/45 | 4/43 |
| PLaMo 3 | 92/0 | 100/0 | 100/0 | 100/0 | 65/33 | 3/97 | 4/96 | 3/97 | 4/96 | 36/38 |
| PLaMo 2 | 92/0 | 100/0 | 100/0 | 100/0 | 69/28 | 2/98 | 3/97 | 3/97 | 4/96 | 27/56 |
| Sarashina2.2 | 55/39 | 49/48 | 46/51 | 43/54 | 41/56 | 36/61 | 35/63 | 33/65 | 32/65 | 33/64 |
| Tokenizer | o200k | cl100k | Gemma 4 | Qwen3.8 | Command A+ | DeepSeek V4 | Mistral Small 4 | GLM-5.1 | Grok 2 | LLM-jp-4 | PLaMo 3 | PLaMo 2 | Sarashina2.2 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| o200k | — | 40 | 40 | 43 | 48 | 33 | 44 | 44 | 43 | 16 | 17 | 16 | 13 |
| cl100k | 40 | — | 23 | 39 | 33 | 35 | 33 | 64 | 50 | 17 | 21 | 21 | 16 |
| Gemma 4 | 40 | 23 | — | 38 | 44 | 31 | 33 | 31 | 34 | 24 | 18 | 16 | 14 |
| Qwen3.8 | 43 | 39 | 38 | — | 45 | 41 | 35 | 55 | 35 | 24 | 18 | 16 | 13 |
| Command A+ | 48 | 33 | 44 | 45 | — | 33 | 37 | 39 | 43 | 22 | 17 | 16 | 14 |
| DeepSeek V4 | 33 | 35 | 31 | 41 | 33 | — | 40 | 49 | 44 | 28 | 23 | 22 | 16 |
| Mistral Small 4 | 44 | 33 | 33 | 35 | 37 | 40 | — | 35 | 44 | 17 | 20 | 18 | 14 |
| GLM-5.1 | 44 | 64 | 31 | 55 | 39 | 49 | 35 | — | 47 | 24 | 21 | 19 | 15 |
| Grok 2 | 43 | 50 | 34 | 35 | 43 | 44 | 44 | 47 | — | 23 | 23 | 22 | 18 |
| LLM-jp-4 | 16 | 17 | 24 | 24 | 22 | 28 | 17 | 24 | 23 | — | 32 | 31 | 33 |
| PLaMo 3 | 17 | 21 | 18 | 18 | 17 | 23 | 20 | 21 | 23 | 32 | — | 84 | 44 |
| PLaMo 2 | 16 | 21 | 16 | 16 | 16 | 22 | 18 | 19 | 22 | 31 | 84 | — | 44 |
| Sarashina2.2 | 13 | 16 | 14 | 13 | 14 | 16 | 14 | 15 | 18 | 33 | 44 | 44 | — |
| Tokenizer | o200k | cl100k | Gemma 4 | Qwen3.8 | Command A+ | DeepSeek V4 | Mistral Small 4 | GLM-5.1 | Grok 2 | LLM-jp-4 | PLaMo 3 | PLaMo 2 | Sarashina2.2 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| o200k | — | 72 | 70 | 68 | 68 | 82 | 83 | 86 | 85 | 52 | 52 | 54 | 52 |
| cl100k | 72 | — | 52 | 50 | 50 | 62 | 71 | 66 | 65 | 39 | 39 | 40 | 39 |
| Gemma 4 | 70 | 52 | — | 90 | 88 | 80 | 70 | 76 | 76 | 67 | 64 | 63 | 67 |
| Qwen3.8 | 68 | 50 | 90 | — | 88 | 78 | 68 | 75 | 73 | 69 | 66 | 64 | 68 |
| Command A+ | 68 | 50 | 88 | 88 | — | 77 | 68 | 74 | 75 | 64 | 62 | 62 | 64 |
| DeepSeek V4 | 82 | 62 | 80 | 78 | 77 | — | 80 | 90 | 88 | 59 | 58 | 59 | 59 |
| Mistral Small 4 | 83 | 71 | 70 | 68 | 68 | 80 | — | 82 | 84 | 52 | 52 | 53 | 52 |
| GLM-5.1 | 86 | 66 | 76 | 75 | 74 | 90 | 82 | — | 88 | 56 | 55 | 56 | 56 |
| Grok 2 | 85 | 65 | 76 | 73 | 75 | 88 | 84 | 88 | — | 55 | 55 | 55 | 55 |
| LLM-jp-4 | 52 | 39 | 67 | 69 | 64 | 59 | 52 | 56 | 55 | — | 75 | 74 | 90 |
| PLaMo 3 | 52 | 39 | 64 | 66 | 62 | 58 | 52 | 55 | 55 | 75 | — | 96 | 77 |
| PLaMo 2 | 54 | 40 | 63 | 64 | 62 | 59 | 53 | 56 | 55 | 74 | 96 | — | 76 |
| Sarashina2.2 | 52 | 39 | 67 | 68 | 64 | 59 | 52 | 56 | 55 | 90 | 77 | 76 | — |
get_user_profile_by_id was 66–84%.
| Tokenizer | o200k | cl100k | Gemma 4 | Qwen3.8 | Command A+ | DeepSeek V4 | Mistral Small 4 | GLM-5.1 | Grok 2 | LLM-jp-4 | PLaMo 3 | PLaMo 2 | Sarashina2.2 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| o200k | — | 98 | 96 | 97 | 99 | 98 | 97 | 98 | 98 | 93 | 77 | 78 | 79 |
| cl100k | 98 | — | 96 | 98 | 98 | 98 | 98 | 100 | 99 | 93 | 77 | 77 | 79 |
| Gemma 4 | 96 | 96 | — | 97 | 96 | 96 | 95 | 96 | 97 | 97 | 78 | 76 | 81 |
| Qwen3.8 | 97 | 98 | 97 | — | 96 | 97 | 96 | 98 | 97 | 95 | 78 | 76 | 80 |
| Command A+ | 99 | 98 | 96 | 96 | — | 98 | 97 | 98 | 99 | 94 | 77 | 78 | 79 |
| DeepSeek V4 | 98 | 98 | 96 | 97 | 98 | — | 98 | 98 | 99 | 93 | 77 | 77 | 79 |
| Mistral Small 4 | 97 | 98 | 95 | 96 | 97 | 98 | — | 98 | 98 | 93 | 76 | 77 | 79 |
| GLM-5.1 | 98 | 100 | 96 | 98 | 98 | 98 | 98 | — | 99 | 93 | 77 | 77 | 79 |
| Grok 2 | 98 | 99 | 97 | 97 | 99 | 99 | 98 | 99 | — | 94 | 77 | 78 | 79 |
| LLM-jp-4 | 93 | 93 | 97 | 95 | 94 | 93 | 93 | 93 | 94 | — | 77 | 76 | 83 |
| PLaMo 3 | 77 | 77 | 78 | 78 | 77 | 77 | 76 | 77 | 77 | 77 | — | 95 | 67 |
| PLaMo 2 | 78 | 77 | 76 | 76 | 78 | 77 | 77 | 77 | 78 | 76 | 95 | — | 66 |
| Sarashina2.2 | 79 | 79 | 81 | 80 | 79 | 79 | 79 | 79 | 79 | 83 | 67 | 66 | — |
| Tokenizer | o200k | cl100k | Gemma 4 | Qwen3.8 | Command A+ | DeepSeek V4 | Mistral Small 4 | GLM-5.1 | Grok 2 | LLM-jp-4 | PLaMo 3 | PLaMo 2 | Sarashina2.2 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| o200k | — | 93 | 76 | 88 | 93 | 93 | 93 | 97 | 95 | 75 | 64 | 60 | 69 |
| cl100k | 93 | — | 73 | 85 | 88 | 88 | 91 | 94 | 90 | 72 | 61 | 58 | 66 |
| Gemma 4 | 76 | 73 | — | 84 | 78 | 76 | 75 | 76 | 78 | 87 | 82 | 77 | 85 |
| Qwen3.8 | 88 | 85 | 84 | — | 90 | 87 | 86 | 90 | 89 | 83 | 71 | 66 | 76 |
| Command A+ | 93 | 88 | 78 | 90 | — | 92 | 90 | 93 | 94 | 77 | 65 | 61 | 70 |
| DeepSeek V4 | 93 | 88 | 76 | 87 | 92 | — | 93 | 93 | 94 | 76 | 65 | 62 | 70 |
| Mistral Small 4 | 93 | 91 | 75 | 86 | 90 | 93 | — | 93 | 94 | 75 | 64 | 61 | 69 |
| GLM-5.1 | 97 | 94 | 76 | 90 | 93 | 93 | 93 | — | 95 | 75 | 64 | 60 | 69 |
| Grok 2 | 95 | 90 | 78 | 89 | 94 | 94 | 94 | 95 | — | 77 | 65 | 62 | 70 |
| LLM-jp-4 | 75 | 72 | 87 | 83 | 77 | 76 | 75 | 75 | 77 | — | 79 | 74 | 86 |
| PLaMo 3 | 64 | 61 | 82 | 71 | 65 | 65 | 64 | 64 | 65 | 79 | — | 93 | 79 |
| PLaMo 2 | 60 | 58 | 77 | 66 | 61 | 62 | 61 | 60 | 62 | 74 | 93 | — | 74 |
| Sarashina2.2 | 69 | 66 | 85 | 76 | 70 | 70 | 69 | 69 | 70 | 86 | 79 | 74 | — |
13. Vocabulary usage and token length
| Tokenizer | Total token occurrences | Share of ordinary vocabulary used | Distinct tokens covering 50% of occurrences | Distinct tokens covering 90% of occurrences | Occurrences ending inside a character |
|---|---|---|---|---|---|
| o200k | 2,189,117 | 51.2% | 959 | 28,672 | 10.0% |
| cl100k | 3,244,038 | 36.1% | 209 | 3,546 | 19.5% |
| Gemma 4 | 2,053,725 | 40.3% | 1,289 | 27,554 | 0.2% |
| Qwen3.8 | 2,378,241 | 35.7% | 695 | 16,629 | 8.0% |
| Command A+ | 2,485,567 | 37.1% | 458 | 16,356 | 13.3% |
| DeepSeek V4 | 2,567,849 | 45.7% | 534 | 9,796 | 9.4% |
| Mistral Small 4 | 2,634,008 | 54.7% | 368 | 13,387 | 15.5% |
| GLM-5.1 | 2,836,136 | 37.4% | 332 | 7,654 | 14.6% |
| Grok 2 | 2,919,106 | 41.5% | 293 | 6,687 | 15.5% |
| LLM-jp-4 | 3,013,270 | 24.5% | 286 | 4,176 | 2.3% |
| PLaMo 3 | 3,344,648 | 41.8% | 147 | 4,425 | 19.2% |
| PLaMo 2 | 3,566,524 | 38.3% | 116 | 3,149 | 22.1% |
| Sarashina2.2 | 3,856,086 | 33.2% | 80 | 2,357 | 23.3% |
| Tokenizer | Mean length (characters) | Mean length (bytes) | Share with ≥8 characters | Maximum length (characters) |
|---|---|---|---|---|
| o200k | 5.55 | 6.99 | 20.9% | 128 |
| cl100k | 6.36 | 6.42 | 29.6% | 128 |
| Gemma 4 | 5.63 | 7.78 | 25.2% | 31 |
| Qwen3.8 | 5.22 | 7.43 | 20.2% | 128 |
| Command A+ | 5.93 | 7.01 | 26.7% | 512 |
| DeepSeek V4 | 4.81 | 6.53 | 18.3% | 128 |
| Mistral Small 4 | 5.48 | 6.75 | 21.0% | 76 |
| GLM-5.1 | 5.36 | 6.64 | 21.7% | 512 |
| Grok 2 | 5.91 | 6.32 | 26.8% | 256 |
| LLM-jp-4 | 4.56 | 7.76 | 18.2% | 16 |
| PLaMo 3 | 4.91 | 7.99 | 17.7% | 32 |
| PLaMo 2 | 5.18 | 8.37 | 21.3% | 27 |
| Sarashina2.2 | 4.41 | 8.55 | 11.5% | 19 |
.translatesAutoresizingMaskIntoConstraints. Qwen3.8 and GLM-5.1 contain Chinese website notices about disclaimers and reposting. DeepSeek V4, GLM-5.1, and LLM-jp-4 include Chinese political slogans. Both PLaMo versions have a token for ␣pic.twitter.com/.
Conclusion
1985 as 198|5, so BPE never gets to use the vocabulary entry for the whole year. In 1234567, the same rule leaves 456 intact. BPE produces 45|6 because the vocabulary lacks 456. Qwen3.8 separates the period from the two line feeds in end.⏎⏎Next for the same reason, even though its pre-tokenizer groups them together.
References
- [1] Taku Kudo, "Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates," ACL, 2018. See also the Hugging Face descriptions of BPE and Unigram.
- [2] Ken Whistler, "Unicode Normalization Forms (Unicode 18.0.0, Revision 58)," Unicode Standard Annex #15, 2026.