Tokenizer playground

Compare how tokenizers split the same text and how many tokens they produce. Enter text below, then choose a tokenizer to inspect its output. Tokenization runs in your browser without sending your text to a server; only the selected tokenizer's vocabulary is downloaded.

Tokenizer

Input text

Reading the results

A large language model (LLM) processes text as a sequence of tokens. A token can represent a whole word, part of a word, whitespace, or part of a character's byte sequence. Each token has a numeric ID within the selected tokenizer's vocabulary. The output changes color at each token boundary. With both “Text” and “Token IDs” selected, each ID appears above the text it represents.

The display makes whitespace and incomplete character encodings visible:

Display	Meaning
␣	Space
↵	Line feed
␍	Carriage return
⇥	Tab
<0xE3>	A byte in hexadecimal. When a token's bytes cannot be decoded as UTF-8 on their own, all its bytes appear in this form.
▁	A space marker. Some tokenizers add a leading space; a token containing only that added space can appear as “▁”.

Token and character counts

The character count is the number of Unicode code points in the original input. Token counts depend on the selected tokenizer's vocabulary and rules. Some tokenizers standardize character representations before splitting the input, so the displayed text can differ from what you entered. For long inputs, the output shows only the first 5,000 tokens, while the count includes the full result.

A code point is a numbered position in Unicode. One visible symbol can contain several code points: “👩‍💻” consists of a woman emoji, a zero-width joiner, and a laptop emoji. It therefore counts as 3 characters here.

For options that display token IDs, the token count covers the input text without automatically adding chat formatting or tokens that mark the start and end of a sequence. Typing a special token's spelling produces its ID when the selected tokenizer recognizes it. The playground counts text only; images and audio are outside its scope.

The Claude options use unofficial estimates from ctok. Their totals include estimated content tokens and a fixed overhead for one user message, labeled “frame.” The colored pieces show the segmentation used to make the estimate. Claude's actual token boundaries and IDs are not public, so these options disable the ID control.

Tokenization rules

A tokenizer's vocabulary defines the available tokens. Its preprocessing rules and tokenization algorithm determine how those tokens represent the input.

Preprocessing can change both the text and the boundaries a token can cross. Some tokenizers add a leading space or apply NFC normalization. GPT tokenizers use regular expressions for pre-tokenization: they divide the input into pieces that are then tokenized separately. For example, separating every digit prevents a token from containing several digits, while grouping up to three digits allows such tokens within each group.

Unicode Normalization Form C (NFC) gives canonically equivalent character sequences a common representation, composing characters where possible. For example, “e” followed by a combining acute accent becomes the single code point “é”. Full-width letters and digits retain their full-width forms.

Byte pair encoding (BPE) starts with characters or bytes and repeatedly merges adjacent pieces in a learned priority order. It stops when no permitted pair remains. The remaining pieces form the token sequence.

Unigram tokenization assigns a score to each vocabulary entry and chooses the sequence with the highest total score. PLaMo 2 and 3 also maximize the sum of token scores, using dynamic programming. Changing the vocabulary or its scores can therefore change the chosen sequence even when preprocessing is the same.

Supported tokenizers

The following notes identify the public tokenizer definitions used for each option. Links point to the source versions used here. Vocabulary sizes count distinct token IDs, including special and reserved tokens; spellings that share an ID count once.

Preferred Networks (PLaMo)

License: PLaMo 3: PLaMo Community License Agreement (Built with PLaMo); PLaMo 2: Apache-2.0; PLaMo 1: PLaMo Non-Commercial License Agreement.

PLaMo 3 has 107,520 vocabulary entries. Before maximizing the total token score, it places boundaries around runs of two or more spaces, runs of four or more identical characters other than line feeds, and special-token spellings.
PLaMo 2 has 100,000 entries. Like PLaMo 3, it maximizes the total token score. Both follow the public implementation's score rounding and tie-breaking rules.
PLaMo 1 uses SentencePiece BPE with 50,112 entries. It has dedicated tokens for runs of spaces and tabs, and adds a leading space before tokenization.

Each vocabulary score is multiplied by 10,000 and rounded to an integer; exact halfway values round to the even integer. Dynamic programming works backward from the end of the input. When two candidate sequences have the same total score, the one with the longer token at the current position wins.

OpenAI (GPT)

License: tiktoken: MIT; Harmony: Apache-2.0.

Both options use byte-level BPE, with different vocabularies and pre-tokenization rules.

“GPT-4o–6 / gpt-oss” uses o200k_harmony, with 201,088 vocabulary entries. It shares its ordinary vocabulary and pre-tokenization rules with o200k_base and uses special tokens from Harmony, the conversation format used by gpt-oss.
The “GPT-3.5–4” option uses cl100k_base, with 100,261 entries.

o200k_harmony contains 199,998 ordinary tokens and 1,090 special tokens. cl100k_base contains 100,256 ordinary tokens and 5 special tokens. Gaps in the ID range mean that the largest ID plus one can exceed the vocabulary size.

Anthropic (Claude)

License: ctok: MIT.

ctok infers token pieces and preprocessing rules by observing Claude's token counts. The playground uses these configurations:

“Claude 4.7–5” uses ctok's v4.8 configuration and adds a 6-token frame. Its candidate pieces match ctok v4.7, but its treatment of the leading space and message frame differs.
“Claude 3–4.6” uses ctok's v3 configuration and adds a 7-token frame.

Before splitting the text, ctok normalizes it and inserts markers for word boundaries and capitalization. It then finds the sequence with the fewest tokens among its reconstructed candidates. In the output, “⟨bow⟩” and “⟨eow⟩” mark the beginning and end of a word, “⟨shift⟩” and “⟨caps⟩” record case transformations, and “⟨0xNN⟩” represents a byte in hexadecimal.

The data used here contains 14,819 string candidates and 468 byte-sequence candidates for v4.8, and 48,332 string candidates and 467 byte-sequence candidates for v3. These are pieces available to ctok's estimator, rather than a published Claude vocabulary.

Google (Gemma and Gemini)

License: Apache-2.0 (Gemma tokenizer); Apache-2.0 (Google SDK).

“Gemini 2.0–3.5 / Gemma 3–4” uses Gemma 4's 262,144-entry vocabulary. Gemma 3, 3n, and 4 share the ordinary vocabulary and splitting rules; this option uses Gemma 4's special tokens. The Gemini models covered by this option follow the mappings to Gemma 3 and 4 in the linked Google SDK.

Tokenization replaces spaces with “▁” and then applies BPE, without adding a leading space.

xAI (Grok)

License: Grok 2: xAI Community License Agreement (Powered by xAI); Grok 1: Apache-2.0.

Grok 2 uses byte-level BPE with 131,072 entries. Pre-tokenization separates individual digits, and no leading space is added.
Grok 1 also has 131,072 entries, but uses SentencePiece BPE and adds a leading space before tokenization.

Cohere (Command)

License: Command A+: Apache-2.0; Command R / R+: CC BY-NC 4.0.

Both options use byte-level BPE without adding a leading space.

Command A+ has 255,032 entries and performs no Unicode normalization. Pre-tokenization groups digits from the right in chunks of up to three and accounts for letter case and combining marks.
Command R / R+ has 255,029 entries. It applies NFC, separates individual digits, and then splits words, symbols, and whitespace. The R definition used here is identical to the August 2024 R+ definition, so both share one option.

Command R / R+ tokenizer data is by Cohere For AI. Modified by imos: the public vocabulary and rules were converted to browser JSON.

Mistral AI

License: Apache-2.0.

“Mistral 4” uses the Tekken v15 tokenizer from Mistral Small 4, with byte-level BPE and 131,072 entries. Its pre-tokenization rules distinguish uppercase and lowercase letters. When a whole piece produced by pre-tokenization matches a vocabulary entry, the tokenizer uses that entry directly instead of merging smaller pieces.

Alibaba (Qwen)

License: Qwen 3.8: Apache-2.0; Qwen 3: Apache-2.0.

Both options apply NFC and separate individual digits before byte-level BPE.

Qwen 3.8 has 248,077 entries. Its pre-tokenization rules group letters with combining marks, such as accents.
Qwen 3 has 151,669 entries. Its pre-tokenization rules differ from Qwen 3.8's in how they group combining marks.

DeepSeek

License: MIT.

“DeepSeek V3–V4” uses V4's 129,280-entry vocabulary. V4 and V3.2 share the 128,000-entry base vocabulary, merge rules, and pre-tokenization rules. This option uses V4's definitions for added tokens, including some IDs that differ from V3.2.

Before byte-level BPE, pre-tokenization groups digits in chunks of up to three and isolates sequences of Japanese and Chinese characters. Word and symbol splitting rules then apply within the resulting pieces.

Z.ai (GLM)

License: MIT.

GLM 5.1 uses byte-level BPE with 154,856 entries. Pre-tokenization groups digits in chunks of up to three. A whole piece that matches a vocabulary entry becomes that token directly; other pieces go through BPE merging.

LLM-jp

License: 4: Apache-2.0; 3: Apache-2.0; 2: Apache-2.0; 1: Apache-2.0.

The four options use unigram tokenization and add a leading space before splitting. Each has its own vocabulary and scores.

Playground option	Source version	Vocabulary size (distinct IDs)
LLM-jp 4	4	196,608
LLM-jp 3	3.1	99,574
LLM-jp 2	2.0	96,867
LLM-jp 1	1.0	50,570

LLM-jp 4 recognizes Harmony conversation markers such as <|message|>. When converting tokens back to text for display, it removes the added leading space both at the start of the text and immediately after these markers.

LLM-jp 3 and 3.1 share an option because their public definitions are identical, including the vocabulary, scores, normalization, and special tokens.

SB Intuitions (Sarashina)

License: MIT.

“Sarashina 2” uses Sarashina 2.2's 102,400-entry vocabulary and SentencePiece unigram tokenization, without an added leading space. Converting added tokens back to text can introduce spaces around them in the displayed output.