• Message API layer: the layer of data structures you specify in Python code, such as messages=[{"role": ..., "content": ...}], tools=[...], and enable_thinking=True/False.
• Chat template layer: the layer of the raw text protocol produced by processor.apply_chat_template(..., tokenize=False). Here, control tokens such as <|turn> and <|tool_call> become visible as strings.
• Processor expansion layer: the layer in which Gemma4Processor internally expands high-level placeholders for images, audio, video, and so on into the long special-token sequences that are actually passed to the tokenizer.

<bos>
<|turn>system
[optional: <|think|>][optional: system/developer content][optional: <|tool>...<tool|> ...]
<turn|>
<|turn>user
...
<turn|>
<|turn>model
[past assistant content / tool_calls / tool_responses]
<turn|>
...
<|turn>model
[generation start position]

• Role conversion: the input assistant role is automatically converted to model by the template. In other words, what you defined as assistant at the API layer is emitted as <|turn>model at the raw-text layer.
• Reasoning trigger: the switch that enables the model's reasoning process is the <|think|> token. It is inserted at the very beginning of the first system turn.
• Tool declaration and invocation: tool declarations are enumerated inside the system turn in the form <|tool>declaration:...<tool|>. The actual tool calls and responses, on the other hand, are placed inside the assistant's turn as <|tool_call>...<tool_call|> and <|tool_response>...<tool_response|>.
• Exclusion of thinking from history: when past assistant outputs are re-fed as history, the contents of the thinking channel are automatically removed. By design, the past thinking process itself is not retained in the multi-turn conversation history.
• Non-JSON format: the format emitted by apply_chat_template(..., tokenize=False) is not JSON. It has a pseudo-JSON-like structure, but keys are not quoted, and string values are enclosed by <|"|>. Type names are also written in uppercase, such as STRING or OBJECT, in a custom format.

• When enable_thinking=True is set.
• When tools are defined.
• When the role of the first message is system or developer.

• assistant is converted to model.
• user is kept as user.
• system is kept as system.
• developer is absorbed into the system block if it is the first message; otherwise the implementation emits it as developer.

<|turn>{role}
{serialized body}<turn|>

<|turn>system
You are a helpful assistant.<|tool>declaration:get_current_weather{...}<tool|><turn|>

• Strings for user, system, and developer are trim-ed and emitted as-is.
• Strings for assistant are processed through strip_thinking() before being emitted. By this design, even if you feed the raw output back into the history, the thinking block is automatically removed.

• text
• image
• audio
• video

<bos><|turn>system
<|think|>You are a helpful assistant.<turn|>
<|turn>user
What is 2 + 2?<turn|>
<|turn>model

<|turn>model
<|channel>thought
<channel|>

<|tool>declaration:{function_name}{...}<tool|>

• Key names use bare, unquoted notation.
• String values are enclosed by <|"|>.
• Booleans are lowercase true / false.
• Type names are all uppercase, such as STRING, OBJECT, and ARRAY.
• Arrays are expressed with [ ... ] and objects with { ... }.

• The function's name and description
• The parameters' type, properties, and required
• Each property's description, type, and nullable
• The enum of a string-typed property
• The structure of array items and an optional response declaration

<|tool_call>call:{function_name}{arg1:value1,arg2:value2,...}<tool_call|>

• Object response: emitted in a form like {temperature:15,weather:<|"|>sunny<|"|>}.
• Scalar-value response: if it is not a mapping form, the template automatically wraps it with a value: key and formats it like {value:<|"|>result<|"|>}.

1. tool_calls
2. tool_responses
3. content

<|tool_call>call:f1{...}<tool_call|><|tool_call>call:f2{...}<tool_call|>

• role: always assistant
• thinking: the reasoning process as an optional string
• content: the answer text as an optional string
• tool_calls: the list of tool calls as an optional array

1. An optional thought block
2. An optional content
3. An optional tool_calls block
4. An optional trailing <turn|>

• When you want to check the raw protocol: decode with skip_special_tokens=False. This lets you directly check the placement of internal special tokens.
• When you want to handle the reasoning process, answer, and tool calls in a structured way: make use of processor.parse_response().

• Image: \n\n<|image|>\n\n
• Audio: <|audio|>
• Video: \n\n<|video|>\n\n

• Image: each <|image|> is replaced, according to the number of soft tokens per image, with multiple <|image|> sandwiched by the surrounding boundary tokens.
• Audio: each <|audio|> is similarly expanded into as many <|audio|> as the token count computed from the audio length, sandwiched by boundary tokens. In the current Gemma4Processor, this is dynamically computed from the audio waveform length, basically treated as one token per 40ms with an upper limit of 750 tokens.
• Video: video is expanded into a format with a timestamp attached to each frame. Notably, even for video, the same <|image> / <image|> boundary tokens as for images are reused. A timestamp in mm:ss format is inserted before each frame's token sequence.

[
  {"type": "image", "url": "..."},
  {"type": "audio", "url": "..."},
  {"type": "text", "text": "What is shown in this image?"}
]

messages = [
  {"role": "user", "content": "Write a haiku about memory."}
]

<bos><|turn>user
Write a haiku about memory.<turn|>
<|turn>model
<|channel>thought
<channel|>

messages = [
  {"role": "system", "content": "You are a concise assistant."},
  {"role": "user", "content": "What is 2 + 2?"},
]

<bos><|turn>system
<|think|>You are a concise assistant.<turn|>
<|turn>user
What is 2 + 2?<turn|>
<|turn>model

<|channel>thought
Compute 2+2 briefly.<channel|>4<turn|>

tools = [WEATHER_TOOL]
messages = [
  {"role": "system", "content": "You are a helpful assistant."},
  {"role": "user", "content": "Hey, what's the weather in Tokyo right now?"},
]

<bos><|turn>system
You are a helpful assistant.<|tool>declaration:get_current_weather{description:<|"|>Gets the current weather in a given location.<|"|>,parameters:{properties:{location:{description:<|"|>The city and state, e.g. "San Francisco, CA" or "Tokyo, JP"<|"|>,type:<|"|>STRING<|"|>},unit:{description:<|"|>The unit to return the temperature in.<|"|>,enum:[<|"|>celsius<|"|>,<|"|>fahrenheit<|"|>],type:<|"|>STRING<|"|>} },required:[<|"|>location<|"|>],type:<|"|>OBJECT<|"|>} }<tool|><turn|>
<|turn>user
Hey, what's the weather in Tokyo right now?<turn|>
<|turn>model
<|tool_call>call:get_current_weather{location:<|"|>Tokyo, JP<|"|>}<tool_call|>

{
  "role": "assistant",
  "tool_calls": [
    {"function": {"name": "get_current_weather", "arguments": {"location": "Tokyo, JP"}}}
  ],
  "tool_responses": [
    {"name": "get_current_weather", "response": {"temperature": 15, "weather": "sunny"}}
  ]
}

<|turn>model
<|tool_call>call:get_current_weather{location:<|"|>Tokyo, JP<|"|>}<tool_call|><|tool_response>response:get_current_weather{temperature:15,weather:<|"|>sunny<|"|>}<tool_response|>

<|turn>model
<|tool_call>call:get_current_weather{location:<|"|>Tokyo, JP<|"|>}<tool_call|><|tool_response>response:get_current_weather{temperature:15,weather:<|"|>sunny<|"|>}<tool_response|>The current weather in Tokyo is 15 degrees and sunny.<turn|>

messages = [
  {
    "role": "user",
    "content": [
      {"type": "image", "url": "https://.../cat.jpg"},
      {"type": "text", "text": "What is shown in this image?"},
    ],
  }
]

<bos><|turn>user

<|image|>

What is shown in this image?<turn|>
<|turn>model
<|channel>thought
<channel|>

<bos><|turn>user

<|image><|image|><|image|>...<image|>

What is shown in this image?<turn|>
<|turn>model
<|channel>thought
<channel|>

• Check the raw string when prompting and debugging: during development, always check the output of apply_chat_template(..., tokenize=False) at least once. Understanding what string is actually passed to the model is the first step in troubleshooting.
• Do not skip special tokens when checking the raw output: if you want to check the behavior of protocol-level tokens, specify skip_special_tokens=False when decoding.
• Use the dedicated method for parsing in production: in a real application, use processor.parse_response() instead of parsing the raw output yourself.
• Manage tool execution with a whitelist approach: the tool calls the model returns are merely string requests. Execute them only after validating the function name and arguments.
• Define complex tool schemas manually: for complex functions involving nested objects or custom classes, it is safer to define the schema manually than to rely on automatic schema generation.
• Be aware of the internal expansion of multimodal input: even when only one placeholder such as <|image|> is visible on the template, it is internally expanded into many soft tokens. Design with a clear distinction between this placeholder layer and the internal expansion layer.

1. [S1] Hugging Face chat_template.jinja: the chat template implementation of google/gemma-4-31B-it. https://huggingface.co/google/gemma-4-31B-it/blob/main/chat_template.jinja
2. [S2] Hugging Face tokenizer_config.json: defines the tokenizer configuration and the response schema. https://huggingface.co/google/gemma-4-31B-it/blob/main/tokenizer_config.json
3. [S3] Hugging Face Model Card: the official model card for Gemma 4 (google/gemma-4-31B). https://huggingface.co/google/gemma-4-31B
4. [S4] Google AI for Developers: the official guidelines on function calling with Gemma 4. https://ai.google.dev/gemma/docs/capabilities/text/function-calling-gemma4
5. [S5] Hugging Face Transformers Docs: the gemma4.md documentation in the Transformers library. https://github.com/huggingface/transformers/blob/main/docs/source/en/model_doc/gemma4.md
6. [S6] Hugging Face Transformers Source: the processor implementation file processing_gemma4.py. https://github.com/huggingface/transformers/blob/main/src/transformers/models/gemma4/processing_gemma4.py