The analysis uses PLaMo 3 NICT 2604 31B Base (the 2604 release), pinned to revision e26c257e2175b05db69f01b52f4edf6fa4118f53 of pfnet/plamo-3-nict-2604-31b-base on Hugging Face. Its public implementation, modeling_plamo.py, targets Transformers 5. The repository also ships sitecustomize.py for vLLM.

PLaMo 3 NICT 31B Base[5], used for comparison, is the 4K release with a maximum sequence length of 4096; its revision is 9c1fa626. Its tokenizer definition files and code are byte-for-byte identical to those of the 2604 release.

The Gemma 4 comparison target is the pretrained google/gemma-4-31B at revision 5bbc2fb1c1b2c611d06e3d9f23c170ba21659d89. For its implementation I refer to modeling_gemma4.py in Transformers 5.12.1[7]. The execution environment and measurement conditions are given in §7.

1. [5] Preferred Networks, Inc., "PLaMo 3 NICT 31B Base (model card, config.json, modeling_plamo.py; revision 9c1fa626)," Hugging Face, 2026.
2. [7] Hugging Face, "modeling_gemma4.py (Transformers v5.12.1)," GitHub, 2026.

Per block, the QKV projection has 52,428,800 parameters, the output projection 41,943,040, the gate/up projection 262,144,000, the down projection 131,072,000, and RMSNorm \(4 \times 5120 + 2 \times 128 = 20{,}736\). The block total is 487,608,576, and the whole model has \(550{,}502{,}400 + 64 \times 487{,}608{,}576 + 5{,}120 = 31{,}757{,}456{,}384\).

The config file does not set tie_word_embeddings, so the public implementation uses the default True. The element count of all stored tensors is the same number, and there are \(64 \times 10 + 2 = 642\) tensors. RoPE frequencies are not learned parameters; they are created as buffers at run time. The public implementation pads the output head's row count up to a multiple of 16 at or above the vocabulary size, but 107,520 is already divisible by 16, so no rows are added.

The distinction affects weight decay, as the Plamo3RMSNorm docstring notes. A code comment also reports that setting every offset to 1 degraded performance, without giving the evaluation conditions, the size of the drop, or the reason for choosing 0.2 and \(5^{-1.5}\).

Weight decay pulls learned weights toward 0. If it is applied to \(w\) separately from the gradient update, as in AdamW, the offset changes the point toward which the gain decays. With learning rate \(\eta\) and decay coefficient \(\lambda\), the decay step gives \(w' = (1 - \eta\lambda)w\). The effective gain then becomes \(\gamma' = (1 - \eta\lambda)\gamma + \eta\lambda c\), moving toward \(c\). Applying the same decay directly to \(\gamma\) would move it toward 0. Folding the offset into the stored weight therefore preserves the inference formula but changes the effect of weight decay if training resumes.

1. [21] Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, Yuxuan Chen, "Query-Key Normalization for Transformers," Findings of EMNLP, 2020. (Query-Key Normalization with L2 normalization and a learnable scalar)

In local layers, Transformers' DynamicSlidingWindowLayer keeps at most the last \(W - 1 = 2047\) tokens after the cache update. The next decode step adds one new token, assigns K positions \(0, \dots, 2047\) to the 2048 tokens, and gives Q position 2047. With fixed-frequency RoPE, shifting the Q and K positions by the same amount leaves the dot product unchanged. This implementation's YaRN uses fixed frequencies and a common amplitude correction, so it keeps the same property, although finite-precision rounding need not match exactly.

The position_ids argument is not passed to attention; the internally generated sequence is used instead. To specify non-uniform positions, or to reset positions in an input that concatenates several sequences, the implementation must be changed to pass the position argument through to attention.

swa_mask builds a band mask with triu(diagonal=-(window_size - 1)). The distance to an attended token is at most \(W - 1\), so at most \(W = 2048\) tokens including the current one are attended. The local-layer cache keeps \(W - 1\) tokens, and the next decode step adds one, so prefill and decode attend to the same range.

The older implementation shipped with the 4K release uses triu(diagonal=-window_size) and attends to at most \(W + 1 = 2049\) tokens including the current one; its own cache class keeps the last \(W\) tokens. The number of tokens covered by "window size 2048" thus differs by one between implementation versions.

The 2604 release still has an assert that permits skipping the mask when \(S \le W + 1\). In training mode with the mask skipped and no cache, an input of \(T = W + 1\) tokens therefore attends to one token that should be outside the window. On a small random-weight model (\(W = 4\)), this produced a difference of up to 0.15 in the logits between training and evaluation mode (§7).

Multiplying both Q and K by \(1 + 0.1\ln256\) scales their dot product by \((1 + 0.1\ln256)^2 \approx 2.4165\). Including the usual attention scale gives a multiplier of \(2.4165 / \sqrt{128} \approx 0.2136\) on the rotated dot product before amplitude correction. The YaRN paper introduces the Q/K multiplier as \(\sqrt{1/t}\) for a temperature \(t\); the public implementation applies it through the cos/sin tables.

For a derivation of the frequency and amplitude adjustments, see the YaRN article.

Plamo3Config.rope_parameters is a property that generates the RoPE settings for local and global layers from the values in config.json. In Transformers 5.12.1, the same values are also available under the old name rope_scaling[22]. The absence of rope_scaling in the JSON does not by itself mean that YaRN is disabled.

The cited vLLM implementation[15] reads rope_parameters for each layer type. The bundled sitecustomize.py implements this separately and reads the older rope_scaling name. Its compatibility with settings that differ by layer type, and the outputs produced under vLLM, were not checked.

1. [22] Hugging Face, "configuration_utils.py (Transformers v5.12.1)," GitHub, 2026.
2. [15] vLLM Project, "vllm/model_executor/models/plamo3.py (commit d881812)," GitHub, 2026.

The reference implementation reads weights directly from safetensors and runs on CPU in FP32 with batch size 1, without padding or a cache. It takes w, a dictionary of weights converted to FP32, and ids, a token ID tensor of shape \([1, T]\). It repeats K/V heads explicitly to express GQA and constructs the attention scores directly.

import math
import torch
import torch.nn.functional as F

EPS, DH = 1e-6, 128
OFF = dict(
    pre_mixer=1.0, post_mixer=1 / 5,
    pre_mlp=1.0, post_mlp=5 ** -1.5,
    q=1.0, k=1.0, final=1.0,
)

def rms_norm(x, weight, offset):            # (c + w) ⊙ x / sqrt(mean(x^2) + ε)
    y = x.float() * torch.rsqrt(x.float().pow(2).mean(-1, keepdim=True) + EPS)
    return (offset + weight) * y.to(x.dtype)

def rope(x, theta, yarn):                    # split-half rotation; YaRN on global layers
    T = x.shape[-2]
    freq = theta ** (torch.arange(0, DH, 2).float() / DH)
    inv, conc = 1.0 / freq, 1.0
    if yarn:                                 # L0 = 4096, s = 256, β_fast = 32, β_slow = 1
        low = 64 * math.log(4096 / (32 * 2 * math.pi)) / math.log(theta)
        high = 64 * math.log(4096 / (1 * 2 * math.pi)) / math.log(theta)
        mask = 1 - ((torch.arange(64).float() - low) / (high - low)).clamp(0, 1)
        inv = (1.0 / (256 * freq)) * (1 - mask) + (1.0 / freq) * mask
        conc = 0.1 * math.log(256) + 1.0      # amplitude correction, applied to both cos and sin
    ang = torch.arange(T).float()[:, None] * inv[None]
    cos, sin = (
        torch.cat([ang, ang], -1).cos() * conc,
        torch.cat([ang, ang], -1).sin() * conc,
    )
    x1, x2 = x[..., :DH // 2], x[..., DH // 2:]
    return x * cos + torch.cat([-x2, x1], -1) * sin

def block(x, w, i):
    p = f"model.layers.layers.{i}."
    is_global = (i + 1) % 8 == 0
    T = x.shape[1]
    u = rms_norm(x, w[p + "pre_mixer_norm.weight"], OFF["pre_mixer"])
    q, k, v = torch.split(
        u @ w[p + "mixer.qkv_proj.weight"].T,
        [64 * 128, 8 * 128, 8 * 128], -1,
    )
    q = q.view(1, T, 64, 128).transpose(1, 2)        # [B, 64, T, 128]
    k = k.view(1, T, 8, 128).transpose(1, 2)         # [B, 8, T, 128]
    v = v.view(1, T, 8, 128).transpose(1, 2)
    q = rms_norm(q, w[p + "mixer.q_norm.weight"], OFF["q"])
    k = rms_norm(k, w[p + "mixer.k_norm.weight"], OFF["k"])
    theta = 1000000 if is_global else 10000
    q, k = rope(q, theta, is_global), rope(k, theta, is_global)
    # GQA: each group of 8 consecutive query heads uses one K/V head
    k = k.repeat_interleave(8, 1)
    v = v.repeat_interleave(8, 1)
    s = (q @ k.transpose(-1, -2)) / math.sqrt(128)
    i_, j_ = torch.arange(T)[:, None], torch.arange(T)[None, :]
    allowed = j_ <= i_
    if not is_global:
        allowed &= (i_ - j_) <= 2047        # local layer: the current token and the 2047 before it
    s = s.masked_fill(~allowed, float("-inf"))
    a = (torch.softmax(s, -1) @ v).transpose(1, 2).reshape(1, T, 64 * 128)
    x = x + rms_norm(
        a @ w[p + "mixer.o_proj.weight"].T,
        w[p + "post_mixer_norm.weight"], OFF["post_mixer"],
    )
    u = rms_norm(x, w[p + "pre_mlp_norm.weight"], OFF["pre_mlp"])
    g, up = (u @ w[p + "mlp.gate_up_proj.weight"].T).chunk(2, -1)
    m = (F.silu(g) * up) @ w[p + "mlp.down_proj.weight"].T
    return x + rms_norm(m, w[p + "post_mlp_norm.weight"], OFF["post_mlp"])

def forward(ids, w):
    x = w["model.embed_tokens.weight"][ids]          # no √D scaling of the embedding
    for i in range(64):
        x = block(x, w, i)
    x = rms_norm(x, w["model.norm.weight"], OFF["final"])
    return x @ w["model.embed_tokens.weight"].T     # output head tied to the embedding

Errors below are absolute differences between logits. On the public-weight test input, the largest absolute logit was 18.2. The small-model tests use random weights in the public implementation's classes; they are separate from the tests with the released 31B weights.

The checks against the public weights used PyTorch 2.12.1 (CPU) and Transformers 5.12.1. The model card specifies PyTorch 2.9.0 or earlier, but these checks ran on 2.12.1. Both implementations load the weights in FP32, so each needs about 127 GB for the weights alone. The CPU runs used 96 threads. The small model is the public implementation's class with random weights. The checks of the 4K release's older implementation used Transformers 4.57.1.

Gemma 4's 10 global layers have K and V of 4 heads × 512 dimensions, 8 KiB per layer per token. Its 50 local layers have 16 heads × 256 dimensions, 16 KiB per layer per token, and their total plateaus at about 800 MiB.

attention_k_eq_v specifies the shared projection. With \(Z\) as its output and \(N_\gamma\) denoting RMSNorm with gain \(\gamma\), global layers compute \(K = \operatorname{RoPE}(N_{\gamma_k}(Z))\) and \(V = N_1(Z)\). The table follows Transformers, which stores these processed K and V tensors separately. An implementation that caches only \(Z\) and recomputes K and V would require a different storage estimate.

For PLaMo 3, assuming 1 byte per element gives 128 MiB at 1K tokens and about 2.2 GiB at 128K tokens, matching PFN's published comparison table[8] after rounding. Parameters, masks, activations, logits, allocator overhead, and temporary prefill buffers require additional memory.

1. [8] Shuji Suzuki, "大規模言語モデルの次期バージョン PLaMo 3 シリーズにおける8B, 31Bの小規模モデルによる事前学習の検証," Preferred Networks Tech Blog, 2025.

During prefill, the public implementation builds the \(T \times S\) causal and local masks as arrays and passes them to scaled_dot_product_attention. Restricting attention through a mask does not guarantee that the Q/K dot products outside the window are skipped. At the configured maximum sequence length \(T = S = 262144\), a single FP32 mask takes 4 bytes for each of its \(T \times S\) elements, or 256 GiB. This storage requirement follows from the mask shape and dtype.

The branch that called flash_attn in the 4K release's older implementation has been removed in the 2604 release. The vLLM implementation uses an attention backend that accepts a window size.

The sequence length of 4096 includes the current token. Global layers attend to all 4096 tokens and local layers to 2048. Linear projections include the output head, and a multiply-add counts as 2 FLOPs. Each projection matrix is assumed to be read once per batch step.

Let \(V\) be the vocabulary size, \(D\) the hidden dimension, \(L\) the layer count, \(F\) the MLP intermediate dimension, \(H_q,H_{kv}\) the query and KV head counts, and \(d\) the head dimension. The number of matrix elements in the linear projections including the output head, \(P_{\mathrm{lin}}\), is given below: each block's QKV and output projections have \(2Dd(H_q+H_{kv})\) elements, the gate/up/down projections have \(3DF\), and the output head has \(VD\).

Let \(N_g=L/8\) be the number of global layers, \(N_l=7L/8\) the number of local layers, \(N\) the sequence length, and \(W\) the window size, and let \(A=N_gN+N_l\min(N,W)\) be the total number of attended tokens over all layers. The total operation count \(C\) is estimated as below, with the first term for the linear projections and the second for the attention dot products and weighted sums. The table substitutes \(B=1\), \(N=4096\), \(W=2048\), and \(d=128\), and divides the second term by \(C\) to get the share of dot products and weighted sums.

BF16 uses 2 bytes per element, so the weight reads are \(2P_{\mathrm{lin}}\) bytes and the K and V reads total \(4BH_{kv}dA\) bytes. K and V are reused across the query heads that share a KV head under GQA, but not across sequences.

For 31B, these reads total \(59.15 + 0.5625B\) GiB per batch step, or \(59.15/B + 0.5625\) GiB per generated token. For a batch of \(B\) sequences, multiply the table's per-token operation count and KV reads by \(B\).

The estimates leave out normalization, softmax, RoPE, activation functions, residual additions, and similar operations, as well as the embedding lookup and the reads of normalization weights and intermediate data. The attended range includes the new K and V, and the difference when they are used before being written to memory is ignored. Actual memory traffic depends on whether weights and K/V stay in hardware caches, on kernels rereading weights or K/V, and on how intermediate data is stored.

Gemma4TextDecoderLayer multiplies the block output by layer_scalar after the residual additions and passes the result to the next block. In Transformers it is registered as a one-element buffer rather than a learnable parameter[7]. Its initial value is 1, but the public weights store a different value for each layer (§10.5).

RMSNorm nearly cancels positive input scaling when the mean square remains well above \(\epsilon\) before and after scaling. Shrinking a block's output can therefore leave the normalized input to the next attention sublayer almost unchanged while reducing the residual carried alongside it. This changes the size of subsequent attention and MLP contributions relative to the residual. Because the direction of their sum can change too, the effect of a middle layer's scalar can persist after the final RMSNorm.

1. [7] Hugging Face, "modeling_gemma4.py (Transformers v5.12.1)," GitHub, 2026.

Transformers' proportional type[23] computes the angular frequencies with the original head dimension 512 in the exponent's denominator and then sets the low-frequency ones to 0. The rotation pairs are \((j, j + 256)\), with angular frequencies

The rotated components have indices 0 to 63 and 256 to 319, giving 128 components in total. Both their pairing and their frequencies differ from applying ordinary 128-dimensional RoPE to the first 128 components. A zero-frequency pair has cosine 1 and sine 0, so its values pass through unchanged.

Based on their analysis of Gemma 7B, Barbero et al. interpret low-frequency pairs as matching token content and propose leaving them unrotated[19].

1. [23] Hugging Face, "modeling_rope_utils.py (Transformers v5.12.1)," GitHub, 2026.
2. [19] Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, Petar Veličković, "Round and Round We Go! What makes Rotary Positional Encodings useful?," ICLR, 2025.

The analysis covers all 642 tensors (31,757,456,384 elements) of the PLaMo 3 31B 2604 release and all 832 tensors (30,697,345,340 elements) of the Gemma 4 31B text part. The Gemma 4 count includes the 60 layer_scalar values in addition to the learnable parameters.

BF16 represents values in 16 bits, so there are 65,536 possible bit patterns. The analysis reads the safetensors byte stream, counts each bit pattern per tensor, and computes statistics from the corresponding values in 64-bit floating point (FP64). No sampling was used.

Standard deviations use the full population of tensor elements (ddof = 0).

PLaMo's QKV projection was split into Q, K, and V at the boundaries in §4.1, and the gate/up projection into gate and up at the boundary in §6, with statistics computed per part. These are parts of the original tensors and are not double-counted in the total parameter count. The effective gain is the sum of weight and offset computed in FP64, without the rounding that BF16 addition would introduce.

Neither model's weights contain NaN or Inf, and PLaMo's weights contain no exact zeros.

Each mean averages all 5120 components of an RMSNorm gain. At the attention output, it is 0.116 in layer 0 and 0.269 in layer 63. At the MLP output, it reaches a minimum of 0.021 in layer 7 and ends at 0.358. The initial gains are 0.2 and about 0.089, respectively. Input-side means range from 0.977 to 1.035 before attention and from 0.905 to 1.204 before the MLP.

In the 4K release, the attention-output mean rises from 0.104 in layer 0 to 0.247 in the last layer. Between the two checkpoints, the correlation of per-layer mean gains is 0.93 for the attention output and 0.99 for the MLP output.

Figure 9 follows the last token after embedding, after each block, and after the final RMSNorm. The input from §7 gives 12 tokens including the beginning-of-sequence (BOS) token in PLaMo and 17 in Gemma.

The rise near the first global layer also appears at other non-BOS positions. The 4K release reaches 0.17 at layer 7 on this input. In the 2604 release, BOS behaves differently from the other tokens: its residual RMS reaches 6.3 in later layers.

Across all layers, there are 8192 gain components for Q and another 8192 for K. Of these, 282 Q gains and 246 K gains have absolute value below 0.05. All belong to high-frequency pairs (\(j \le 16\)) in layer 31 or later, and no pair has both components below the threshold.

Component \(i\) belongs to rotation pair \(j = i \bmod 64\). Averaging the gains by pair gives about 0.6 near the high-frequency end \(j = 0\) and about 1.4 on the low-frequency side \(j \ge 48\), with an almost monotonic increase in between. The high-frequency gains vary considerably across layers and between the two components of a pair.

The smallest gain is 0.0039 for both Q and K, with no negative values. The largest is 2.55 for Q and 2.26 for K.

In the 4K release, 273 Q components and 166 K components meet the same threshold, all at \(j \le 16\). In the 16-layer 610M Fin Instruct, 153 Q components lie at \(j \le 14\) and 101 K components at \(j \le 18\). In both models, no rotation pair had both components below the threshold.

The first two scalars are 0.097 and 0.070, and the last is 0.040; middle-layer values range from 0.46 to 0.99. The first two multiply to about 1/150, scaling the input embedding term carried through the residual path by that factor. The block outputs also include attention and MLP contributions added along the way.

The table lists each layer's stored BF16 scalar, rounded to three decimal places. Values within a row follow increasing layer index.

The last layer's output, multiplied by 0.040, goes to the final RMSNorm with nothing further added. For the last token of the input sentence used here, the RMS before the final RMSNorm was about 0.7. The ratio of \(\epsilon = 10^{-6}\) to the mean square is about \(2 \times 10^{-6}\), small enough that the final RMSNorm almost entirely cancels the shrinking factor.

Let \(W\) be the projection matrix and \(\hat{x}\) the normalized input before the gain. In column-vector notation, the output is \(W\operatorname{diag}(\gamma)\hat{x}\). Dividing a column of \(W\) by a nonzero factor while multiplying the corresponding gain component by that factor leaves the result unchanged. If several projections share the same RMSNorm output, the corresponding columns in all of them must be transformed together. This is a mathematical equivalence, not evidence that Gemma 4 underwent such a conversion.

A following RMSNorm can also cancel an overall matrix scale. PLaMo 3 has this arrangement after the Q/K, attention-output, and MLP down projections. Multiplying one of these matrices by a positive constant scales the projection output by that constant; if its mean square remains well above \(\epsilon\) before and after, the normalization denominator scales by almost the same amount. Layer 7's small MLP output (§10.3) illustrates a case where this condition fails.

Excess kurtosis measures the influence of values far from the mean: it is the fourth standardized moment minus 3, giving zero for a normal distribution.

For PLaMo 3's o_proj, median excess kurtosis across layers is 115, reaching 3506 in layer 16. These large values reflect the strong influence of weights far from the mean. The down_proj matrix in layer 6 reaches 4270, whereas gate/up matrices from layer 8 onward stay at 0.1–0.6.

For Gemma 4, median excess kurtosis across layers, calculated separately for each projection type, ranges from about 0.34 to 3.01. Individual matrices can be higher, including a v_proj at 9.38; the embedding has excess kurtosis 111.