charsim: Character n-gram similarity without per-dataset tuning
Published on July 2, 2026
charsim: Character n-gram similarity without per-dataset tuning
When evaluating a generated answer or translation, we often compare it with one or more reference texts. The generated text, or candidate, might reproduce a useful phrase while leaving out the rest of an answer, or repeat that phrase several times. A score based on shared wording should give the matching text credit while accounting for what is missing or extra.
I propose charsim for evaluating sentences and short texts this way. It adds up weighted matches between character sequences and divides by the larger of the candidate and mean reference lengths. This selects the smaller of precision and recall, handling omissions and excess without a coefficient to balance them. Weighting each order by \(1/n\) reduces the contributions added by longer n-grams relative to uniform weighting. Section 5.7 explains when changing the maximum order has little effect and when it changes the score.
Two established metrics help explain the design. BLEU can give a sentence zero when none of the longer word sequences match, even if shorter matches remain[1]. Avoiding this loss of credit requires an adjustment such as smoothing. charsim gives partial matches credit directly. chrF does too, but uses a coefficient to balance the penalties for missing and extra text[2]. charsim handles both with the same length rule. Section 5 develops these comparisons.
A first example
With cat as the single reference, the exact match scores 1. Removing a character, adding one, or repeating the word lowers the score, but each candidate still receives credit for the parts that match.
Candidate
Change from reference
charsim
cat
Exact match
1.000
ca
One character removed
0.498
cats
One character added
0.592
catcat
Repeated word
0.449
The candidate ca leaves part of the reference uncovered; cats contains extra text. Repeating the word as catcat adds length, but a repeated sequence earns credit only up to its count in the reference. Section 3.4 works through the calculation, including matches at the start and end of each string.
What the score measures
The sequences that charsim counts are character n-grams: runs of \(n\) consecutive characters. Each occurrence contributes \(1/n\), up to a maximum of 32 characters per n-gram. A single character therefore contributes 1, a two-character sequence contributes \(1/2\), and so on. Summing these contributions gives a string's weighted length; counting only shared occurrences gives the weighted overlap.
For one reference, charsim divides the weighted overlap by the larger of the two weighted lengths. A short candidate is scored by how much of the reference it covers; a long candidate is scored by how much of its own text matches. A high score therefore requires both substantial coverage and little extra text.
With several references, charsim averages the overlap with each reference and uses their mean weighted length. This measures agreement across the set: matching just one reference earns less credit when the others differ. The score lies between 0 and 1. Section 2 gives the full definition, including empty inputs.
The references should reflect the evaluation goal. Averaging suits a task that rewards wording shared across the set. If the references are alternatives and matching any one should suffice, take the maximum per-reference score instead. Section 3 explains how these choices affect the result.
charsim measures shared wording. A paraphrase can score poorly even when it preserves the meaning. The examples and simplified models in this article explain the metric's behavior; order-cap sensitivity on real datasets, comparisons with other metrics, and validation against human judgments remain future work.
1. Counting character n-grams
1.1. Strings and references
Let \(t\) be the candidate and \(\mathcal R=(r^{(1)},\dots,r^{(K)})\) its reference set, containing \(K\) texts. The set is a multiset: order does not matter, and duplicates are allowed. Including a reference twice gives it twice the weight of one included once. The score is undefined when \(K=0\).
charsim counts characters as Unicode code points. We write \(|x|\) for the number of code points in string \(x\). Whitespace counts as text, and case matters. The metric compares the input as supplied. If you preprocess it, report those steps as part of the evaluation conditions, including any Unicode normalization.
A displayed character can contain more than one code point. For example, é can be represented by one code point or by e followed by a combining acute accent. charsim counts these as one and two characters, respectively, unless preprocessing normalizes them to the same representation.
The quantities below depend on \(t\) and \(\mathcal R\); their arguments are omitted when the inputs are clear.
1.2. Include matches at the string boundaries
The number \(n\) is the n-gram's order. Before extracting n-grams, pad each string with one start marker and one end marker. Writing those markers as ^ and $, cat becomes ^cat$. Its 2-grams are ^c, ca, at, and t$; its 3-grams are ^ca, cat, and at$. Boundary markers make a match at the start or end distinct from the same text elsewhere.
In an implementation, use dedicated marker values distinct from every possible input character. Exclude n-grams made entirely of markers: otherwise, every pair of strings would share a match just because both were padded. For an empty string, this excludes all three possible n-grams: ^, $, and ^$.
Let \(\ell_{x,n}\) be the number of order-\(n\) n-grams extracted from \(x\), counting every occurrence. A nonempty string has \(|x|\) 1-grams and \(|x|+1\) 2-grams because the 2-grams include the pairs at the start and end markers. Beyond order 2, each increase in order removes one possible starting position, until no n-grams remain. The complete count formula also handles empty strings.
A padded nonempty string has length \(|x|+2\), so it has \(|x|+3-n\) starting positions for an n-gram when \(2\le n\le|x|+2\). Including the unigram and empty-string rules gives:
\[
\ell_{x,n}=\begin{cases}
0 & (|x|=0)\\[2pt]
|x| & (|x|\gt0,\ n=1)\\[2pt]
|x|+3-n & (|x|\gt0,\ 2\le n\le|x|+2)\\[2pt]
0 & (n\gt|x|+2)
\end{cases}
\]
For reference \(r^{(i)}\), write \(\ell_{i,n}\) as shorthand for \(\ell_{r^{(i)},n}\).
1.3. Count shared occurrences
Let \(c_x(g)\) be the number of times n-gram \(g\) occurs in \(x\). The order is determined by the length of \(g\). For a given order \(n\), the overlap with reference \(i\) is
\[
m_{i,n}=\sum_g\min\bigl(c_t(g),c_{r^{(i)}}(g)\bigr),
\]
where the sum runs over distinct n-grams of that order, excluding boundary-only n-grams.
If an n-gram occurs five times in the candidate and twice in a reference, it contributes two matches. Taking the smaller count clips the candidate's occurrences at the reference count, so increasing the candidate count to six earns no additional credit for that n-gram.
2. Defining charsim
2.1. Fixed weights and a fixed order cap
When at least one input string is nonempty, the largest order present in any input is
\[
N'=\max\bigl(|t|,\max_i|r^{(i)}|\bigr)+2.
\]
Orders above \(N'\) contain no n-grams. Computing all orders is therefore finite, but becomes expensive for long strings. charsim uses
\[
N=\min(N',32).
\]
All orders are included when the longest input has at most 30 characters; otherwise the sum stops at 32. This common setting includes longer matches while keeping the number of extracted n-grams proportional to input length (Section 7.5). The same cap applies to the candidate and every reference.
Weight each occurrence by \(1/n\), then sum across orders:
\[
\begin{aligned}
M_i&=\sum_{n=1}^{N}\frac{m_{i,n}}{n},\\
L_t&=\sum_{n=1}^{N}\frac{\ell_{t,n}}{n},\qquad
L_i=\sum_{n=1}^{N}\frac{\ell_{i,n}}{n}.
\end{aligned}
\]
Here, \(\ell_{i,n}\) is the n-gram count for reference \(i\). \(M_i\) is its weighted overlap with the candidate, while \(L_t\) and \(L_i\) are their weighted lengths. A long matching passage contributes n-grams at several orders. The \(1/n\) factor reduces the weight of each longer n-gram while retaining credit for the passage's continuity. Section 5.7 examines the effect of changing the cap, and Section 6.4 compares these weights with uniform weighting.
2.2. Average first, then normalize
Average the overlap and the reference length separately:
\[
\bar M=\frac1K\sum_{i=1}^{K}M_i,\qquad
\bar L_r=\frac1K\sum_{i=1}^{K}L_i.
\]
For \(L_t\gt0\) and \(\bar L_r\gt0\), the main definition is
\[
\boxed{\;\operatorname{charsim}(t;\mathcal R)
=\frac{\bar M}{\max(L_t,\bar L_r)}=\min(P,R)\;}
\]
with
\[
P=\frac{\bar M}{L_t},\qquad R=\frac{\bar M}{\bar L_r}.
\]
Precision \(P\) measures the fraction of the candidate's weighted n-grams that match, averaged across references. Recall \(R\) measures the fraction of the references' total weighted n-gram count that is matched. Dividing mean overlap by mean reference length gives this fraction because the factors of \(1/K\) cancel. Both precision and recall use counts summed across orders with the \(1/n\) weights applied.
Dividing by the larger length selects the smaller fraction. High precision with low recall means the candidate contains matching text but leaves much of the reference material uncovered. High recall with low precision means it covers the reference material but contains substantial extra text. Taking the minimum requires both fractions to be high for a high score.
For every reference, \(M_i\le L_t\) and \(M_i\le L_i\), because shared occurrences cannot outnumber the occurrences on either side. Averaging preserves both bounds: \(\bar M\le L_t\) and \(\bar M\le\bar L_r\). Thus charsim lies between 0 and 1.
2.3. Empty strings and missing references
Handle zero lengths before dividing:
If the candidate and every reference are empty, the score is 1.
If exactly one of \(L_t\) and \(\bar L_r\) is zero, the score is 0.
If there are no references (\(K=0\)), the score is undefined. An empty reference set is different from a set containing an empty string.
These rules complete the main definition. The per-reference form in Section 3 applies the same empty-string convention to each pair before averaging.
2.4. Why use the mean reference length?
Using the mean for both overlap and reference length keeps recall interpretable as a matched fraction between 0 and 1. It also allows the reference counts to be combined before any candidate is evaluated. The mean length and a lookup table for mean overlap can then be reused for many candidates (Section 7.2).
I call this the mean-length version; charsim without a qualifier refers to it throughout the article. Another approach is to normalize each candidate–reference pair first, then average the scores. That changes how references of different lengths affect the result.
3. Per-reference scores and worked examples
3.1. The base form: normalize first, then average
For each reference, normalize the overlap using that reference's own length:
\[
s_i=\frac{M_i}{\max(L_t,L_i)}.
\]
If both strings are empty, set \(s_i=1\); if only one is empty, set \(s_i=0\). For nonempty pairs,
\[
P_i=\frac{M_i}{L_t},\qquad R_i=\frac{M_i}{L_i},\qquad
s_i=\min(P_i,R_i).
\]
The base form is the mean of these scores:
\[
\boxed{\;\operatorname{charsim}_{\mathrm{base}}(t;\mathcal R)
=\frac1K\sum_{i=1}^{K}s_i\;}
\]
The two forms differ in the order of operations. The base form normalizes each pair, then averages the scores. The mean-length version averages the overlaps and reference lengths, then normalizes once. In the base form, each pair's normalized score has equal weight, so the result is also the expected score when one reference is drawn uniformly from the multiset. Different denominators generally prevent the overlaps from being combined before division.
3.2. When the two forms agree
The two forms are equal when all reference lengths are equal. They also agree when the candidate is at least as long as every reference. In either case, the base form has a common denominator, so averaging and normalization can be interchanged.
If every \(L_i=L_r\), the common denominator is \(\max(L_t,L_r)\). For a positive denominator,
\[
\frac1K\sum_i\frac{M_i}{\max(L_t,L_r)}
=\frac{\bar M}{\max(L_t,L_r)}
=\operatorname{charsim}.
\]
If \(L_t\ge L_i\) for every reference, the common denominator is \(L_t\). When all lengths are zero, both forms equal 1 by the empty-input convention.
3.3. When reference lengths differ
Consider candidate a and references ["a", "abcdefghij"]. The base form scores about 0.528, while the mean-length version scores about 0.134. The candidate matches the first reference exactly, but the longer second reference raises the mean length used by the main definition. A perfect match to one reference does not make the aggregate score 1.
This difference matters when a reference set mixes short and long answers. The base form gives each pair's normalized score equal weight, allowing an exact match to a short reference to contribute a full 1 before averaging. The mean-length version uses a common length baseline, so the longer references also affect how the overlap with that short reference is normalized.
Both forms still average over references. If any one acceptable answer should suffice, use the maximum per-reference score instead. Grouping references by length or content is another way to make the evaluation target more specific.
Before using the mean-length version, check whether the mean represents the answers you want to reward. Mixing brief summaries with detailed explanations creates an intermediate length that may represent neither group. A few unusually long references can also raise the baseline. If these are distinct kinds of acceptable answer, evaluate them in separate groups rather than treating their average as the target.
3.4. Working through the first example
Return to the four candidates compared with cat in the first example. With one reference, the main and base forms give the same scores. For the exact match, \(P=R=1\). For ca, recall is the smaller fraction; for cats and catcat, precision is smaller. The cap \(N\) is 5 for cat and ca, 6 for cats, and 8 for catcat.
For ca, count the shared n-grams at each order before applying the weights. The shared unigrams are c and a, the shared bigrams are ^c and ca, and the only shared trigram is ^ca. There are no matches at orders 4 or 5.
The padded strings are ^ca$ and ^cat$, with \(N=5\). Since there is only one reference, omit the reference index from \(m_n\) and write its counts as \(\ell_{r,n}\).
Order \(n\)
1
2
3
4
5
Overlap \(m_n\)
2
2
1
0
0
\(m_n/n\)
2
1
\(1/3\)
0
0
Candidate count \(\ell_{t,n}\)
2
3
2
1
0
\(\ell_{t,n}/n\)
2
\(3/2\)
\(2/3\)
\(1/4\)
0
Reference count \(\ell_{r,n}\)
3
4
3
2
1
\(\ell_{r,n}/n\)
3
2
1
\(1/2\)
\(1/5\)
Summing the weighted rows gives \(\bar M=10/3\), \(L_t=53/12\), and \(\bar L_r=67/10\). The reference is longer, so the denominator is \(\bar L_r\):
\[
\begin{aligned}
P&=\frac{10/3}{53/12}=\frac{40}{53}\approx0.755,\\
R&=\frac{10/3}{67/10}=\frac{100}{201}\approx0.498,\\
\operatorname{charsim}(\texttt{ca};[\texttt{cat}])
&=\min(P,R)\approx0.498.
\end{aligned}
\]
About three quarters of the candidate's weighted n-grams match, but they cover only about half of the reference's weighted n-grams. Recall is therefore the limiting fraction and sets the score. A high matched fraction within a short candidate cannot compensate for the reference material it leaves uncovered.
4. How the score changes
4.1. Approaching a score of 1
For nonempty inputs, a score near 1 requires \(\bar M\approx L_t\approx\bar L_r\). The candidate's weighted length must be close to the mean reference length, and nearly all of that weight must come from shared occurrences. A candidate of the right length still scores poorly when few of its n-grams match the references.
4.2. Short candidates are limited by recall
When \(L_t\lt\bar L_r\), recall determines the score. Since \(\bar M\le L_t\),
\[
\operatorname{charsim}\le\frac{L_t}{\bar L_r}.
\]
For a fixed reference set, this upper bound falls as the candidate gets shorter. Even a fragment whose characters all match cannot cover the rest of the reference material.
4.3. Long candidates are limited by precision
When \(L_t\gt\bar L_r\), precision determines the score. Since \(\bar M\le\bar L_r\),
\[
\operatorname{charsim}\le\frac{\bar L_r}{L_t}.
\]
Every candidate n-gram up to the cap contributes to \(L_t\), whether it matches or not. If positive overlap stays unchanged while the candidate grows, only the denominator increases and the score falls.
4.4. Separate the contribution of each order
With the candidate and reference lengths fixed, the denominator \(D=\max(L_t,\bar L_r)\) is constant. For \(D\gt0\), define \(\bar m_n=K^{-1}\sum_i m_{i,n}\) and decompose the score as
\[
\operatorname{charsim}=\sum_{n=1}^{N}\frac{\bar m_n}{nD}.
\]
Each term measures the contribution from n-grams of that order. For candidates of equal length, we can therefore trace a score difference to matches of particular lengths: short fragments, longer passages, or both. The same formula gives the score change for a given change in overlap.
With the lengths fixed, an increase of 1 in the mean overlap \(\bar m_n\) adds \(1/(nD)\) to the score. An extra match with just one reference adds \(1/(KnD)\). An edit that preserves character count can change overlaps at several orders; summing their contributions gives the total score change.
4.5. Exact matches and character edits
For a single reference, identical strings score 1 because their weighted overlap and lengths are equal. This holds for any positive order cap. Replacing a character changes the n-grams that contain it, while unaffected matches continue to contribute. Even if all matches at higher orders are lost, the score remains positive as long as shorter matches survive.
For very short inputs, one character can account for all the overlap. Replacing candidate a with b against reference a changes the score from 1 to 0.
A single-reference score of 1 means the compared n-gram multisets are identical, including occurrence counts. With a truncated order range, different strings can have the same multisets. Use direct string comparison when exact text equality is the requirement.
4.6. Repetition earns credit only up to the reference count
The overlap for an n-gram is \(\min(c_r(g),c_t(g))\). Repetition in the candidate earns credit only until it reaches the count in the reference; further occurrences increase candidate length without adding overlap for that n-gram. BLEU and chrF also limit the occurrences counted as matches.
5. Comparing the design with BLEU and chrF
BLEU compares word n-grams, while chrF compares character n-grams[1, 2]. Here I use the original unsmoothed BLEU definition and the chrF formulation that averages precision and recall across orders before combining them into an F-score. Comparing their operations shows where charsim changes the treatment of length, partial matches, and multiple references.
5.1. Combining precision and recall
BLEU combines precision values across orders using a geometric mean: with four equally weighted orders, it takes the fourth root of their product. Precision penalizes extra text; a separate brevity penalty reduces the score when the candidate is shorter than the effective reference length.
chrF combines precision and recall through an F-score, a weighted harmonic mean. Its coefficient \(\beta\) gives the two fractions equal weight when \(\beta=1\) and gives recall more weight when \(\beta\gt1\). The coefficient therefore controls the balance between penalizing extra text and penalizing missing text.
charsim takes the smaller fraction. Precision limits a candidate whose weighted length exceeds the mean reference length; recall limits one that falls short. No coefficient sets their relative weight. Section 5.5 compares the minimum with the F-score for the same precision and recall values.
5.2. Zero matches at one order do not erase other matches
The original BLEU definition combines counts across an evaluation corpus before computing precision at each order. If any of those precisions is zero, their geometric mean is zero too. This is a particular concern for sentence evaluation, where longer matching sequences are scarce: a sentence can receive zero despite sharing shorter sequences with a reference. Smoothing avoids that collapse, for example by replacing a zero precision with a small positive value, but the choice of smoothing method then affects the score.
charsim adds weighted counts, so an order with no matches contributes zero without erasing other orders' contributions. Partial matches therefore receive credit without smoothing. chrF also preserves that credit by using arithmetic averages.
5.3. What multiple references mean
BLEU clips each candidate word n-gram count at the largest count found in any reference. Matches from different references can therefore contribute to the same score. Multiple-reference chrF depends on the implementation: sacreBLEU v2.5.1, for example, selects the reference with the highest F-score for each candidate[4].
charsim averages overlap across references, so weak agreement with the rest of the set reduces the credit from a close match to just one reference. The base form also averages, but does so after normalizing each pair. These choices serve different evaluation goals from selecting the best reference (Section 3.3).
5.4. Combining n-gram orders
chrF calculates precision and recall at each order, then averages each set of ratios. charsim sums the weighted overlap and n-gram counts across orders, then divides. Averaging ratios gives each order equal weight; dividing sums lets the number of n-grams, as well as the \(1/n\) weight, determine each order's influence. With a common denominator, we can attribute score differences to overlap at each order, as in Section 4.4.
5.5. Taking the minimum instead of an F-score
To isolate the choice between a minimum and an F-score, give both formulas the same precision \(P\) and recall \(R\). The F-score lies between these two values:
\[
\min(P,R)\le F_\beta\le\max(P,R)\qquad(\beta\gt0).
\]
For example, \(P=1\) and \(R=0.5\) give \(F_1\approx0.67\), while \(\min(P,R)=0.5\). Taking the minimum means that a high value on one side cannot compensate for a low value on the other. High scores require both little excess and little missing material.
For \(P,R\gt0\),
\[
F_\beta=\frac{(1+\beta^2)PR}{\beta^2P+R},\qquad
\frac1{F_\beta}=\frac1{1+\beta^2}\frac1P+
\frac{\beta^2}{1+\beta^2}\frac1R.
\]
The reciprocal is a weighted average of \(1/P\) and \(1/R\), which gives the bounds. When either \(P\) or \(R\) is zero, take \(F_\beta=0\).
charsim and chrF compute \(P\) and \(R\) differently because they use different boundary rules and ways of combining orders. The bound therefore does not determine which metric scores a text higher.
5.6. Character sequences and the order cap
Both charsim and chrF work without a word tokenizer and can count substrings that survive spelling changes. chrF++ adds word unigrams and bigrams to character n-grams, incorporating word-level matches and adjacency[3].
The original BLEU paper uses word n-grams through order 4, and the chrF paper uses character n-grams through order 6[1, 2]. Both give each order equal weight in their respective means. The linguistic span of an n-gram still depends on the language and writing system: a character 6-gram covers only part of the 18-character German word Lebensversicherung, for example. Word n-grams depend on how text is divided into words.
I set charsim's cap at 32 to include longer matching substrings, and use \(1/n\) weights to reduce the credit for each long match. This combination rewards long matches while emphasizing shorter ones. To assess the effect of extending the cap, we need to examine what the added orders contribute to both overlap and length.
5.7. Limiting high-order contributions and assessing cap sensitivity
Two choices make larger order caps easier to use. Summing matches before dividing preserves credit at lower orders even when an added order has no matches. Weighting by \(1/n\) also reduces the amount that each high-order overlap or length term adds compared with uniform weighting. These mechanisms explain the design's reduced dependence on order selection; the size of a score change still depends on the added counts.
Raising the cap includes more orders in both the overlap and length sums. Weighting by \(1/n\) reduces these high-order terms relative to uniform weighting. To isolate the effect of the weights, suppose each order has the same number of n-grams. At a cap of 32, orders above 4 account for about 49% of the total weight with \(1/n\) weighting, compared with 88% under uniform weighting. The weight sums give the proportions below.
For cap \(N\), the fraction of weight above order \(N_0\) is \(1-H_{N_0}/H_N\) for \(1/n\) weights and \(1-N_0/N\) for uniform weights, where \(H_N=\sum_{n=1}^{N}1/n\). The table uses \(N_0=4\) and rounds to two decimal places.
Fraction of weight above order 4
Cap 8
Cap 16
Cap 32
Cap 64
\(1/n\) weights
0.23
0.38
0.49
0.56
Uniform weights
0.50
0.75
0.88
0.94
Many small terms can still add up to a substantial contribution: at cap 32, orders above 4 account for about 49% of the weight in the table. A cap change has little effect when the added overlap and length are small relative to the original denominator, or when their increases preserve the original ratio. The change in score makes these conditions explicit.
Write the numerator, denominator, and score at cap \(N\) as \(A_N,D_N,S_N=A_N/D_N\). Increasing the cap from \(N_0\) to \(N_1\) adds \(\Delta A=A_{N_1}-A_{N_0}\) and \(\Delta D=D_{N_1}-D_{N_0}\). For \(D_{N_0}\gt0\),
\[
S_{N_1}-S_{N_0}
=\frac{\Delta A-S_{N_0}\Delta D}{D_{N_0}+\Delta D}.
\]
If both increments are small relative to the original denominator, the change is small. If \(\Delta A=0\), then \(S_{N_1}=S_{N_0}D_{N_0}/D_{N_1}\). If the candidate and every reference are identical, numerator and denominator grow by the same amount and the score stays 1.
For example, if the added orders have no matches but increase the denominator by 10%, the score becomes \(1/1.1\), or about 91%, of its previous value. Zero high-order overlap alone does not make the score insensitive to the cap. If the candidate and every reference are identical, both sums grow equally and the score stays 1.
charsim therefore removes the choice of smoothing and a precision–recall balance coefficient, while cap sensitivity depends on the input. Long matches in set phrases, repeated text, or source code can also add to the numerator at high orders. When comparing a variant with a different cap, check scores and candidate rankings on the same data and report the cap as part of the evaluation conditions.
6. What simplified models reveal
The following models separate the effect of length from the effect of matching text. Sections 6.1–6.3 hold the match rate constant across n-gram orders to compare length penalties. Section 6.4 holds the number of matching characters constant and changes their distribution to examine the reward for continuity. All calculations use assumed overlap counts rather than measurements of actual sentences.
6.1. Hold the match rate constant across orders
Start with one reference of length \(\lambda\gt0\) and a candidate of length \(\hat\lambda\gt0\). To isolate the length rule, ignore boundary markers, the distinction between words and characters, and the variation in n-gram counts across orders. At every order, set the candidate count to \(\hat\lambda\) and the reference count to \(\lambda\).
Assume that a fixed fraction \(p\) of the shorter side's n-grams match, where \(0\lt p\le1\). The overlap is then
\[
m_n=p\min(\hat\lambda,\lambda).
\]
Precision and recall are each constant across n-gram orders:
\[
P=\frac{p\min(\hat\lambda,\lambda)}{\hat\lambda},\qquad
R=\frac{p\min(\hat\lambda,\lambda)}{\lambda}.
\]
Substituting these values into the three score formulas lets us compare their length penalties at a fixed match rate. I use \(\beta=2\) for the chrF model, giving recall more weight. All three model scores are independent of the order cap because every order has the same precision and recall. For an empty candidate, set all three to 0.
The model scores are
\[
\begin{aligned}
S_{\mathrm{charsim}}&=\min(P,R),\\
S_{\mathrm{chrF}}&=\frac{5PR}{4P+R},\\
S_{\mathrm{BLEU}}&=P\,\mathrm{BP},\qquad
\mathrm{BP}=\begin{cases}
1 & (\hat\lambda\ge\lambda),\\
\exp(1-\lambda/\hat\lambda) & (0\lt\hat\lambda\lt\lambda).
\end{cases}
\end{aligned}
\]
In charsim, the factor \(\sum_n1/n\) cancels between weighted overlap and length. BLEU's geometric mean of identical precisions is \(P\), and chrF's arithmetic means are \(P\) and \(R\). Including more orders therefore leaves all three model scores unchanged.
6.2. All three models peak at the reference length
All three scores reach their maximum \(p\) when \(\hat\lambda=\lambda\). For charsim, the model reduces to
\[
S_{\mathrm{charsim}}
=p\,\frac{\min(\hat\lambda,\lambda)}{\max(\hat\lambda,\lambda)}.
\]
Shorter candidates lose recall; longer candidates lose precision. The models share the peak's position and height, but differ in how they penalize departures from it.
6.3. Equal length ratios and equal character differences
The charsim model is symmetric in the length ratio: half the reference length and twice the reference length both score \(p/2\). Equal increases and decreases in character count produce different scores, however. With \(\lambda=100\) and \(p=0.8\), candidates of lengths 70 and 130 score 0.56 and about 0.62. The curve is linear below the reference length and falls in proportion to inverse length above it.
The chrF model weights recall more heavily, so it penalizes shortfall more than excess. Its scores at candidate lengths 70 and 130 are about 0.60 and 0.75, respectively.
BLEU's brevity penalty sharply reduces scores for very short candidates: the model scores about 0.015 at length 20 and 0.078 at length 30. At or above the reference length, the penalty is 1, so the BLEU model depends only on precision and equals charsim's \(p\lambda/\hat\lambda\).
Figure 1: Candidate length and model score (\(\lambda=100,\ p=0.8\))
The dashed blue curve is charsim, the solid orange curve is BLEU, and the solid green curve is chrF. The red dashed guide decreases linearly at the same rate in either direction from the peak. BLEU and charsim coincide for candidate lengths of 100 or more.
We can summarize the imbalance between the two sides of the peak with a score-weighted mean candidate length, or centroid. Use the integer lengths from 50 to 150, weighting each length by its model score. If equally distant lengths on either side of 100 had equal scores, the centroid would be 100. The results are
\[
\bar\mu_{\mathrm{chrF}}\approx103.3,\quad
\bar\mu_{\mathrm{BLEU}}\approx102.7,\quad
\bar\mu_{\mathrm{charsim}}\approx101.5.
\]
All three centroids lie above the reference length of 100, reflecting the higher scores on the long side of the peak. By this measure, charsim has the smallest imbalance over the chosen range, with a shift of about 1.5 characters.
For model score \(S(j)\) at candidate length \(j\),
\[
\bar\mu=\frac{\sum_{j=50}^{150}jS(j)}{\sum_{j=50}^{150}S(j)}.
\]
The reported values are rounded to one decimal place and can be reproduced with the script linked in Section 7.4.
6.4. Contiguous and dispersed matches
A candidate that copies only a long stock opening from a reference can accumulate matches at many n-gram orders in that one passage. If the evaluation should reward agreement throughout an answer, the size of this reward matters. Holding the matching character count fixed lets us isolate how much the weighting scheme favors one continuous passage over dispersed matches.
When matching characters form a shared passage, they also contribute longer matching n-grams. To examine how the weights reward this continuity, fix both string lengths at \(\lambda\) and the unigram overlap at \(q\lambda\), where \(0\lt q\le1\). Compare two patterns with that same unigram overlap:
Dispersed matches: treat each position as matching independently with probability \(q\), and set \(m_n=q^n\lambda\).
A contiguous block: let a single block of length \(b=q\lambda\) match, and set \(m_n=\max(b-n+1,0)\).
The model excludes boundary markers and chance matches at different positions. For dispersed matches, it approximates the number of n-gram starting positions by \(\lambda\). Divide the block's score by the dispersed pattern's score to measure the reward for continuity: a ratio of 2 means the block scores twice as high. Within each weighting scheme, the denominators cancel, so this score ratio depends only on weighted overlap. Compare the ratios for \(1/n\) weights and uniform weights.
For order weights \(w_n\), the common denominator cancels:
\[
B_N=
\frac{\sum_{n=1}^{N}w_n\max(q\lambda-n+1,0)}
{\sum_{n=1}^{N}w_nq^n\lambda}.
\]
The curves use \(w_n=1/n\) and \(w_n=1\). Both sum overlap counts before normalization. The uniform curve therefore represents a variant of the charsim calculation; chrF instead averages per-order precision and recall.
At \(\lambda=100\), \(q=0.8\), and cap 32, the contiguous block's score is about 5.2 times the dispersed pattern's score under uniform weights. With \(1/n\) weights, that ratio is about 1.8. Raising the cap to 100 increases the ratios to about 8.1 and 2.0, respectively.
Figure 2: Match distribution and order weighting (\(\lambda=100,\ q=0.8\))
The horizontal axis is the order cap; the vertical axis is the block's score divided by the dispersed pattern's score. The blue curve uses \(1/n\) weights and shows a smaller reward for continuity than the orange curve with uniform weights.
Both schemes reward the contiguous block. The \(1/n\) weights preserve that reward while reducing the gap between a single matching passage and matches spread throughout the text. This is one reason to use them when including longer n-grams: they limit how much a copied passage can dominate the score. Tasks that specifically value long continuous matches may instead need greater emphasis on those matches.
7. Implementation notes
7.1. Direct calculation
Check for missing references and empty inputs first (Section 2.3). Otherwise, set \(N=\min(N',32)\), pad each string with dedicated boundary markers, and process each order:
Build an n-gram frequency table for the candidate and each reference, excluding boundary-only n-grams.
Compute each overlap \(m_{i,n}=\sum_g\min(c_t(g),c_{r^{(i)}}(g))\).
Count the candidate and reference n-grams, \(\ell_{t,n}\) and \(\ell_{i,n}\).
Multiply each count by \(1/n\) and add it to the corresponding total: \(M_i\) for overlap, \(L_i\) for each reference's length, and \(L_t\) for the candidate's length. Count the candidate's length once per order.
For the main definition, average the overlaps and reference lengths before normalizing. For the base form, normalize each pair before averaging.
7.2. Precompute a fixed reference set
When many candidates share the same reference set, the mean-length version can reuse precomputed reference data. For each n-gram \(g\), precompute its mean clipped overlap as a function of the candidate count \(m\):
\[
F_g(m)=\frac1K\sum_{i=1}^{K}\min\bigl(c_{r^{(i)}}(g),m\bigr).
\]
To evaluate a candidate, count its n-grams and look up the mean overlap for each count:
\[
\bar m_n=\sum_gF_g\bigl(c_t(g)\bigr),\qquad
\bar M=\sum_{n=1}^{N}\frac{\bar m_n}{n}.
\]
Each distinct candidate n-gram requires one lookup, regardless of the number of references. This avoids a separate comparison with every reference for each new candidate.
Build \(F_g\) from the distribution of occurrence counts across references. Counts above the largest reference count all give the same overlap, so the table needs only finitely many entries. Store \(\bar L_r\) as well; each candidate then requires only \(\bar M\) and \(L_t\) to complete the score.
The identity
\[
\min(c,m)=\sum_{a=1}^{m}\mathbf1[c\ge a]
\]
expresses a clipped count as a sum of indicators, where \(\mathbf1[c\ge a]\) is 1 if \(c\ge a\) and 0 otherwise. For each \(g\), count how many references contain it at least once, at least twice, and so on. The cumulative sum through \(m\), divided by \(K\), is \(F_g(m)\).
Store \(F_g(0)=0\) and values through the largest reference-side count. Larger candidate counts use the last value because the overlap has saturated. An n-gram absent from all references contributes zero.
A table covering orders through 32 works for short candidates too. Orders absent from both the candidate and every reference contribute zero, so the table can be reused regardless of candidate length.
7.3. Replacing the mean with a median
A variant uses the median reference length to reduce the influence of unusually long references on the denominator:
\[
\frac{\bar M}{\max(L_t,\operatorname{median}_iL_i)}.
\]
The ratio of mean overlap to median length can exceed 1, so it no longer represents a matched fraction. Using the mean for both quantities preserves the interpretation of recall in the main definition.
For an even number of references, define the median as the average of the two central weighted lengths. Even when the median is positive, \(\bar M/\operatorname{median}_iL_i\) can exceed 1; when it is zero, that ratio is undefined. The final score still cannot exceed 1 whenever its denominator is positive, because \(\bar M\le L_t\) and the denominator is at least \(L_t\).
7.4. Check edge cases and reproduce the examples
Branch on the empty-input rules before division. For \(K=0\), return an error or a missing value. The following inputs exercise the distinction between empty strings and missing references, as well as the difference between the two forms. Decimal scores are rounded to three places.
Candidate
References
Mean-length version
Base form
"cat"
["cat"]
1
1
"a"
["b"]
0
0
""
[""]
1
1
""
["a"]
0
0
"a"
[""]
0
0
""
["", "a"]
0
0.5
"ca"
["cat"]
0.498
0.498
"a"
["a", "abcdefghij"]
0.134
0.528
"a"
[]
Undefined
Undefined
Also test literal ^ and $ characters in the input. Treating them as the dedicated boundary markers would incorrectly discard real text. The calculation and figure-generation script reproduces the numerical examples in Section 3 and the models in Section 6 using only the Python 3 standard library. Running it also regenerates the figures in both languages.
7.5. N-gram counts and processing cost
A string of length \(L\) contains \(O(NL)\) n-gram occurrences through order \(N\). Using all orders gives \(O(L^2)\) occurrences; fixing the cap at 32 limits this to \(O(32L)\). Reference preprocessing scales with the corresponding counts for each reference.
Runtime also depends on how n-grams are represented and hashed. With the cap fixed, both the number of orders and the length of each n-gram are bounded. An implementation with expected constant-time hash-table operations then has expected runtime proportional to the input length.
Copying and hashing all \(n\) characters of each extracted n-gram costs \(O(N^2L)\) time, reaching \(O(L^3)\) when all orders are used. With a fixed cap of 32 and expected constant-time dictionary operations, this is still expected \(O(L)\).
If \(N\) is treated as a variable, expected \(O(NL)\) generation and counting require a representation that supports constant-time updates per n-gram as well as the dictionary assumption. If hashes serve as identifiers, handle collisions so that different n-grams are not counted as equal.
8. Choosing evaluation conditions
When deciding whether and how to use charsim, consider the task:
For meaning, supplement charsim with human review or a separate semantic evaluation. Paraphrases can preserve meaning while sharing little wording with the references.
For word order, consider how much context the n-grams capture. Unigrams ignore order, and rearranging words can leave many matches at lower orders intact.
For long set phrases or source-code matches, check whether the cap of 32 and the \(1/n\) weights give long contiguous matches the emphasis the task needs.
For comparisons between candidates, keep the references and preprocessing fixed. When comparing languages or tasks, account for differences in those evaluation conditions.
9. Using charsim in an evaluation
charsim measures shared wording in sentences and short texts. Weighted sums retain credit for partial matches without smoothing, and taking the smaller of precision and recall penalizes both missing and extra material without a balance coefficient. The \(1/n\) weights reduce each high-order term's contribution and limit the reward for concentrated matches. The cap of 32 includes longer matches while bounding the processing cost; sensitivity to changing it depends on how much overlap and length the added orders contribute.
Choose how to combine references according to what a good answer should match. The mean-length version measures agreement across the set against a common length baseline and supports precomputation for repeated evaluation. The base form gives each pair's normalized score equal weight; the maximum accepts a match to any one reference. Report this choice and the preprocessing so that others can reproduce the scores.
10. References
The comparisons draw on the BLEU, chrF, and chrF++ papers and the sacreBLEU implementation cited above.
Two of my related papers study evaluation based on character n-grams[5, 6]. The translation-evaluation paper uses the same overlap-counting principle, with a different length correction.
[5] Kentaro Imajo, Masanori Hirano, Shuji Suzuki, Hiroaki Mikami, "pfgen-bench: A Text Generation Performance Evaluation Benchmark for Japanese Pretrained Models," Jxiv, 2024. An evaluation method for a Q&A benchmark. Its Fluency metric sums reference-answer frequencies for character n-grams present in the candidate. It handles occurrence counts differently from the clipped overlap \(\min(c_r,c_t)\) used here.
[6] Kentaro Imajo, Masanori Hirano, "A Study of Automatic Evaluation Methods for High-Precision Translation Models," Jxiv, 2026. An evaluation against many reference translations using binary features \(f_{w,i}\) that indicate whether an n-gram occurs at least \(i\) times. It corrects length with a separate factor based on the median reference character count, rather than the normalization by \(\max(L_t,\bar L_r)\) used here.
