Data and method

Token cost by language: the data behind the comparison

The Token Counter can tell you how much more or less a language costs than English. This page shows where that comes from: the sentences we started with, how every language was counted, what the numbers look like, and how the tool turns them into the figure you see next to your text. The formal write-up of the same measurement is a separate PDF.

Open the Token CounterFormal report (PDF)Results (CSV)

108

languages measured

28

writing systems

20

sentences per language

1.80×

median language vs English (o200k_base)

27

languages at 2× English or more (o200k_base)

1. The data we started from

We did not write the sentences or the translations. They come from a published benchmark.

FLORES-200

FLORES-200 is a benchmark from Meta AI's NLLB project. Its sentences were written in English and professionally translated into more than 200 languages, so line n says the same thing in every language. That is what makes the comparison fair: every language is counted on the same meaning.

We used its devtest split. It is released under CC BY-SA 4.0, so we credit it, say what we changed, and keep the sentence file under the same licence.

What we took from it

  • 20 sentences: devtest lines 10, 60, 110, 160, 210, 260, 310, 360, 410, 460, 510, 560, 610, 660, 710, 760, 810, 860, 910, 960, which is every 50th line starting at line 10. Fixed in advance, nothing picked by hand.
  • 108 languages, chosen by hand (see section 5), each with the same 20 lines.
  • For every language: its code, name, script, the sentences and the language-detection codes. Whitespace was trimmed; nothing else was altered.
  • Not used: the dataset's other columns (topic, domain) and its dev split.

2. One sentence, many languages

This is sentence 10 from the sample, in 14 of the 108 languages, with its exact token count under each encoding. It is the whole method in miniature.

One sentence in several languages with its characters and token counts for o200k_base and cl100k_base
LanguageSentenceCharacterso200k tokenscl100k tokens
EnglishLatinRing also settled a lawsuit with competing security company, the ADT Corporation.811515
GermanLatinRing hat außerdem eine Rechtsstreitigkeit mit der konkurrierenden Sicherheitsfirma ADT Corporation beigelegt.1092126
SpanishLatinRing también llegó a un acuerdo en el litigio con la compañía de seguridad de la competencia, ADT Corporation.1102327
RussianCyrillicКроме того, Ring урегулировала судебный процесс с конкурентами, охранной фирмой ADT Corporation.962540
ArabicArabicكما قامت رينج بتسوية دعوى قضائية مع شركة الأمن المنافسة، شركة ADT.662247
UrduArabicRing نے مسابق سیکیورٹی کمپنی ADT Corporation کے ساتھ اپنے قانونی دعوے کا تسویہ بھی کر لیا۔902665
HindiDevanagariरिंग ने अपनी प्रतिस्पर्धी कंपनी ADT कॉर्पोरेशन के साथ एक मुकदमा निपटाया है.752770
Chinese (Simplified)Han (Simplified)铃声 (Ring) 公司还与竞争对手 ADT 安保公司在一起官司中达成了庭外和解。412939
JapaneseHan + KanaRingはまた、競合するセキュリティ企業であるADTとの訴訟で和解しました。382634
KoreanHangulRing은 경쟁 보안회사인 ADT 사와의 소송 또한 합의를 보았다.372034
ThaiThaiริงยังได้ดำเนินการระงับข้อพิพาทกับบริษัทรักษาความปลอดภัย ADT Corporation ซึ่งเป็นคู่แข่งอีกด้วย953581
GeorgianGeorgianკომპანია Ring-მა ასევე მოაგვარა სარჩელი უსაფრთხოების კონკურენტ კომპანიასთან ADT.8024137
AmharicEthiopicሪንግ ከተፎካካሪ የደህንነት ኩባንያም ADT ኮርፖሬሽን፣ ጋር ክስ መስርቷል።4883112
SantaliOl Chikiᱨᱤᱝ ᱫᱚ ᱟᱨᱦᱚᱸ ᱦᱮᱯᱨᱟᱣ ᱥᱩᱨᱚᱠᱷᱭᱟ ᱠᱚᱢᱯᱟᱱᱤ, ᱮ.ᱰᱤ.ᱴᱤ ᱠᱚᱨᱯᱚᱨᱮᱥᱚᱱ ᱥᱟᱶ ᱢᱚᱠᱚᱨᱫᱚᱢᱟ ᱚᱝᱥᱟᱨ ᱥᱚᱞᱦᱮ ᱟᱠᱟᱫ-ᱟᱭI91239227

English takes 15 tokens here under o200k_base. The whole measurement repeats this for all 20 sentences and adds the counts up. Sentences are from FLORES-200 (CC BY-SA 4.0).

3. How every language was counted

Four steps, all exact for the two encodings. No model was asked anything; it is plain tokenizer counting.

  1. 1Count

    Each sentence is tokenized with gpt-tokenizer using o200k_base (GPT-4o) and cl100k_base (GPT-4 and GPT-3.5). Both encodings are published, so the counts are exact.

  2. 2Add up

    For each language we sum tokens, characters and words over its 20 sentences. Characters are UTF-16 code units. Words come from the JavaScript engine's word segmenter, because splitting on spaces would treat a Chinese, Japanese or Thai sentence as one word.

  3. 3Divide

    Tokens per character and tokens per word are the language's totals divided by its totals. These describe how finely the tokenizer cuts that language.

  4. 4Compare

    The ratio to English is the language's total tokens divided by English's total on the same 20 sentences, so English is exactly 1.00. English totals 457 tokens under o200k_base and 463 under cl100k_base.

4. What the numbers look like

A few views of the same results file. The full table is in the CSV and in the Token Counter's Compare languages section.

  • under 1.5×19
  • 1.5× to 2×61
  • 2× to 3×17
  • 3× to 5×3
  • 5× to 10×6
  • 10× and over1
Non-English languages by ratio to English, o200k_base.
  • under 1.5×0
  • 1.5× to 2×20
  • 2× to 3×40
  • 3× to 5×18
  • 5× to 10×20
  • 10× and over9
The same bands under cl100k_base.
  • Santali15.29×
  • Dzongkha9.49×
  • Tibetan8.98×
  • Lao8.78×
  • Tigrinya6.20×
Highest ratio to English, o200k_base.
  • Portuguese1.29×
  • Chinese (Simplified)1.29×
  • Indonesian1.30×
  • Dutch1.33×
  • Danish1.38×
Lowest ratio to English, o200k_base (English excluded).

In this sample the 54 other Latin-script languages have a median ratio of 1.69× under o200k_base, against 1.94× for the languages in other scripts. The script with the highest median is Ol Chiki (15.29×, 1 language). Several scripts hold only one or two languages, so group medians describe this sample, not the script in general.

5. How the Token Counter uses the data

The measurement is done once, ahead of time. In the browser the tool only looks things up.

From pasted text to a ratio

  1. Text of 20 characters or more triggers language detection, in your browser, with the franc library.
  2. If the best match is not clearly ahead of the runner-up, nothing is shown. Short or mixed text gets no guess.
  3. If the detected language is in this data, the panel shows the language's measured ratio to English, for both encodings, and how many sentences it comes from.
  4. Beside it, tokens per word and per character are measured on your text. The ratio is from the sample, not your text, and the panel says so.
  5. For any non-English language, the estimated models (the ones scaled from the GPT-4o count) are marked less reliable, because their fixed scale was set for English.

Which languages can be detected

Each language carries the detection code(s) franc uses for it. We keep a code only if franc returns it for at least 70% of that language's 20 sentences. That gives 79 of 108 languages detectable.

The other 29are in the comparison table but can't be recognised from pasted text, usually because franc confuses them with a close relative:

Slovenian, Croatian, Norwegian (Bokmål), Icelandic, Basque, Welsh, Irish, Maltese, Indonesian, Malay, Xhosa, Kinyarwanda, Māori, Samoan, Northern Kurdish, Serbian, Dari, Sindhi, Central Kurdish, Egyptian Arabic, Sanskrit, Maithili, Assamese, Odia, Dzongkha, Amharic, Tigrinya, Chinese (Traditional), Cantonese.

6. Choices we made, and what they limit

Where we used judgment, and what that means for how far the numbers go.

  • Languages were hand-picked. We started from the languages we were asked to cover, then added others to spread the writing systems and resource levels. It is not a random sample, and 55 of the 108 languages are Latin-script.
  • 20 sentences is small. It supports comparing languages on this text. It does not promise a ratio for yours; domain, numbers, names and code change the result.
  • Two encodings only.These are OpenAI's published tokenizers. Claude, Gemini and others use tokenizers we cannot run in a browser, so nothing here describes them.
  • Detection is a guess. Persian and Dari both lead to the Persian row, and some languages are read as a close relative, such as Assamese as Bengali. The language shown is the one detected, not a verified fact about your text.

7. Reproduce it and reuse it

Two scripts in the project repository regenerate everything on this page from FLORES-200:

npm run tokens:data    # select the sentences and languages
npm run tokens:report  # count them and write the results
npm run tokens:pdf     # rebuild the PDF report

The results CSV has one row per language with characters, words, tokens, tokens per character and word, and the ratio to English for both encodings. It holds counts only, no sentences.

Attribution

Sentences: FLORES-200, NLLB Team (Meta AI), CC BY-SA 4.0. Costa-jussà, M. R., Cross, J., Çelebi, O., et al. (2022). No Language Left Behind: Scaling Human-Centered Machine Translation. arXiv:2207.04672. Tokenizers: o200k_base and cl100k_base as implemented by gpt-tokenizer.