Fewer tokens for the same text, so more text for every dollar of compute. Every model reads text as tokens, and every token costs compute, context and money. We measured how many tokens twelve tokenizers spend on the same Indian-language, English, mathematics and code text, including Dhīr, the tokenizer behind ZenteiQ's Dhīr models.
What a tokenizer does
A language model never sees letters. A tokenizer first cuts text into pieces from a fixed vocabulary, and the model reads one piece at a time. A vocabulary learned mostly from English text has few entries for Indian scripts, so it breaks Hindi or Odia words into small pieces, sometimes into single bytes. The same sentence then becomes a much longer sequence of tokens, and the model pays for every one of them.
That matters for three reasons. Longer sequences cost more compute to train and serve. They fill the context window sooner. And API prices are quoted per token, so people writing in Indian languages pay more for the same content.
Dhīr is a byte-level tokenizer. Its starting alphabet is the 256 possible byte values, not a list of characters, and every vocabulary entry is a sequence of bytes learned from data. Text is first turned into UTF-8, where an English letter is one byte and a Devanagari, Tamil or Odia letter is three. Training merges frequent byte sequences into single entries, so a common Hindi word such as भाषा (12 bytes, 13 with the space before it) becomes one token in running text. When a character has no entry of its own, the tokenizer falls back to its individual bytes, and one letter costs up to three tokens.
That is why the explorer below works with bytes, not characters. Each token is mapped back to the exact bytes it stands for and decoded as UTF-8. When a tokenizer has to split one character across several byte tokens, the explorer joins those tokens back into the character and draws it in a dashed box with the number of tokens it took: த3 tokens. A vowel sign that ends up in a token of its own is drawn on a dotted circle, ◌ு, as Unicode charts do. Hover over any token to see its ID and bytes. For every tokenizer, the bytes of its tokens join back into exactly the input text; we checked this for each sample.
This is one example sentence from IN22-Gen, shown in six languages. Single sentences vary. Across all 1,024 IN22-Gen sentences, Dhīr needs strictly fewer tokens than all seven other tokenizers in the explorer on 72% of English sentences, 54% of Gujarati, 53% of Hindi, 39% of Telugu, 34% of Marathi and 34% of Tamil; the rest are ties or sentences where another tokenizer needs fewer. That is why the rest of this note reports whole benchmarks, not single sentences.
Code is the clearest case. Dhīr needs the fewest tokens on the short Python, Rust and C/C++ samples above, which were written for this page and carry a lot of English in comments and names. On the benchmark's real source files the result reverses: Llama 3.2 needs fewer tokens than Dhīr on Python (7.6% fewer), Rust (8.8%), C++ (12.2%) and C (13.3%).
How we measured
All numbers come from BrahmAI Tokenizer Eval, our open-source toolkit, at version 1.0.0. It runs every tokenizer on the same 184 corpora and pins each one by the SHA-256 of the exact file it loads.
- Tokens per 1,000 bytes. Tokens divided by the UTF-8 size of the text. Lower is better. Bytes are a fair unit across scripts; words are not, because languages split words differently.
- Paired ratio. Another tokenizer's token count divided by Dhīr's on the same items, with a 95% bootstrap interval. A ratio of 1.21 means it needs 21% more tokens.
The corpora are public benchmarks: IN22-Gen and FLORES-200 for languages, arXiv and PubMed abstracts, arXiv LaTeX source, MATH and GSM8K, and code in nine programming languages.
Dhīr is the most compact of the twelve tokenizers on English, scientific prose, arXiv LaTeX and MATH: on scientific abstracts it needs 187 tokens per 1,000 bytes against 205 for BLOOM, the next best, and on standalone Wikipedia formulas it is level with BLOOM (365 and 363). On the twelve Indian languages of its corpus it spends 121 tokens per 1,000 bytes, and only two tokenizers spend fewer, BLOOM (106) and Sarvam-30B (118), both with vocabularies about twice the size of Dhīr's (250,680 and 262,144 entries against 131,072). Code is where Dhīr is weakest: on our code corpora Llama 3.2 needs 12.2% fewer tokens than Dhīr, and o200k 11.8% fewer.
Language by language
Pooled numbers hide a lot. The chart below shows all 22 scheduled languages of India on IN22-Gen. Choose a tokenizer to set beside Dhīr.
A closer look at Hindi
Hindi is the most widely spoken language in the set, so we check it on two independent benchmarks: IN22-Gen, 1,024 sentences of general Hindi text, and FLORES-200, 1,012 sentences translated from English web articles. If the result held on only one of them, it could be a quirk of that dataset.
The easiest way to see the difference is in words. A 100-word Hindi paragraph costs Dhīr 141 tokens. BLOOM needs about the same (142), Sarvam-30B and Gemma 4 need 149, Param-2 needs 152, Gnani Evon v3.3 and OpenAI's o200k need 172 and 176, and Llama 3.2 needs 283, twice as many. Fewer tokens per word also means more text in the same context window: 4,096 tokens hold about 2,900 Hindi words with Dhīr and about 1,450 with Llama 3.2. The order of the tokenizers is the same on FLORES-200.
Show the numbers
What fewer tokens buy
Training and serving costs scale with the number of tokens processed. If two models of the same size charge the same per token, the one whose tokenizer needs fewer tokens processes the same text for less. On the twelve Indian languages of Dhīr's corpus, the same text takes:
- 17.5% fewer tokens with Dhīr than with Gemma 4,
- 28.7% fewer than with OpenAI's o200k,
- 76.0% fewer than with Llama 3.2.
On English, Dhīr needs 11.2% fewer tokens than o200k, and on MATH 23.6% fewer. The same saving applies to the context window: a 4,096-token window holds about 33,800 bytes of Indian-language text with Dhīr, against 27,800 with Gemma 4.
Further reading
- BrahmAI Tokenizer Eval, ZenteiQ's open-source toolkit used for every number here.
- Tokenization in Transformers v5, Hugging Face. Normalisation, pre-tokenisation and tokenizer architectures.
- Summary of the tokenizers, Hugging Face. BPE, WordPiece, Unigram and SentencePiece compared.
- IndicTrans2, AI4Bharat. Translation across the 22 scheduled languages, and the source of the IN22 benchmarks.
Data: BrahmAI Tokenizer Eval v1.0.0, full suite, 184 corpora, 232,501 items per tokenizer. Every tokenizer is pinned by commit and the SHA-256 of its tokenizer file. The six language samples in the explorer are item 551 of IN22-Gen (AI4Bharat, CC BY 4.0); the scientific, mixed, Python, Rust and C/C++ samples were written for this page.