On a laptop’s integrated graphics, Dhīr 5B MoE decodes at 20.5 tokens per second. Dhīr 2B, a model with less than half the parameters, decodes at 18.6. The larger model is faster.
The reason is the mixture-of-experts (MoE) design of Dhīr 5B. Each of its 28 blocks holds twelve expert networks, and a router sends every token through only two of them, so decoding one token reads about 1.46 billion weights from memory, against all 2.03 billion of the dense Dhīr 2B. On edge devices, decode is limited by memory bandwidth, so fewer weights read means more tokens per second. This saves memory traffic, not memory: all twelve experts stay loaded.
This article explains the mechanism, shows measured speeds on edge devices from a Raspberry Pi to a mobile NPU and on consumer and data-centre GPUs, and sets out what each model costs in memory.
Running a model on the device in front of you answers three problems that cloud inference leaves open.
- Privacy boundaries. Confidential company code or patient records should not have to cross the internet.
- Unreliable connections. Field apps, factory-floor systems and rural kiosks cannot depend on a link that drops out.
- Perpetual cost. Paying per token every time a user asks a question adds up.
Both Dhīr models were built for that setting. Dhīr 2B is a dense model with 2.03 billion parameters. Dhīr 5B MoE has 5.09 billion parameters. Each of its 28 blocks holds twelve expert networks, and a router sends each token to two of them, so about 1.79 billion parameters are active for any one token. Neither needs a multi-thousand-dollar GPU rig.
What the model needs in memory
A model file is not the whole memory bill. Weights that take 1.21 GB need more than 1.21 GB to run. Think of the memory a model uses as a study desk with four things on it.
The model’s parameters. At 4 bits (Q4_0) they take 1.21 GB for Dhīr 2B and 2.95 GB for Dhīr 5B MoE. This part never grows.
As a conversation goes on, the model keeps notes on what came before. Dhīr 2B adds 115 kB per token (114,688 bytes) and Dhīr 5B MoE adds 72 kB (71,680 bytes), because it has fewer key–value heads (5 against 8). At the full 16,384-token context that is 1.88 GB and 1.17 GB.
The engine needs scratch space for its calculations. llama.cpp reserves about 0.3 GB at its default batch size, and in our measurements 0.1 to 0.15 GB of it was in use at peak.
The operating system and other apps need memory too, so a model should never take all of it.
Will it fit?
Choose a model, a precision, a context length and a device.
An estimate: weights (4-bit means Q4_0), plus a 16-bit KV cache at the per-token sizes above, plus 0.2 GB of buffers. The bar spans the whole device; the dashed line marks half of it. “Fits comfortably” means the total stays under half the device’s memory, which leaves room for the operating system and your own app.
We checked the estimate by filling the context completely with llama.cpp (Q4_0, flash attention on) and recording peak memory. On an x86 CPU, Dhīr 2B peaked at 1.82 GB at 4,096 tokens and 3.24 GB at 16,384, and Dhīr 5B MoE at 3.37 and 4.27 GB, within 0.1 GB of the estimate. On the mobile’s CPU, Dhīr 5B MoE needed the same, but Dhīr 2B needed about 0.2 GB more (2.04 and 3.47 GB), because llama.cpp’s Arm backend keeps a second copy of the embedding matrix that Dhīr 2B shares with its output layer. With flash attention off, memory grows faster with context: 4.00 and 5.25 GB at 16,384 tokens.
| Peak memory, GB (Q4_0) | 2,048 tokens | 4,096 | 8,192 | 16,384 |
|---|---|---|---|---|
| Dhīr 2B, estimate above | 1.64 | 1.88 | 2.35 | 3.29 |
| Dhīr 2B, x86 CPU | 1.58 | 1.82 | 2.29 | 3.24 |
| Dhīr 2B, mobile CPU | 1.81 | 2.04 | 2.52 | 3.47 |
| Dhīr 2B, mobile NPU | 2.06 | 2.30 | – | 3.74 |
| Dhīr 5B MoE, estimate above | 3.30 | 3.44 | 3.74 | 4.32 |
| Dhīr 5B MoE, x86 CPU | 3.22 | 3.37 | 3.67 | 4.27 |
| Dhīr 5B MoE, mobile CPU | 3.21 | 3.36 | 3.66 | 4.25 |
| Dhīr 5B MoE, mobile NPU, experts on the NPU | 3.55 | 3.70 | – | does not fit |
Measured with llama-batched-bench: one prompt of n − 16 tokens fills the context in 512-token batches, then 16 tokens are decoded at full depth. Weights are read into memory rather than memory-mapped, so none of the peak is reclaimable file cache. Peak memory is the process’s maximum resident set plus, on the NPU, the shared buffers mapped into it; on the mobile it agreed with the fall in available memory to within 0.15 GB (0.23 GB for Dhīr 2B at 16,384 tokens on the CPU). One run per point. GB is 109 bytes, MB 106 and kB 1,000.
In practice, Dhīr 2B at 4 bits fits comfortably in a 4 GB or 8 GB device for regular chats, with a context of 2,048 to 4,096 tokens, and Dhīr 5B MoE wants 8 GB. Mobiles need more room: on our 12 GB test mobile, Android and its apps held about 4 GB with the mobile idle, so our ZiqeX app asks for at least 8 GB. The 5B model has the smaller cache per token, which partly offsets its larger weights as contexts get long.
Why memory bandwidth sets the speed
People usually look at compute power when judging AI speed. On edge devices, decode is limited by memory bandwidth instead: how fast memory can feed numbers to the processor. Inference has two phases.
All prompt tokens are processed in parallel, so each weight fetched from memory serves many tokens. Prefill is compute-bound and much faster than decode: from a few tokens per second on the Raspberry Pi 4 to over a thousand on the Jetson Orin NX.
Tokens are generated one at a time, and every token reads the model’s active weights and the KV cache from memory. Decode is bound by memory bandwidth.
Why the 5B MoE model can decode faster than the 2B
Dhīr 5B MoE has about 2.5 times as many parameters as Dhīr 2B, yet it usually decodes faster. The reason is how many bytes each token reads from memory.
What one token reads from memory
Dhīr 2B is dense: every token uses every parameter. In Dhīr 5B MoE, each block holds twelve experts and a router picks two for each token. Generate a few tokens to see the routing change.
Dhīr 2B, dense
2.03 billion weights read per token
Dhīr 5B MoE, 2 of 12 experts
1.46 billion weights read per token
That is about 28% fewer weights to read per token. (Counting the input embedding, 1.79 billion parameters are active per token, but a token reads only its own row of that table.) The gain depends on the device: the 5B model decoded 10% faster on the laptop, 17% faster on the Jetson, 23% faster on the Raspberry Pi and 45% faster on the mobile’s CPU. It is not universal. In the technical report’s seven like-for-like comparisons the 5B model decoded faster in six, and on our test mobile it was faster on the CPU, on the GPU and, with its experts placed on the NPU, on the NPU. Its one slower result is Q8_0 on the laptop’s integrated GPU, 12.9 against 13.3 tokens per second.
The trade-off is memory. All twelve experts must be held in memory even though each token uses only two, so the 5B model needs 2.4 to 2.5 times the weight memory of the 2B, depending on the format.
Checking the explanation
If decode is limited by memory bandwidth, then decode speed multiplied by the bytes read per token should come out the same for both models on one device. We counted those bytes in our Q4_0 files. Dhīr 2B reads all 1,212 MB of its weights for every token, because its embedding matrix doubles as the output layer. Dhīr 5B MoE reads 822 MB: attention (258 MB), the output layer (189 MB), router and normalisation weights (4 MB), and two of the twelve experts in every block (372 MB of 2,230 MB). From its input embedding a token reads only its own row.
| Q4_0, mobile CPU | Weights in memory | Read per token | Decode (tok/s) | Bytes per second |
|---|---|---|---|---|
| Dhīr 2B | 1.21 GB | 1,212 MB | 10.7 | 13.0 GB/s |
| Dhīr 5B MoE | 2.95 GB | 822 MB | 15.5 | 12.7 GB/s |
The two models move data at the same rate, within 3%. Reading 32% fewer bytes predicts a 47% higher decode speed for Dhīr 5B MoE, and we measured 45%. The technical report makes the same calculation against the theoretical bandwidth of the laptop (51.2 GB/s) and the Jetson (102.4 GB/s): decode reaches 35 to 44% of the ceiling that bandwidth sets.
The advantage is largest for one request at a time. When eight requests are decoded together, different tokens pick different experts, so more of Dhīr 5B MoE’s weights are read at every step: on the Jetson, with eight 512-token prompts in Q4_0, the two models decode at the same rate (150.5 and 150.3 tokens per second). With longer contexts the smaller KV cache of Dhīr 5B MoE tips the balance back: with eight 4,096-token prompts it decoded at 104.7 tokens per second against 90.3.
Where it runs: measured speeds
We ran both models on a mobile, a laptop, an embedded board and a Raspberry Pi with llama.cpp, a 16-bit KV cache and one request at a time. Prefill (prompt processing) and decode (generation) are given in tokens per second.
Tokens per second by device
Choose what to measure. The mobile appears once for each processor.
| Device | Processor used | Model and format | Prefill (tok/s) | Decode (tok/s) |
|---|---|---|---|---|
| Motorola Edge 70 Fusion, 12 GB (Snapdragon 7s Gen 4) | Hexagon NPU | Dhīr 2B Q4_0 | 145.2 | 10.6 |
| Dhīr 5B MoE Q4_0, experts on the NPUa | 44.7 | 15.2 | ||
| CPU, 4 performance cores | Dhīr 2B Q4_0 | 106.6 | 10.7 | |
| Dhīr 5B MoE Q4_0 | 79.9 | 15.5 | ||
| Adreno 810 GPU, OpenCL | Dhīr 2B Q4_0b | 73.5 | 9.5 | |
| Dhīr 5B MoE Q4_0 | 51.2 | 10.5 | ||
| Core i5-1245U laptop | Iris Xe integrated GPU, Vulkan | Dhīr 2B Q4_0 | 385.0 | 18.6 |
| Dhīr 5B MoE Q4_0 | 370.8 | 20.5 | ||
| Jetson Orin NX, 16 GB | Integrated Ampere GPU | Dhīr 2B Q4_0 | 1,327.2 | 34.1 |
| Dhīr 5B MoE Q4_0 | 1,295.4 | 39.9 | ||
| Raspberry Pi 4, 4 GB | Cortex-A72 CPU | Dhīr 2B Q4_0 | 4.5 | 2.3 |
| Dhīr 5B MoE Q4_0 | 5.8 | 2.8 |
All these devices ran Q4_0, a 4-bit format. Prefill is measured on a 512-token prompt (128 tokens on the Raspberry Pi 4). Decode is measured over 128 generated tokens, from an empty context on the mobile and after a 512-token context on the laptop and the Jetson; on the Raspberry Pi, over 64 tokens after a 128-token context. After a 512-token context the mobile CPU decodes at 9.5 tokens per second (Dhīr 2B) and 13.7 (Dhīr 5B MoE). The mobile figures are the mean of 10 runs on our own Q4_0 conversions (standard deviation at most 3.6 tokens per second for prefill and 0.12 for decode), with llama.cpp built from pull request 29740, which adds integer matrix kernels for this mobile’s NPU. a From a separate, cooler session, in which the CPU gave 87.3 and 16.6 tokens per second. b From an earlier session, on an earlier Dhīr 2B checkpoint with an identical file layout; speed does not depend on the values of the weights. Compare models within a device, not across devices: the llama.cpp builds and settings differ.
What these speeds feel like:
- Mobile. Dhīr 2B prefills a 512-token prompt in about 3.5 seconds on the NPU and 4.8 seconds on the CPU, and Dhīr 5B MoE in about 6.4 seconds on the CPU. Decode runs at 11 tokens per second for Dhīr 2B and 15 for Dhīr 5B MoE, faster than most people read.
- Laptop. Decode runs at 18 to 20 tokens per second on integrated graphics alone, with two CPU threads and turbo boost off.
- Jetson Orin NX. A 512-token prompt is prefilled in 0.39 seconds, and Dhīr 2B decodes at 34 tokens per second.
- Raspberry Pi 4. A 128-token prompt takes 22 to 29 seconds to prefill, and decode runs at 2 to 3 tokens per second. That suits background work such as triage, sensor logs and queued jobs, not live chat.
Mobile: the NPU has the fastest prefill and uses the least energy
On the mobile’s Hexagon NPU, Dhīr 2B prefills at 145.2 tokens per second, 1.4 times its speed on the CPU and twice its speed on the GPU, and decodes at 10.6 tokens per second, the same as on the CPU. An NPU adds compute, and prefill is compute-bound. Decode is bound by memory bandwidth, which the NPU shares with the CPU. The speed-up is specific to Q4_0, the format that the NPU’s integer matrix kernels accelerate: the same model in Q4_K_M prefilled at only 24.3 tokens per second on the NPU and 52.1 on the CPU.
The NPU also does the work on less energy. Measured from the battery with the mobile unplugged, Dhīr 2B used 3.3 times less energy per token in prefill on the NPU than on the CPU, and 24% less per token in decode. The GPU used the most.
| Dhīr 2B Q4_0 on the mobile | Prefill energy (mJ per token) | Decode energy (mJ per token) | Mobile power while decoding |
|---|---|---|---|
| Hexagon NPU | 8.4 | 171 | 2.36 W |
| CPU, 4 performance cores | 28.0 | 224 | 3.04 W |
| Adreno 810 GPU | 32.6 | 265 | 2.95 W |
Energy above idle, from the battery fuel gauge (current × voltage) with the mobile unplugged; the idle draw of 0.44 W is subtracted. One session, 5 repetitions per cell, on an earlier Dhīr 2B checkpoint with the same file layout. Power is the whole mobile’s, idle included.
Dhīr 5B MoE is a different case. llama.cpp’s NPU backend has no kernel yet for the expert layers on a batch of prompt tokens, so those layers fall back to the CPU, and with its experts on the NPU the model prefills at about half the CPU’s speed (44.7 against 87.3 tokens per second in the same session). Decode is where the NPU pays off. We placed the experts on the NPU, let the CPU threads sleep between steps, and ran simulated chats through our app: the same 156 turns, in six conversation sets, on each processor. Dhīr 5B MoE used 0.204 J per generated token, prefill included, against 0.276 J on the CPU, 26% less, and the mobile drew 2.16 W instead of 3.10 W. It decoded 8% slower (12.5 against 13.6 tokens per second) and used less energy than the CPU in each of the six conversation sets. These runs were made on the charger.
The mobile’s Adreno GPU (OpenCL) was slower than its CPU for both models.
The NPU also costs memory: at 4,096 tokens, 2.30 GB for Dhīr 2B and 3.70 GB for Dhīr 5B MoE, 0.26 to 0.34 GB more than on the CPU. Everything placed on it must fit its 32-bit address space, so Dhīr 5B MoE with its experts there cannot hold a 16,384-token cache.
Mobile CPU: what a longer prompt costs
On the mobile’s CPU, prefill speed falls as the context fills, and decode slows sharply: every new token reads the whole cache and computes attention over it. At 8,192 tokens the cache nearly doubles the bytes Dhīr 2B reads per token, yet its decode falls almost fourfold, so on this CPU the attention computation itself becomes the bottleneck. Dhīr 5B MoE, with the smaller cache, holds up better.
| Mobile CPU, Q4_0: context (tokens) | Dhīr 2B: prefill time | Dhīr 2B: decode at that depth (tok/s) | Dhīr 5B MoE: prefill time | Dhīr 5B MoE: decode at that depth (tok/s) |
|---|---|---|---|---|
| 2,048 | 25 s | 6.3 | 38 s | 10.4 |
| 4,096 | 64 s | 4.6 | 1.7 min | 7.9 |
| 8,192 | 3.1 min | 2.8 | 5.4 min | 4.4 |
| 16,384 | 11.8 min | 1.6 | 19.1 min | 2.4 |
Motorola Edge 70 Fusion, CPU only (four threads on the four fastest cores), Q4_0, flash attention on, llama-batched-bench: the prompt is prefilled in 512-token batches, then 16 tokens are decoded at full depth. These are the runs of the memory table, one run per point. Prefill time for 16,384 tokens: 706 s for Dhīr 2B (23.2 tokens per second on average) and 1,146 s for Dhīr 5B MoE (14.3).
The NPU has a limit of its own. It prefilled Dhīr 2B’s first nine 512-token batches (4,608 tokens) in 3.5 to 7.4 seconds each, faster than the CPU. From the tenth batch on, each batch took more than a minute, and longer with every batch (73 seconds for the tenth, 172 seconds for the 22nd), so prefilling a 16,384-token context did not finish within our 30-minute limit. We have not yet traced the cause. The 4,096-token context of our app stays below it.
Jetson Orin NX GPU: what a longer prompt costs
Prefill speed stays between 1,200 and 1,550 tokens per second as prompts grow, so prefill time grows roughly in step with prompt length: Dhīr 2B takes 0.39 seconds for 512 tokens and 6.7 seconds for 8,192. Decode slows more gently, because a longer context means more KV cache to read for every token, and Dhīr 5B MoE, with the smaller cache, slows less: by 8,192 tokens its decode has fallen 19%, Dhīr 2B’s 25%.
| Jetson GPU, Q4_0: prompt (tokens) | Dhīr 2B prefill (tok/s) | Dhīr 2B decode (tok/s) | Dhīr 5B MoE prefill (tok/s) | Dhīr 5B MoE decode (tok/s) |
|---|---|---|---|---|
| 512 | 1,327 | 34.1 | 1,295 | 39.9 |
| 2,048 | 1,326 | 32.1 | 1,533 | 38.1 |
| 4,096 | 1,303 | 29.6 | 1,494 | 35.9 |
| 8,192 | 1,218 | 25.6 | 1,370 | 32.2 |
Jetson Orin NX integrated GPU (CUDA), llama-batched-bench with one sequence: the prompt is prefilled, then 128 tokens are decoded after it.
Serving many users: consumer and data-centre GPUs
On a GPU serving many people at once, requests are batched, and which model delivers more depends on prompt length. We measured both models with vLLM in BF16 on three GPUs, at prompt lengths from 128 to 16,384 tokens. For each we recorded the most concurrent requests that still meet an interactive target (99th-percentile time to first token within 500 ms, 99th-percentile time per output token within 50 ms, no failed request) and the highest output throughput at any concurrency.
- Short prompts favour Dhīr 2B. With 128-token prompts it delivers 14 to 26% more output tokens per second at the same concurrency on every GPU. Decode dominates this workload, and the tokens of many batched requests select different experts, so most of Dhīr 5B MoE’s weights are read at every step.
- Longer prompts favour Dhīr 5B MoE. With prompts of 1,024 tokens or more, its maximum throughput is 15 to 27% higher on all three GPUs, and on the L4 and the RTX PRO 6000 it meets the latency target with up to twice as many concurrent requests. We attribute this to two properties: prefill is compute-bound, and each token’s matrix multiplications use 28% fewer weights; and its KV cache is 37.5% smaller per token, so attention reads less memory and more requests fit.
The crossover lies between 256 and 1,024 tokens, depending on the GPU.
| GPU (vLLM, BF16) | Prompt (tokens) | Dhīr 2B within target (requests · tok/s) | Dhīr 5B MoE within target (requests · tok/s) | Dhīr 2B maximum (tok/s) | Dhīr 5B MoE maximum (tok/s) |
|---|---|---|---|---|---|
| NVIDIA L4, 24 GB | 128 | 32 · 1,309 | 32 · 1,096 | 2,904 | 2,526 |
| 1,024 | 4 · 188 | 8 · 318 | 707 | 814 | |
| 4,096 | 2 · 84 | 2 · 102 | 191 | 241 | |
| RTX PRO 6000 Blackwell Max-Q, 96 GB | 128 | 128 · 14,685 | 128 · 12,894 | 14,685 | 12,894 |
| 1,024 | 16 · 2,397 | 32 · 3,202 | 4,140 | 4,879 | |
| 4,096 | 4 · 625 | 8 · 847 | 1,098 | 1,395 | |
| DGX Spark (GB10), 120 GB shared | 128 | 32 · 1,248 | 32 · 992 | 2,900 | 2,364 |
| 1,024 | 8 · 296 | 8 · 287 | 748 | 872 | |
| 4,096 | 2 · 77 | 2 · 93 | 205 | 254 |
Output tokens per second with 128 generated tokens per request, vLLM 0.31.0 (vllm bench), BF16 weights. “Within target”: the highest tested concurrency that meets the target above, and the throughput there. “Maximum”: the highest output throughput measured at any concurrency up to 128, usually outside the target. Selected rows from a grid of eight prompt lengths; some appear in the Dhīr technical report (Table 41).
On a consumer GeForce RTX 3050 with 8 GB, Dhīr 2B in BF16 meets the target with up to eight requests of 128 tokens or four of 512 tokens, and reaches 1,357 tokens per second when saturated with 64. Dhīr 5B MoE in BF16 does not fit: its weights alone take 10.19 GB, so on an 8 GB card it needs 4-bit weights.
The cache difference grows with context. At 72 kB per token against 115 kB, the same cache memory holds 1.6 times as many tokens for Dhīr 5B MoE, which matters on a large GPU where the cache, not the weights, fills memory.
Quantization: how a 10.2 GB model becomes 2.95 GB
Dhīr is trained and released in BF16, 16 bits per weight. Quantization stores the weights at lower precision. llama.cpp’s Q8_0 and Q4_0 formats split each weight matrix into blocks of 32 values and store every block as 8-bit or 4-bit integers with one 16-bit scale, 8.5 or 4.5 bits per weight; at inference the integers are multiplied by the scale to approximate the original weights. At 4 bits (Q4_0) the weights shrink by 3.4 to 3.5 times, and decode, which is bound by memory bandwidth, roughly doubles in speed on both the laptop and the Jetson.
Dhīr is released as BF16 weights. The 8-bit and 4-bit variants in this section (the llama.cpp formats Q8_0, Q4_0 and, in the fidelity table, Q4_K_M) are the ones we evaluated; the quantized files are not part of the release.
Weight memory and decode speed by precision
Decode on the laptop’s Intel Iris Xe integrated GPU (Vulkan), llama.cpp, one request. Choose a model.
Full precision (BF16) against 8 and 4 bits
Both models were also run unquantized, from their released BF16 weights. The technical report measured all three formats on the laptop and the Jetson. Dhīr 5B MoE decodes faster than Dhīr 2B in BF16 too, on both devices. Prefill gains from quantization on the laptop but not on the Jetson, where Dhīr 2B prefilled faster in BF16 than in Q4_0.
| Device | Format | Dhīr 2B prefill (tok/s) | Dhīr 2B decode (tok/s) | Dhīr 5B MoE prefill (tok/s) | Dhīr 5B MoE decode (tok/s) |
|---|---|---|---|---|---|
| Laptop: Intel Iris Xe integrated GPU, Vulkan | BF16 | 181.9 | 9.3 | 200.9 | 10.4 |
| Q8_0 | 282.7 | 13.3 | 333.4 | 12.9 | |
| Q4_0 | 385.0 | 18.6 | 370.8 | 20.5 | |
| Jetson Orin NX: integrated Ampere GPU, CUDA | BF16 | 1,447.0 | 16.4 | 1,303.3 | 20.9 |
| Q8_0 | 1,337.1 | 25.5 | 1,291.0 | 26.7 | |
| Q4_0 | 1,327.2 | 34.1 | 1,295.4 | 39.9 |
llama.cpp, one request, 16-bit KV cache: prefill on a 512-token prompt, then 128 tokens decoded after a 512-token context (laptop: llama-bench; Jetson: llama-batched-bench). The technical report (Table 39) quotes Dhīr 5B MoE in BF16 on the Jetson from a 1,024-token prompt (1,525.8 and 20.8). The GPU serving results above (vLLM) are also BF16.
The decode speed-up, about 2 times, is smaller than the size reduction, 3.4 to 3.5 times, because some work does not shrink with the weights: the KV cache and activations stay 16-bit, and the integers must be unpacked before use. With the embedding matrix kept at 6 bits, the Q4_0 files average 4.8 bits per weight for Dhīr 2B and 4.6 for Dhīr 5B MoE.
What rounding costs
We compared each quantized Dhīr 2B file with the BF16 model token by token, on text that was not used to calibrate the quantization: English (WikiText-2), the ten Indian languages of FLORES-200, code and mathematics. For every position we measured the Kullback–Leibler divergence between the two models’ next-token distributions (lower is better) and whether both rank the same token first.
| Dhīr 2B against BF16 | Q8_0 | Q4_K_M | Q4_0 |
|---|---|---|---|
| English, WikiText-2 (265K tokens) | 0.0011 · 98.0% | 0.0322 · 90.4% | 0.0612 · 86.9% |
| Ten Indian languages, FLORES-200 (746K tokens) | 0.0013 · 97.4% | 0.0280 · 89.3% | 0.0586 · 84.7% |
| Code (118K tokens) | 0.0006 · 99.0% | 0.0193 · 94.9% | 0.0363 · 93.2% |
| Mathematics (298K tokens) | 0.0007 · 98.8% | 0.0231 · 94.3% | 0.0442 · 92.0% |
| Perplexity change, the four sets | +0.0 to +0.1% | +1.4 to +2.9% | +2.6 to +4.3% |
Each cell: mean KL divergence in nats · share of positions where both models rank the same token first. Windows of 1,024 tokens advance by 512, so every scored token has at least 512 tokens of context. 95% confidence intervals, from a block bootstrap over consecutive windows, lie within 8% of each mean. Q4_0 and Q4_K_M were calibrated with an importance matrix computed on chat conversations. The Dhīr 2B checkpoint is the one of our mobile measurements. The technical report gives WikiText-2 perplexity for both models (Section 8.4).
Q8_0 is close to lossless. Q4_K_M diverges about half as much as Q4_0, but on our test mobile it is much slower: Dhīr 2B in Q4_K_M prefilled at 52.1 tokens per second on the CPU and 24.3 on the NPU, against 145.2 for Q4_0 on the NPU. Our app therefore ships Q4_0. Indian-language text loses about as much as English.
Which model to run
Choose it for 4 GB devices, the smallest download, serving many short requests on one GPU, or work in Indian languages, where it is first in its class on 13 of 15 benchmarks in the technical report.
Choose it for devices with 8 GB or more when you want faster decode for one user at a time, for serving long prompts to many users, or for using tool results, code, mathematics or English knowledge, where it leads Dhīr 2B in the technical report.
Running it yourself
Get the model files from the AIKosh listings. Check the version and available formats:
The release provides the BF16 weights in Hugging Face Transformers format. To run them with llama.cpp, convert them to a GGUF file, optionally quantize it to 4 bits, build llama.cpp for your target backend and point the command below to the exact filename you produced. The example is for a Vulkan-capable laptop; it is not a universal command for mobiles, NPUs or every operating system.
Dhīr uses its own tokenizer, which upstream llama.cpp does not recognise yet: as of its source of 28 October 2026, conversion stops with “BPE pre-tokenizer was not recognized”. Use a llama.cpp that includes Dhīr tokenizer support. [TBD: link to the patch, fork or upstream pull request.]
On a Linux laptop
Build llama.cpp with Vulkan support, then start a terminal chat. This shell example is for Linux; Windows and macOS require the corresponding build and executable paths:
# 1. Clone and build with GPU (Vulkan) support
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && cmake -B build -DGGML_VULKAN=ON && cmake --build build --config Release
# 2. Convert the downloaded weights to GGUF, then quantize to 4 bits (optional)
pip install -r requirements.txt
python convert_hf_to_gguf.py ./models/dhir-2b --outtype bf16 --outfile ./models/dhir-2b-bf16.gguf
./build/bin/llama-quantize ./models/dhir-2b-bf16.gguf ./models/dhir-2b-q4_0.gguf Q4_0
# 3. Start an interactive terminal chat
./build/bin/llama-cli \
-m ./models/dhir-2b-q4_0.gguf \
-cnv \
-t 2 \
-ngl 99 \
-c 2048On a mobile
You do not need to build anything. ZiqeX is ZenteiQ’s app for running compact Dhīr models on a mobile, on its CPU or NPU. It needs a mobile with at least 8 GB of memory.
What to keep in mind
Context limits
Set a limit that fits the job, such as 2,048 or 4,096 tokens. Long prompts cost time and memory, as the tables above show. On our test mobile’s CPU, prefilling a 16,384-token context took 12 minutes for Dhīr 2B and 19 minutes for Dhīr 5B MoE.
Heat and battery
The speeds here come from short benchmark runs. Mobiles and thin laptops use passive cooling and will slow down under hours of continuous load, so validate sustained speed, memory and battery use on your target device. On our test mobile, Dhīr 5B MoE prefilled at 79.9 tokens per second and decoded at 15.5 with the mobile warm (battery at 37 to 41 °C), against 87.3 and 16.6 in a cooler session.
Facts
Compact models are weak at recalling facts without a source. For current or consequential information, pair the model with retrieval from authoritative documents.
Summary
- Decode speed on edge devices follows memory traffic, so a mixture-of-experts model that reads fewer parameters per token can decode faster than a smaller dense one, for one user at a time. Serving many users, Dhīr 2B delivers more with short prompts, and Dhīr 5B MoE more with prompts of 1,024 tokens or longer.
- Mixture-of-experts saves memory traffic, not memory: all experts stay loaded.
- On the mobile, the NPU gives Dhīr 2B the fastest prefill and the lowest energy per token. For Dhīr 5B MoE it saves energy in decode, but prefills at half the CPU’s speed.
- Memory is weights plus a cache that grows with context. Plan for both.
- 4-bit weights take under a third of the BF16 memory and, on the laptop and the Jetson, decode about twice as fast. On the devices we tested, both models run offline at a usable speed on a mobile, a laptop and an embedded board.
How we measured
Every figure above comes from one of three sources: our own runs on the mobile and an x86 desktop, the benchmark runs behind Section 8 of the Dhīr technical report (laptop, Jetson, Raspberry Pi and GPU serving), or the model files themselves (weight sizes, bytes per token). Speed and memory depend on a model’s architecture and file layout, not on the values of its weights, so they hold for any checkpoint of each model.
| Device | Processors and memory | Backend | llama.cpp |
|---|---|---|---|
| Motorola Edge 70 Fusion, Android 16 | Snapdragon 7s Gen 4 (SM7635): eight Arm cores (4 × 1.80, 3 × 2.40, 1 × 2.71 GHz), Adreno 810 GPU, Hexagon v73 NPU; 12 GB | CPU, four threads on the four fastest cores; Hexagon NPU; OpenCL GPU | pull request 29740 (head 58a71bc) with our Dhīr tokenizer patches |
| x86 desktop, memory only | Core i9-14900K | CPU | commit f5222f2 with the same patches |
| Laptop, Ubuntu 26.04 | Core i5-1245U with Iris Xe graphics; 16 GB dual-channel DDR4-3200; performance power profile, turbo boost off | Vulkan 1.4.341; two CPU threads | build 11312 (0c1e570) |
| Jetson Orin NX developer kit, Ubuntu 22.04 | 8 Cortex-A78AE cores and an Ampere GPU; 16 GB LPDDR5 at 6,400 MT/s, 102.4 GB/s; MAXN power mode | CUDA; two CPU threads | build 11474 (b9acf13) |
| Raspberry Pi 4 Model B, Ubuntu 22.04 | 4 Cortex-A72 cores at 1.5 GHz; 4 GB LPDDR4 | CPU, four threads | build 11312 (0c1e570) |
- Speed. One request and a 16-bit KV cache throughout. Mobile, laptop and Raspberry Pi:
llama-bench; Jetson:llama-batched-bench. Prefill on a 512-token prompt (Raspberry Pi: 128) and decode of 128 tokens (Raspberry Pi: 64), after a 512-token context on the laptop and the Jetson (Raspberry Pi: 128) and from an empty context on the mobile. On the mobile: 10 runs per figure, flash attention onllama.cpp’s automatic setting, and a pause of up to 10 minutes before each block for the battery to cool below 32 °C. In the main session the mobile still started each block at 37 to 41 °C. - Memory.
llama-batched-benchfills the context completely, with weights read rather than memory-mapped and flash attention on. On the mobile we sampled the process’s resident memory, its mapped NPU buffers and the system’s available memory every second. - Energy. Dhīr 2B: battery fuel gauge with the mobile unplugged, one session, 5 repetitions per cell, idle draw subtracted. Dhīr 5B MoE: whole-mobile energy through our app on the charger, the same 156 simulated chat turns in six conversation sets on each processor.
- Fidelity. Next-token distributions of each quantized file against BF16 over 1,024-token windows, with confidence intervals from a moving-block bootstrap.
- Serving. vLLM 0.31.0 (
vllm bench, PyTorch 2.13.0) with BF16 weights, 128 generated tokens per request, prompts of 128 to 16,384 tokens and up to 128 concurrent requests, on an NVIDIA L4 (CUDA 12.9), an RTX PRO 6000 Blackwell Max-Q Workstation Edition (CUDA 13.2), a DGX Spark (CUDA 13.0) and a GeForce RTX 3050 (CUDA 13.0). - Bytes per token. Summed from the tensor sizes in each GGUF file: every tensor of Dhīr 2B; for Dhīr 5B MoE, everything except the input embedding plus two twelfths of the expert tensors.
To repeat the mobile measurements (CPU shown; for the NPU replace -ngl 0 -dev none with -ngl 99 -dev HTP0, and for Dhīr 5B MoE with its experts on the NPU add -ot exps=HTP0 --poll 0):
# speed: 10 runs, 512-token prompt, 128 generated tokens
llama-bench -m dhir-2b-q4_0.gguf -ngl 0 -dev none -t 4 -C 0xF0 --cpu-strict 1 -p 512 -n 128 -r 10
# peak memory with a full 4,096-token context
toybox time -v llama-batched-bench -m dhir-2b-q4_0.gguf -ngl 0 -dev none -t 4 -C 0xF0 --cpu-strict 1 \
-fa on -lm none -c 4096 -b 512 -ub 512 -npp 4080 -ntg 16 -npl 1