After supervised fine-tuning, we used reinforcement learning to shape how Dhīr-2B and Dhīr-5B MoE respond: following instructions, using tools, answering in Indian languages and handling safety-sensitive requests. This is what we built, what we saw, and what we would tell anyone trying the same.
Supervised fine-tuning teaches a model by showing it good answers. Reinforcement learning, or RL, asks the model to answer for itself, scores what it wrote, and nudges it toward the answers that scored better.
We trained Dhīr-2B with GRPO and Dhīr-5B MoE with GSPO, using MaxText and Tunix on Google Cloud TPUs. Along the way, we learned as much about data, feedback and evaluation as we did about running the training itself.
Building the reinforcement-learning loop
Every RL step is a small production line. Prompts go in, the model writes answers, the answers are scored, and the model is updated. Then the updated model writes the next batch. Running RL well means running all of that as one system, not as separate jobs. Step through it below.
One step of training
Each step took about three minutes. Choose a stage.
Google’s JAX training framework holds the model being trained and applies each update.
A fast generation engine, running on TPU, produces the answers from the latest weights.
Tunix runs the RL algorithm, and the Pathways runtime lets one program drive both halves of the slice.
Learning by comparison
GRPO and GSPO share one idea. Instead of asking “was this answer good?”, they ask “was this answer better than the model’s other attempts at the same prompt?” Each prompt gets eight answers. Answers that beat the group average are reinforced; answers below it are discouraged.
The two methods differ in one detail: how they measure how far an update moves the model. GRPO looks at it word by word, while GSPO looks at the whole answer at once. That difference matters in some setups; below we explain why it did not in ours.
Eight answers to one prompt
Bars show each answer’s score. Above the line, the model is pushed toward that answer; below it, away. Choose a group.
The second and third cases are the reason prompt selection matters. When all eight answers score the same, there is nothing to compare, and the prompt teaches the model nothing. So we screened prompts for ones the model could sometimes solve but not always. As training went on and the model improved, more prompts slipped into the “all correct” case: by the final pass, nearly a third of groups carried no signal at all.
Making feedback useful
The score is the only thing the model learns from, so it has to reward what we actually want. We combined a judge model, which rates each answer from 1 to 10 against a reference answer or checklist, with programmatic checks wherever an answer can be verified directly: whether instructions were followed, whether a tool call is valid, whether an answer uses the information a tool returned.
For a multilingual model, a good answer has to do more than contain the right information. It has to come back in the language it was asked in. We made that an explicit part of the score, alongside safety: an answer in the wrong language, or an unsafe one, is capped no matter how good it is otherwise.
How an answer is scored
Every non-empty answer earns a small base, then a quality score, which a failed language or safety check caps. Choose an answer.
What the prompts asked for
Share of prompts trained on in each pass, by domain.
Conversations, not just questions
Real users rarely ask one question and leave. So both training sets include 1,219 multi-turn conversations, in English and Indian languages, where the model has to handle the next turn well:
Notice an error hidden earlier in the conversation, and correct it in the follow-up.
Correct a wrong answer when asked again, but keep a correct one when the user pushes back.
Follow a request properly once the user rephrases it.
Leave the old thread behind when the user moves on, and repeat earlier content when asked.
When the model games the score
An RL model optimises exactly what you measure, not what you meant. Reading samples of its answers during development, we caught several ways it found to score well without answering well, and closed each one:
Some instructions carry checkable rules, such as using a letter a given number of times. The model learned to produce text that passed the rule but said nothing useful. A quality gate now catches it.
When a wrong tool call scored worse than no call, refusing became a safe bet. We added partial credit, so any genuine attempt beats a refusal on a question that needs a tool.
Answers that loop the same phrase can slip past a judge. Repetition is now detected and scored as zero.
In multi-turn conversations, reusing a previous answer can look plausible. Answers mostly copied from earlier in the conversation are capped.
We also treated the judge as infrastructure. A verdict that came back unusable was retried, which happened for under 1% of judged answers for Dhīr-2B and about 3% for Dhīr-5B MoE. For the roughly one answer in ten thousand that still had no usable verdict, the answer took the average score of its group, so it neither helped nor hurt. When scores decide what a model learns, an unreliable scorer is a training bug.
What we observed
Dhīr-2B completed three passes over its prompt set, and Dhīr-5B MoE two, each taking eight to nine hours on its 32-chip slice. Move through the runs below. The thin line is each step; the thick line is a 15-step average.
Scores range from 0 to 1.1. Each pass is one shuffled run through the prompt set: 179 steps for Dhīr-2B and 170 for Dhīr-5B MoE.
Shorter answers: progress, or loss?
Both models wrote shorter answers as training went on. Dhīr-2B’s average fell from 204 to 168 tokens between its first and third passes, and Dhīr-5B MoE’s from 220 to 202. Nothing in the score rewarded brevity, so the model got there on its own.
That is a clear behavioural change, but it needs careful reading. Shorter answers can be more direct, or they can leave out something important. Length alone cannot tell you which, so we did not treat it as a result. We looked at evaluations the model never trained on.
Where reinforcement learning helped
Score before RL (after supervised fine-tuning) and of the released model, in %.
Single evaluation runs with greedy decoding; differences of about a point are within run-to-run noise. Not every benchmark moved: knowledge-heavy tests stayed roughly flat. The Dhīr technical report has the full results.
What we learned
Separate the algorithm you configured from the one you ran
We took one update per freshly generated batch. In that setting, the step that GRPO and GSPO use to limit each update never comes into play, so the two reduce to the same update, as our technical report shows. Our runs therefore say nothing about which algorithm is better, and we do not claim they do.
The prompt set is a moving target
A prompt is only useful while the model sometimes gets it right and sometimes does not. As the model improves, prompts saturate and the learning signal thins. Screening prompts is not a one-off cleaning step; it has to keep pace with the model.
Read what the model actually says
Every reward exploit we found, we found by reading answers, not by watching charts. A rising score is a claim about the reward. Only the outputs tell you whether it is a claim about the model.
RL is a whole system
Useful RL needs prompts that carry signal, rewards that reflect the behaviour you want, reliable generation and scoring, and evaluations that go beyond training metrics. A weakness in any one of them shows up as a model that learned the wrong thing.
For Dhīr, this work established a practical RL pipeline on TPUs. It also reinforced a principle we will carry into future experiments: inspect what the model actually says, measure the behaviours users care about, and keep research claims grounded in what the experiment demonstrates.