The first book made one assumption on every page: the model is given. You do not build it. You wrap it, steer it, feed it context, and hold it accountable. The intelligence arrives from a lab, and the engineering begins after the weights are frozen.
That assumption is where this book starts, and it is the first thing to break.
The center of gravity in model-building has moved. For years the story was pre-training: scale the parameters, scale the data, and watch the loss curve fall. That story has not ended, but it has stopped being where the interesting gains come from. The gains now come from what happens after pre-training — supervised fine-tuning, preference optimization, and, above all, reinforcement learning. This is the post-training turn. It is what converted raw language models into systems that reason and act, and it is why a chapter on training belongs at the front of a book about building with AI.
Alessandro Cappelli of Adaptive ML names the trap that makes this turn easy to miss. Teams assume the demo is the hard part and production is "the last mile." He argues the inverse: the MVP is the first mile, and the real work is getting a system to keep improving after it meets the real world, which introduces defects no pre-launch evaluation ever covered (Alessandro Cappelli, Adaptive ML). Nathan Lambert of Ai2 describes the same moment from the research side: six months after DeepSeek and OpenAI's o1, everybody has a reasoning model, the basics are widely understood, and the live question is where the technique goes next (Nathan Lambert, Ai2). Both are describing a field that has stopped treating the model as a fixed input. The behavior you want is trained in. Jack Rae, who leads thinking within Gemini, frames the underlying method plainly: progress comes from finding an intelligence bottleneck in the current model and removing it — in the architecture or in the training (Jack Rae, Google DeepMind).
This chapter argues that once you accept that behavior is trained in rather than prompted in, the rest of Part I follows. How you serve the trained model is Chapter 2. Who can afford to build it is Chapter 3. Which modality it targets is Chapter 4. But the hinge is here: the model is no longer given. Increasingly, you are training it.
Behavior is trained in, not prompted in
Post-training is not one technique. It is a menu, and the rungs climb in cost and power.
The workshop material from OpenAI's developer experience team lays the menu out directly: supervised fine-tuning, direct preference optimization, and reinforcement fine-tuning are distinct tools, each suited to a different kind of gap (Ilan Bigio, OpenAI). Supervised fine-tuning teaches a model to imitate examples of the output you want. Preference optimization teaches it to prefer one response over another. Reinforcement fine-tuning goes further: it optimizes against a reward, so the model can discover behavior no example set spelled out. The names matter less than the ladder they form. Each rung absorbs a different signal, and the highest rung absorbs the one that matters most in production — a measure of whether the work was actually good.
The reason the top rung has taken over is that the target moved from text to action. A model that only answers can be tuned on answers. An agent — a model that calls tools and takes steps in the world to finish a task without checking in at every turn — has to be tuned on whole trajectories of behavior. OpenAI's fine-tuning team frames agent reinforcement fine-tuning as the most powerful way to raise an agent's performance precisely because an agent is defined by its ability to interact with the outside world to get things done on its own (Will Hang & Cathy Zhou, OpenAI). You cannot write down every correct trajectory by hand. You can only reward the good ones. That is a training problem, and it is a reinforcement-learning-shaped one.
Reasoning and agency are the same training problem
The field talks about reasoning models and agents as if they were two research tracks. They are one.
Will Brown of Prime Intellect makes this the thesis of his talk on training agentic reasoners. People treat reasoning as one thing and agents as another, with separate considerations for each. He rejects the split: "The high level thesis of this talk is like — no, they're kind of the same thing." (Will Brown, Prime Intellect) The unification is mechanical, not rhetorical. A reasoning model is trained to produce a long internal chain before it answers. An agent is trained to produce a long external chain — call a tool, read the result, call another — before it finishes. Train multi-turn tool use with reinforcement learning and the "reasoning" and the "acting" become the same loop: a sequence of steps optimized against whether the final result was correct. Modern systems built this way, like o3-style deep research, are reasoners and agents at once because the training made no distinction.
This is why the recent capability jump did not come from a bigger base model. It came from training models to think and to act in the same loop. Aakanksha Chowdhery of Reflection.ai, who led research on PaLM and worked on Gemini before turning to autonomous coding, traces the same arc: the frontier moved from pre-training scale, through inference-time tricks like majority voting, to reinforcement learning at training time as the durable source of gains (Aakanksha Chowdhery, Reflection.ai). Once you see reasoning and agency as one training problem, the strategic questions collapse into one question too: what reward are you training against, and can you compute it at scale.
Thinking is a training signal that scales
The reason to invest in training-time reasoning is that it compounds. More training-time reinforcement learning buys more inference-time gain.
Nathan Lambert states the relationship cleanly: "You can scale RL at training time and the numbers will go up and that's deeply correlated with being able to then do this inference time scaling." (Nathan Lambert, Ai2) The two scaling knobs are linked. Spend more compute teaching a model to reason and it also gets more out of the extra compute you give it at answer time to think longer. That is a different economics from pre-training, where the only lever was a larger model. Here the lever is how well the model uses a variable thinking budget, and that skill is trained.
Jack Rae's framing explains why labs pour effort into this specific capability rather than raw scale (Jack Rae, Google DeepMind). Advancing intelligence is a matter of finding the current bottleneck and removing it, and for the current generation of systems the bottleneck was the inability to think before answering. Removing it was a training change, not just an architecture change. Thinking, in other words, is not a prompt you write. It is a behavior trained, and it is the behavior that turned a fast autocomplete into something that can work a hard problem.
Reinforcement learning is what carries a model into production
Of the post-training methods, reinforcement learning has a property the others lack: it can keep learning from production.
Cappelli puts the at its strongest: "Reinforcement learning RL is not just any other algorithm for post-training, but is an algorithm that at its core will bring models to production." (Alessandro Cappelli, Adaptive ML) His reasoning is about feedback. Prompt engineering cannot systematically absorb a production signal — patch one defect in the system prompt and you likely introduce another, with no mathematical handle on the tradeoff. Supervised fine-tuning can absorb feedback, but only by rebuilding the dataset every time the world surfaces a new edge case, which is expensive and never finished. Reinforcement learning was designed to integrate a reward signal continuously. It treats production data as part of the training loop rather than as something that lives outside it. That is why, in his account, it is that closes the gap between a demo and a system that improves in the field — and why he opens with the blunt statistic that ninety-five percent of generative-AI pilots never reach production at all.
The payoff is not only accuracy. Cappelli's deployments at Fortune 500 scale turn on economics that reinforcement learning improves in three ways. It reaches the quality of a supervised fine-tune with fewer parameters, so the served model is smaller and cheaper — decisive when a company like AT&T summarizes every customer-agent call and the bill runs into the millions. It fits latency budgets a large frontier model cannot: speech-to-speech support tolerates only a few hundred milliseconds of delay, which forces a small, fast model, and reinforcement learning is the most efficient path to making a small model good enough. And it produces a model the business owns, trained on its own data, so a silent vendor-side update cannot shift the product's behavior overnight. Accuracy is the headline; cost, latency, and ownership are why sticks.
Sylendran Arunagiri of NVIDIA reaches the same destination from the opposite direction, and lands one of the chapter's most useful correctives: "It's not about powering them up with the next biggest LLM in the market. All they need is simple data flywheels." (Sylendran Arunagiri, NVIDIA) The instinct when an agent underperforms is to reach for a larger frontier model. The better move is usually to build the loop that feeds real-world outcomes back into the system so it improves over time. The flywheel is the productionizing mechanism, and reinforcement learning is the training method built to turn it.
Code is the domain where RL scales first
Reinforcement learning needs a reward. The domains where it scales fastest are the ones where the reward is free and honest.
Chowdhery's argument for autonomous coding rests entirely on this (Aakanksha Chowdhery, Reflection.ai). Inference-time and reinforcement-learning techniques only pay off reliably in domains with automated verification — math, with calculators and formal proofs, and code, with unit tests and compilers. In those domains a model's output can be checked by machine, cheaply and without a human in the loop, which gives the training process a reward signal it can trust and run millions of times. She points to results where coverage on a SWE-bench-style benchmark climbs toward roughly eighty percent as sample count rises, and notes the catch: plain majority voting does not carry that gain into practice, because a correct generation can be too rare to find by sampling alone. The way past that ceiling is reinforcement learning at training time rather than tricks at inference time. Her framing of autonomous coding as the "root node" problem follows from the same fact — code is where verifiable rewards are most abundant, which makes it where reinforcement learning has the most room to run.
This is also why unverifiable domains lag. A reward model made of another neural network can be gamed; a compiler cannot. Execution feedback and unit tests reduce the reward-hacking risk that plagues reinforcement learning with neural reward models, which is the specific reason Chowdhery's team treats autonomous coding as the "root node" problem — the place to start building toward broader capability, in the spirit of what David Silver and Rich Sutton call the "era of experience." A roughly thirty-five-person company betting on that thesis is a wager that the reward, not the model, is the scarce resource. The further a task sits from an automatic checker, the harder the reward is to build, and the reward is the whole game. Code is not special because it is important. It is special because it grades itself.
The data recipe beats model size
For reasoning quality, the differentiator has moved from the size of the model to the quality of the data pipeline that trains it.
Ryan Marten of Bespoke Labs makes the case with an open project: "Open Thoughts, which is our project to create the best open-source reasoning data sets." (Ryan Marten, Bespoke Labs) The resulting model, OpenThinker, reached state-of-the-art among small open reasoning models — outperforming DeepSeek distillations — and it got there not by adding parameters but by curating the reasoning traces it trained on. The lesson is that a competitive reasoning model can be built in the open when the data recipe is good enough, which relocates the advantage from the pre-training budget to the dataset.
Chang She and Noah Shpak, working on training-data platforms at LanceDB and Character.AI, give this a structure with their hierarchy of needs for training-dataset development: model quality depends critically on how the dataset is constructed, and dataset construction is part craft, part science (Chang She & Noah Shpak). It is the same point Arunagiri's data flywheels make from the production side (Sylendran Arunagiri, NVIDIA): the pipeline that produces and refines training data is now the lever, not the parameter count. When the differentiator moves to the data, the discipline moves with it.
You probably don't need fine-tuning — until you do
The turn to training does not mean every team should train. Most should climb the cheaper rungs first.
Kyle Corbitt, who runs a fine-tuning platform, opens against his own interest: "The title of my talk is that you probably don't need fine tuning — this is an awkward title, given that I am doing a fine-tuning platform." (Kyle Corbitt) His maturity curve runs from simple instruction prompting, through few-shot examples and retrieval, and only then to fine-tuning — with reinforcement learning further still. The mistake he warns against is jumping to training before exhausting the cheaper rungs, which is common and expensive. Prompting and retrieval solve a large share of problems at a fraction of the cost, and they solve them today, without a training run.
Abi Aryan's survey of domain adaptation makes the same map from the enterprise side: the methods run from prompt engineering to retrieval to fine-tuning, and the right choice depends on the dataset and the problem, not on fashion (Abi Aryan). Read alongside Cappelli's argument, this is not a contradiction but a sequence. You probably do not need fine-tuning to reach a working MVP. You probably do need training to keep a high-stakes system improving once it is in production and the cheaper rungs stop absorbing new failure modes. The skill is knowing which mile you are on.
The hard part is operations, not the algorithm
The reinforcement-learning algorithms are, by now, well understood. What stops teams is the machinery of running the loop at scale.
Cappelli is direct about the cost: running PPO means orchestrating four models simultaneously, which is hard enough that most teams cannot do it without dedicated infrastructure — the problem his platform exists to absorb (Alessandro Cappelli, Adaptive ML). Chowdhery gives the same figure from the research side and shows it is not merely a PPO quirk: PPO-style RLHF keeps four copies of the model in memory, and even DeepSeek's GRPO, which drops the value model, still needs three (Aakanksha Chowdhery, Reflection.ai). Holding several large models in memory at once, in sync, is an operations problem before it is a research one.
The infrastructure underneath compounds the difficulty. Modern models increasingly use mixture-of-experts designs that train larger networks efficiently by distributing layers across many machines, and that distribution only works over a specialized network fabric — rail-optimized InfiniBand, in the Crusoe Cloud account — engineered to move data between GPUs fast enough to keep the training busy (Crusoe). Daniel Han's full workshop frames the live question bluntly — why is reinforcement learning suddenly everywhere, and is it truly effective — and answers it across kernels, quantization, and agents, which is to say across the operational surface, not the theory (Daniel Han). The technique is not the moat. The ability to run it is. This is the thread the next chapter picks up, where serving and inference economics become the constraint that decides what a trained model can actually do in production.
How you know training worked
Training is only worth doing if you can tell whether it helped. For a specific use case, that is harder than it sounds.
Emmanuel Turlay puts the difficulty plainly: there are metrics and benchmarks, but they mostly apply to generic tasks, and there is no one-size-fits-all process to evaluate a model's performance on a particular use case (Emmanuel Turlay). A leaderboard score tells you little about whether a fine-tune improved the job in front of you. You have to build the evaluation for the use case, and that evaluation is what turns a training run from a gamble into a measurement.
The common shortcut is to let a strong model grade the outputs. Alex Volkov of Weights & Biases endorses but with a guardrail: you should not try an LLM judge without a bit of human in the loop (Alex Volkov). The judge scales the scoring; the human keeps the judge honest, calibrated against real preferences. This is the same shape as the training loop itself — a machine-computed signal, checked against human judgment where it matters. Evaluation is not a separate discipline bolted on after training. It is the reward signal, viewed from the other side. If you cannot measure whether training worked, you cannot train toward anything, and the turn to reinforcement learning was, at bottom, a turn toward optimizing a signal you can trust.