The first book took the model as given and began after the weights froze. Chapter 1 broke that assumption on the training side: the model is increasingly something you build, not something you receive. This chapter breaks it on the other side. Once you have a model — trained, tuned, and ready — you still have to run it. Running it is where the money, the latency, and the hard constraints actually live.
That is the subject of inference economics. Training is a capital cost you pay once. Inference is a recurring cost you pay on every request, for as long as the product lives. It is also where the user waits, where the bill compounds, and where a good model can still fail because the system around it is slow or expensive. The center of the AI business is not the training run. It is the served token.
Alex Atallah built OpenRouter into a marketplace that routes traffic across dozens of model providers, so he watches the whole flow from the middle of it. He states the scale of it plainly: "Inference might be the largest market ever in software." (Alex Atallah, OpenRouter) The whole open-model ecosystem — the serving frameworks, the inference clouds, the on-device runtimes this chapter surveys — exists because serving is where recurring cost and competitive advantage live. If the model is the product, inference is the cost of goods sold.
This chapter runs in two movements. The first is serving performance: how you make a given model fast and cheap — the software layer of engines, compilers, quantization, and on-device runtimes that you control. The second is the compute substrate: the GPUs, networks, data centers, and economics underneath, the floor you cannot move, up to and including geopolitics. Movement A is the frontier you can push. Movement B is the ceiling that pushes back. Together they decide what a trained model can actually do once real users arrive. Chapter 1 fed this one — reinforcement learning produces models you then have to serve. Chapter 3 asks who can afford to build at the frontier. Chapter 4 meets the hardest latency budgets of all, in voice and media.
Movement A — Serving Performance
Speed and cost are a system property, not a model property
A model's latency and cost are not printed on the checkpoint. They are set by the system you run it on.
Kyle Kranen ran one of NVIDIA's largest inference deployments — tens of millions of dollars a quarter — and his central point is that the model alone tells you almost nothing about how it will perform in production. "A good model and a good system that takes into account the actual constraints for what you need from your deployment is actually key to the success of both your deployment and the application." (Kyle Kranen, NVIDIA) The same weights, served two ways, differ by an order of magnitude in throughput and price. The variable that moves is the deployment, not the checkpoint. Kranen frames it as an inference Pareto frontier — the curve trading latency against cost against quality — and the engineering work is knowing where on that curve your application needs to sit, then building the system that puts you there.
Mark Moyou, an NVIDIA architect who works this problem with enterprise clients, gives the same warning from the cost side: LLM inference is not a normal model deployment, and it is not trivial once you have to manage scale, performance, and cost at once (Mark Moyou, NVIDIA). The naive path — take the model, wrap it in a simple server, ship it — leaves most of the performance and most of the savings on the table. Dmytro Dzhulgakov of Fireworks puts a figure on the gap: making open models roughly ten times faster is a serving problem, solved with secure, latency-sensitive, low-cost serving rather than a new model (Dmytro Dzhulgakov, Fireworks). The lesson under all three is one : if you want a model to be fast and cheap, you optimize the system, not the model.
The serving stack is the lever
If the system sets the cost, the serving framework is the first thing you reach for.
A serving engine is the software that turns model weights into a running API endpoint — batching requests, managing the key-value cache, and squeezing the hardware. The open ecosystem has matured fast here. Charles Frye of Modal spent a year measuring how fast open inference engines actually are, and his framing captures the shift: open-weights models and open-source inference servers have made major strides, to the point where running a serious agent on open models is now a real option rather than a compromise (Charles Frye, Modal). SGLang, an open-source serving framework, aims to serve models like DeepSeek and Qwen at state-of-the-art speeds on launch day (SGLang, Baseten). TensorRT-LLM sits at the high-performance end — the fastest framework in Baseten's account, with a steep learning curve as the price of that speed (TensorRT-LLM, Baseten).
The point is not which framework wins. It is that the framework is now a design decision with a large payoff, and the payoff is open to anyone. Dzhulgakov's whole pitch at Fireworks is productionizing open models — taking a checkpoint anyone can download and serving it faster and cheaper than the naive path allows (Dmytro Dzhulgakov, Fireworks). A team that treats serving as an afterthought pays the order-of-magnitude penalty Kranen described. A team that treats it as a lever gets frontier-adjacent economics out of open weights.
Compilers and generated kernels are automating the expert layer
The deepest performance work used to require a scarce specialist. Increasingly, it is done by search and by models.
A kernel is the low-level code that runs one operation on the GPU. Hand-writing fast kernels is expert work, and experts are rare. Two shifts are removing that bottleneck. The first is search-based compilation: Joe Fioti's Luminal is a deep-learning compiler for CPUs, GPUs, and ASICs that discovers efficient kernels — flash attention among them — automatically, by search rather than by hand (Joe Fioti, Luminal). His theme is radical simplification through search: a far simpler library that gives up nothing on performance, because the compiler finds the fast path. The second shift is generation: Natalie Serrino's Gimlet Labs works on AI-generated kernels that can meaningfully speed up custom code with no human effort at all (Natalie Serrino, Gimlet Labs). The optimization that used to gate a deployment behind a specialist is becoming something a compiler or a model produces on demand.
Portability is the other half of this. Yusuf Olokoba of Muna frames the practical question as one of reach: "How do I use more models in more places without having to rebuild or extend?" (Yusuf Olokoba, Muna) Chris Lattner's MAX at Modular attacks the same seam — real applications span GPU compute, CPU preprocessing, and orchestration, often across different vendors, and the productivity win is a stack that spans them without a rewrite (Chris Lattner, Modular). The direction is consistent: the expert layer of inference optimization is being automated and abstracted, which widens the set of people who can deploy a model well.
Inference is moving onto the device
For a growing class of models, the cheapest and most private data center is the one already in your pocket.
The hardware crossed a threshold. Adrien Grondin runs Gemma 4 on an iPhone at 40 tokens per second through Apple's MLX framework — a capable open model, on a phone, fast enough to use (Adrien Grondin, Locally AI). Prince Canuma, a core MLX contributor, makes the economic case directly: "I want to convince you today to offload some of that subscription completely on device and then all you need to pay is your energy bill." (Prince Canuma, Neywa Labs) He backs it with working demos on consumer Macs — vision, speech, even a million-token context enabled by his own cache-compression work — and with a reason that is not about cost at all. His father went blind in rural Africa, where cloud subscriptions are unaffordable and connectivity is unreliable; for that user, on-device is not a preference but the only option. On-device inference changes who can reach AI, not just what it costs.
The pattern is not confined to Apple silicon. Mozilla's Llamafile, from Stephen Hood and Justine Tunney, makes open models run fast on ordinary consumer CPUs, with the explicit goal of democratizing access (Stephen Hood & Justine Tunney, Llamafile). Microsoft's Foundry Local, presented by Emma Ning, brings the same capability to cross-platform apps through the ONNX Runtime, and her first argument for it is the conference-Wi-Fi problem — cloud AI fails the moment the network does (Emma Ning, Microsoft). A per-month subscription becomes a one-time energy cost. A round trip to a data center becomes a local call. The move from demo to default is underway.
The tightest budget is real time
Some workloads have no slack. When a human is waiting to hear a reply, the latency budget is a few hundred milliseconds, and every part of the serving stack has to fit inside it.
Neil Dwyer of Gabber ran realtime text-to-speech in production, and reports the discipline it takes to serve voice at roughly a dollar an hour — open-source models, LoRAs, and careful load balancing, all bent toward latency (Neil Dwyer, Gabber). Voice is where serving performance stops being an optimization and becomes a hard gate: miss the budget and the product does not work at all. These realtime economics are the seam where this chapter meets Chapter 4, which takes up voice and media models on their own terms. Here the point is narrower and belongs to inference economics: the tighter the latency budget, the more the system — not the model — decides whether the product is possible.
Movement B — The Compute Substrate
Underneath the stack is a substrate that sets the ceiling
Every serving trick in Movement A runs on physical hardware, and that hardware sets a ceiling no software can lift.
For years most AI engineers built above an API boundary — the OpenAI call, the Anthropic call — and never touched the metal. Charles Frye argues that boundary is no longer enough: AI's extreme demand for mathematical throughput means engineers now need to understand the GPU underneath — its processors, its memory, its limits — the way every programmer once had to learn a little about hardware (Charles Frye, Modal). The substrate is not a footnote to the serving stack. It is the thing the serving stack is fighting against.
And the substrate is more than chips. Paul Gilbert of Arista Networks describes the full picture as infrastructure that must be built and wired together: "What we do is a plumbing… how you train models, what the infrastructure looks like, and how you do inferencing on the infrastructure." (Paul Gilbert, Arista Networks) Building an AI data center means power, cooling, networking, and the fabric that moves data between GPUs fast enough to keep them busy. That plumbing is now a first-order engineering discipline, because it sets the ceiling the software in Movement A is trying to reach.
The substrate is contested — in silicon and in geopolitics
Because the substrate sets the ceiling, whoever changes the substrate changes what is possible — which makes it a place of intense competition.
New silicon is one front. Sunny Madra of Groq frames his company's specialized inference chip against a historical marker: "Twenty-five years ago we crossed the 1 gigahertz speed barrier in microprocessors." (Sunny Madra, Groq) The analogy is deliberate. A step change in raw inference speed does not just make existing products faster; it unlocks applications that were impossible at the old speed, the way the gigahertz threshold once did. Purpose-built inference hardware is a bet that the GPU is not the last word on the substrate.
The other front is national. Dylan Patel of SemiAnalysis argues that the infrastructure behind AI — chips, data centers, power, supply chains — has become an arena for geopolitical competition, with China, the United States, and the Middle East each maneuvering, and players like Huawei closing the chip gap by engineering around export controls (Dylan Patel, SemiAnalysis). This is where inference economics stops being a purely technical discipline. The cost and availability of the substrate now depend on trade policy, fabrication capacity, and national strategy. The ceiling on what any serving stack can do is, in part, a political variable — a strange and important thing to have to say about the cost of running a model.
Trust is becoming a first-class inference requirement
For a large class of buyers, a model is unusable until they can run it on sensitive data without surrendering the data or the result. That is an inference problem, and it has become a product category.
Mike Bursell names the blocker directly: "AI is transforming everything. Healthcare, finance, automation, digital marketing. But one thing holds it back, and that's trust. How do you run models on sensitive data without handing it over?" (Mike Bursell) His answer is confidential computing — running inference inside a hardware-protected environment so the data stays private even from the operator of the machine. The demand is real, because the alternative — shipping regulated data to someone else's GPU — is a non-starter for the industries with the most to spend.
Sahil Yadav and Hariharan Ganesan of Telemetrak reach trust from the enterprise side, after a decade deploying AI in healthcare, industrial IoT, and telecom. Their recurring question is the one a CIO actually asks: can we trust the model's decisions when they drive mission-critical outcomes and the bottom line? (Yadav & Ganesan, Telemetrak) Between the two talks, trust splits into two requirements inference must now meet: confidentiality — the data is safe — and reliability — the decision can be relied upon. Neither is a property of the model in isolation. Both are properties of how and where it is served. Trust has moved from an afterthought to a design input, alongside speed and cost.
Inference is the discipline, not the afterthought
Put the two movements together and the chapter's is simple. A model's cost and speed profile is a design input as fundamental as its accuracy. You set that profile in the serving stack — the engines, compilers, quantization, and on-device runtimes of Movement A — and you set it against a substrate you do not fully control — the GPUs, networks, power, and politics of Movement B.
This is why inference, not training, is the discipline most AI engineers will spend their careers inside. Most engineers will never run a pre-training job. Nearly all of them will serve a model, and the moment they do, they inherit the whole problem: the Pareto frontier, the framework, the kernel, the device, the chip, the data center, the trust boundary. Atallah's line was not hyperbole. If inference is the largest market in software, it is because it is the part of AI that never stops charging — and the part where good engineering compounds on every request. The next chapter turns to who can afford to build at the frontier of that market at all.