The first three chapters took one thing for granted, the same thing this book set out to break: that "the model" means a text chatbot. Training, serving, and frontier strategy were all argued over language models. But the discipline those chapters described — build the model, budget its serving, choose your strategy at the frontier — has stopped being a property of text. It is now being pointed at speech, at recommendation, at generated images and video and music, at driving, and at the movement of physical robots. One Nvidia session put the shift in a single line of framing: "Foundation models don't just write or draw anymore—they're starting to move." GR00T N1
The striking thing across this cluster is not that every modality got its own foundation model. It is that they are all converging on the same shape, and by two distinct mechanisms. The first is tokenization: modalities that were never sequences of words are being cut into discrete tokens and generated one at a time, so a speech model or a ranking model comes to look like a language model on the inside. The second is consolidation: the zoo of narrow, task-specific models that used to run a product is collapsing into a single end-to-end model that does the whole job. This chapter follows both mechanisms across five modalities, and closes Part I where it began — on the argument that model-building is one discipline, whatever the model outputs.
Speech models are becoming language models
Text-to-speech used to be its own stack, with its own vocabulary and its own tricks. It is now, structurally, a language model.
Samuel Humeau of Mistral makes the architectural case directly: modern TTS increasingly is language modeling over discrete speech tokens. Speech gets cut into a sequence of discrete units, and the model generates that sequence the way an LLM generates text — autoregressively, token by token, or by diffusion — optimized to start producing audio with the lowest possible latency inside an interactive agent. Samuel Humeau The older offline synthesis pipeline, which rendered a whole utterance from a fixed representation, is giving way to something that inherits the LLM's scaffolding wholesale.
The practitioner side confirms the shift in the plainest possible way: the thing you now learn, when you learn TTS, is how token-based text-to-speech models work. Ronan McGovern's fine-tuning workshop trains exactly such a model — Sesame's token-based CSM-1B — on a single target voice, using the same data-preparation-and-fine-tune loop an engineer would recognize from any language model. Ronan McGovern When speech is tokens, speech engineering becomes token engineering, and the skills transfer.
The zoo of task-specific models is collapsing into one
The second mechanism is louder, because it is happening on the largest consumer surfaces on the internet, and it is quiet only because users never see it. Recommendation — the system that decides what you watch, read, and are shown next — is being rebuilt on foundation models, and it may be the biggest LLM application of them all.
Devansh Tandon of YouTube states the scale of it without hedging: "The application of LLMs to recommendations is going to be a bigger consumer application than search." Devansh Tandon His reasoning is that recommendation is "a bigger problem that is underhyped because it's kind of transparent to the user" — it drives most of the watch time for billions of people while attracting a fraction of the attention that the search-versus-chatbot fight gets. YouTube's move is to adapt Gemini itself into a recommender, replacing the traditional stack of large embedding models with a language model taught to speak in video IDs.
The pattern is not one company's bet. It is the same play at three of the largest personalization surfaces in the world, and each frames it as consolidation. Netflix's Yesu Feng describes the goal as "to use one foundation model to cover all the recommendation use cases" — one model behind every row of a homepage that used to be served by a collection of specialized ones. Yesu Feng LinkedIn's 360Brew team tells the same story: leveraging large language models for personalization and ranking, and taking a single foundation model built for ranking and recommendation all the way to production across the platform's many surfaces. 360Brew, LinkedIn The consolidation move — replace many bespoke models with one foundation model — is now the default architecture for recommendation at scale, and it is the same move that produced the general-purpose chatbot, applied to a problem that predates chatbots by two decades.
For live modalities, latency is a design constraint, not a tuning pass
When the model talks back in real time, its serving budget stops being something you optimize after the fact. It becomes something the architecture is built around from the start.
Chapter 2 already argued the cost side of this — that the tightest budget in serving is real-time voice, and that the economics of a $1-per-hour voice call push every decision. This chapter owns the other side: latency as a property of the model's design, not just its bill. Humeau's account of streaming TTS is exactly this — the model is shaped to emit its first audio as fast as possible, because a that pauses to think has already broken the illusion. Samuel Humeau Neil Zeghidour of Gradium AI sharpens the point by naming what still blocks the natural, cinematic "Her" conversation: not raw model quality, but end-to-end latency, tool-call delay, interruption handling, and the ability to manage overlapping speech and turn-taking. Those are architecture problems, not polish. Neil Zeghidour As synthesis and recognition get faster and cheaper, the bottleneck simply moves up the stack to real-time interaction design — which means the hard part of a voice model is increasingly a systems problem baked into how the model generates, not a knob turned at deployment.
Generative media runs the opposite way: compose, don't consolidate
Consolidation is the dominant move in this chapter, but generative media is the deliberate exception, and the exception is instructive. Here the craft is not one great model. It is orchestrating several.
Gorkem Yurtseven of fal, who runs an inference platform that serves many of these models, defines the field broadly: "It's so nice to see a generative media track in the AI engineer conference this year… the way we define it is a generative video, audio, or image." Gorkem Yurtseven From the vantage of a platform serving all of them at once and partnering with closed-source providers alongside open ones, the shape of the field is a catalog of specialized models, not a single generalist — and the product is the ability to chain them under structured control. Google DeepMind's GenMedia stack is the concrete instance: a workshop that illustrates a whole book by threading Gemini as the prompt author into Imagen for images, Veo for video, and Lyria for music, each a separate model family orchestrated in sequence rather than merged into one. Guillaume Vernade Falling video-generation prices are what make that composition practical — cheap enough per clip to iterate a prompt many times before committing to a final render — which shifts the discipline from training one model toward composing several well.
Demand for these systems arrives faster than the infrastructure under them. Keegan McCallum of Luma AI describes the launch of the Dream Machine video model as a scramble to keep up: "We'd allocated about 500 H100 GPUs. We thought that was a lot at the time. It wasn't." Keegan McCallum A million users in four days is the shape of the demand the whole cluster is racing to serve.
Embodiment is the frontier's next modality
The consolidation mechanism does not stop at the screen. It is now aimed at the physical world, where a foundation model has to perceive, plan, and act.
Waymo's EMMA is consolidation applied to driving. Where an autonomous-driving stack was historically a modular pipeline of specialized components, EMMA is an end-to-end multimodal model — one model taking in sensor data and producing driving decisions, in the lineage of the end-to-end driving research that emerged around 2020. The talk's own title, "Teaching Cars to Think," names the ambition: the driving stack framed as a single model problem rather than an assembly of engineered parts. Waymo's EMMA Robotics is the same story one step further from the road. Nvidia's GR00T N1 is presented squarely as a build-the-model problem — "Today we're going to give you a sense of… how you go about building a robotics foundation model" — the humanoid's control treated as a foundation model to be trained, not a controller to be programmed. GR00T N1
This is the seam into Part II. Here the humanoid and the self-driving car are models — the last modality the model-building discipline reaches. In Chapter 5 they become a domain, with its own strange constraints that the generic agent playbook does not fit. The same two talks that close Part I open the argument for the physical world.
The model layer is one discipline
Put the two mechanisms together and the boundary this chapter set out to erase is gone. Speech got tokenized until it looked like language (#663, #385). Recommendation consolidated a zoo of models into one foundation model, at a scale its own champions think exceeds search (#245, #247, #248). Live interaction pushed latency down into the architecture rather than leaving it at the serving layer (#663, #662). Generative media ran the other way, toward composing many specialized models under control (#244, #692) — the exception that shows consolidation is a choice, not a law. And embodiment carried the whole discipline off the screen and into driving and humanoid robots (#174, #165).
That answers the packet's standing question — is this one chapter or two? It is one, held together by two mechanisms rather than one theme. Tokenization and consolidation are different forces, but they push toward the same destination: model-building as a single discipline that no longer cares whether the output is a paragraph, a playlist, a song, or a step forward. Part I has spent four chapters on the model layer — how it is trained, served, built at the frontier, and now extended past text. Part II turns to the places where having a capable model is not enough, because the domain itself breaks the playbook. The first of those domains is the one this chapter just handed over: the physical world.