← All chaptersSecond Book (Draft)

Chapter 03 · 9 min read

Building Frontier Models

Why building a frontier model became its own engineering culture, and why the frontier is now crowded rather than a two-lab race.

CHAPTER 03/2,047 words/Drafting

For a few years the frontier looked like a two-lab race. There were the labs that could train a frontier model, and there was everyone else — the developers who called their APIs. That picture is now wrong.

The first two chapters took the model apart from the inside: how it is trained (Chapter 1) and how it is served (Chapter 2). This chapter zooms out to the labs and the strategy. Building a frontier model has become its own engineering discipline, with its own culture — set by release cadence, by the choice to open or close the weights, by which benchmarks the field decides to chase, and by the hardware that shapes every design decision underneath. And that discipline is no longer scarce. Simon Willison, reviewing the field, had to cut his own scope: "I counted 30 significant model releases in the past six months." Simon Willison That is roughly one serious model release a week, from more labs than any one engineer can track.

The volume underneath that headline is larger still. Eugene Cheah, counting uploads rather than launches, put it at "more than 50,000 AI models have been uploaded to Hugging Face per month… that is more than one AI model a minute." Eugene Cheah The frontier is not a place a handful of labs guard. It is a crowded, fast-moving field, and the strategy of building at it — not just the technique of training — is now worth studying in its own right.

"Stagnation" is a training-budget story, not a ceiling

The most common complaint about the frontier is that it has stalled. The complaint mistakes a spending decision for a physical limit.

Dylan Patel of SemiAnalysis traces the felt stagnation to a plain fact about what the field is actually running: "The models that we're using today are largely the same as the models that were trained in 2022." Dylan Patel The flagships people use are, in his account, smaller models trained for longer — same generation of architecture and pre-training, refined rather than replaced. When a newer model beats an older one at a smaller size, the gain comes from training it better and longer, not from a new idea about how to build it. The apparent plateau is not the ceiling of what the architecture can do. It is the lag between what has been trained and what could be, given a bigger run and more post-training.

Read that way, "stagnation" is a budget and scheduling story. The leap is still available; it just costs a training run nobody has finished paying for yet. This is also why the hardware chapter is not separable from the strategy chapter: what a lab can build at the frontier is set by what it can afford to train, and what it can afford to train is set by the compute substrate underneath (Chapter 2). The complaint about a plateau is really a complaint about the pace at which the field is willing to spend.

Frontier progress is bottleneck-hunting

Labs do not advance by improving everything at once. They advance by finding the single thing most holding the best current system back, and unblocking it.

Raia Hadsell, VP of Research at Google DeepMind, frames frontier research as exactly this search: progress toward general intelligence has been marked by identifying the fundamental bottlenecks inside existing models and building the fix — in the architecture or the training — that removes them. Raia Hadsell The discipline is diagnostic before it is inventive: the hard part is naming the right weakness, not generating candidate improvements. Jack Rae, tech lead for thinking in Gemini, describes the same method at work on one capability. His team's excitement about thinking comes from seeing it as a way of "unblocking bottlenecks towards intelligence" — and, in his telling, if you can identify the crucial shortcoming in your most advanced system, the fix often follows. Jack Rae The training method that produced Gemini's thinking is Chapter 1's subject; the point here is the research culture around it.

Frontier progress is a sequence of correctly chosen bottlenecks, and choosing them is the skill that separates the labs. It also explains how a smaller team can compete without a larger team's compute. If the next gain comes from the single right unblocking rather than from broad, expensive improvement across the board, then a team that diagnoses the bottleneck correctly can leap without matching anyone's budget. Bottleneck-hunting is a strategy that rewards focus over scale — which is part of why the frontier stopped being the exclusive property of the labs that could spend the most.

Code is where labs learn to model computation

Nearly every frontier lab starts its reasoning work in the same place: code. The reason is deeper than code being a lucrative market or a tidy source of training signal.

Jacob Kahn of FAIR describes the goal behind Meta's Code World Model plainly: "Our primary goal is to build models that reason, plan and make decisions. And we start with code because it's an interesting sandbox in which to think about reasoning." Jacob Kahn The word that matters is sandbox. Code is constrained; it has rules; and, unlike most of the world, it can be executed, so the model can be trained to predict what a program does, not merely how it looks. That is a different ambition from writing plausible code — it is modeling computation itself, the semantics under the syntax. Code is chosen as the first world simple enough to simulate honestly, and reasoning learned there is meant to generalize outward. Poolside makes the same bet from the opposite end of the market. "Poolside exists to close the gap between models and human intelligence… We're building our own models from scratch to do this," its founders say — starting from code, and pairing next-token prediction with reinforcement learning to make the leap. Poolside Both treat code not as the destination but as the training ground for general reasoning. (Why code's rewards are so cheap to verify — the mechanics that let this training scale — is Chapter 1's argument; the strategic choice to start there is this chapter's.)

The open-model gap has largely closed

The industry spent years assuming the best models would stay closed, and that open weights would trail by a generation or two forever. That assumption has broken.

The turning point was DeepSeek-R1. Eugene Cheah calls it "the first open source model to catch up and surpass GPT-4," and draws the lesson the whole field took from it: "you do not need a billion dollars to compete with the big labs." Eugene Cheah R1 did not just close a benchmark gap; it broke out of the AI-engineering bubble into mainstream business awareness, which is where Amir Haghighat of Baseten picks up the story — the DeepSeek-R1 news cycle is what put open models on the agenda of the largest enterprises, not just the hobbyists. Amir Haghighat And the competition did not stop at parity: MiniMax now presents its M2 as, in its own framing, the number-one open model, built for agentic coding. Olive Song The on the frontier is no longer "closed beats open." It is that open-weight models trade blows with the best closed ones, from teams that never spent a closed lab's budget.

That change is what matters most to the engineer reading this. It permanently alters who gets to build on the frontier rather than merely rent it. When the best available weights were closed, building on the frontier meant calling an API and living inside another company's rate limits, pricing, and update schedule. When the best available weights are downloadable, a team can run the frontier on its own infrastructure, fine-tune it, and inspect it. The gap the industry assumed was permanent turned out to be a lead measured in months, and a lead measured in months is not a moat.

Open source is a strategy, not charity

When a lab releases its weights, it is easy to read the move as generosity. The labs releasing them describe something colder and more deliberate: a way to build an ecosystem and set the standard others build on.

Mistral is explicit that the openness is the strategy, not a side effect. Devendra Chaplot frames his talk around exactly that question — "why we do open source and how we do open source" — from a lab whose commercial platform and flagship models sit alongside the open ones. Devendra Chaplot Google's Gemma team names the payoff directly. Kathleen Kenealy, the team's technical lead, describes the north star as "building something to empower and accelerate the amazing work being done by the open source community" — and measures success by what gets built on top. Kathleen Kenealy The mechanism is control through adoption: Omar Sanseviero defines an open model as one "that you can take, you can download, you can run in your own infrastructure, your own devices… fine-tune for your own use cases." Omar Sanseviero Every one of those actions builds on the releasing lab's format, its tokenizer, its fine-tuning conventions. Alibaba has run the same play for years, shipping the Qwen series continuously since 2023 toward the stated goal of a generalist agent model. Junyang Lin Open weights are how a lab that is not first in raw capability can still become the substrate the field standardizes on.

Benchmarks are how the field steers itself

A benchmark looks like a neutral scoreboard. It is closer to a steering wheel. What the community agrees to measure becomes what the labs optimize, and therefore what gets built.

Alex Duffy makes the mechanism the whole point of his talk: benchmarks are memes, in the original sense of ideas that replicate and shape behavior, and "what we measure shapes AI — and us." Alex Duffy A benchmark that captures the field's attention pulls training runs, data curation, and architecture choices toward the abilities it rewards; the measurement becomes a self-fulfilling prophecy about what the next generation is good at. This is the political layer of the frontier. Deciding which benchmark counts is deciding, in advance, what the next round of models will be trained to do — and the labs that set the benchmark others chase gain the same kind of standard-setting power that open weights confer.

That gives the choice of benchmark real power — and it explains why the field's inability to measure well is such a load-bearing problem. In Amplify's 2025 survey of 500 practitioners, evaluation was named the single most painful part of AI engineering. Barr Yaron The field steers by benchmark, yet building good ones is the thing practitioners find hardest. That is not a small tension. If measurement is the steering wheel, a field that cannot measure well is a field driving toward whatever is easiest to score.

The frontier is a discipline, not a club

Put the pieces together and the frontier stops looking like a club with two members and starts looking like a discipline with many practitioners.

The felt stagnation is a training-budget lag, not a wall (#476). Labs advance by hunting bottlenecks, not by improving everything at once (#013, #267). They start reasoning work in code because it is the first world they can simulate honestly (#067, #049). Open-weight models have caught the closed frontier, from teams without closed-lab budgets (#439, #189, #074) — and the labs releasing them do so to own the standard, not to give it away (#009, #528, #480, #400). Over all of it, the benchmarks the field chooses to chase quietly decide what the next generation will be good at (#253, #137). None of this is the technique of training a model, which Chapter 1 covered, or the economics of serving one, which Chapter 2 covered. It is the strategy of building at the frontier as a distinct thing worth doing well.

What this chapter leaves open is a question of direction. The frontier is crowded and fast — but crowded and fast toward what? So far the whole account has been about text and code. The next move is into other modalities: vision, audio, action, the physical world. That is where Chapter 4 begins.

Evidence — Source Anchors

Reasoning and agency are the same training problem, not two

  • the high level thesis of this talk is like no, they're kind of the same thing.
    #271 — Will Brown, Prime Intellectconfidence: high
  • agent reinforcement fine-tuning or agent RF is the way to do this
    #80 — Will Hang & Cathy Zhou, OpenAIconfidence: high
  • pre-training is not giving any gains
    #251 — Aakanksha Chowdhery, Reflection.aiconfidence: high

For reasoning quality, the data recipe beats model size

  • Open Thoughts, which is our project to create the best open-source reasoning data sets.
    #233 — Ryan Marten, Bespoke Labsconfidence: high
  • you should really care about what you're training on and you should care for it by giving it a nice format
    #543 — Chang She & Noah Shpakconfidence: high
  • it's not about powering them up with the next biggest LLM in the market. All they need is simple data flywheels.
    #397 — Sylendran Arunagiri, NVIDIAconfidence: high

Inference, not training, may be the largest market in AI software

  • inference might be the largest market ever in software
    #335 — Alex Atallah, OpenRouterconfidence: high
  • possible with Open Weights models where they weren't before.
    #316 — Charles Frye, Modalconfidence: high

The compute substrate is a hard constraint — and now a geopolitical variable

  • they optimize for math bandwidth over memory bandwidth.
    #228 — Charles Frye, Modalconfidence: high
  • what we do is a plumbing uh so I'm not going to talk about agents but more kind of how you train uh models what the infrastructure looks like and how you do inferencing on on the infrastructure
    #402 — Paul Gilbert, Arista Networksconfidence: high
  • what's interesting about the geopolitics of this is that despite the fact that Huawei is a sanctioned entity, China is a sanctioned country,
    #345 — Dylan Patel, SemiAnalysisconfidence: high
  • 25 years ago we crossed the 1 gigahertz speed barrier uh in
    #547 — Sunny Madra, Groqconfidence: medium

The open-model gap has largely closed

  • the first open source model to catch up and surpass GPD
    #439 — Eugene Cheah, Featherless.aiconfidence: high
  • not just use an API based generic model u but but inhouse it and do better than than what they can uh what they can do with just uh API based models.
    #189 — Amir Haghighat, Basetenconfidence: high
  • designed specifically for coding workplace agentic tasks.
    #74 — Olive Song, MiniMaxconfidence: high

Perceived "stagnation" is a training-budget story, not a ceiling

  • uh models that we're using today are largely the same as the models that were trained in 2022
    #476 — Dylan Patel, SemiAnalysisconfidence: high

Frontier progress is bottleneck-hunting, not broad improvement

  • our approach we look for root nodes. You know we're not going to waste time on the leaves.
    #13 — Raia Hadsell, DeepMindconfidence: high
  • unblocking bottlenecks towards intelligence.
    #267 — Jack Rae, DeepMindconfidence: high

Code is where labs learn to model computation, not just imitate it

  • Our primary goal is to build models that reason, plan and make decisions.
    #67 — Jacob Kahn, FAIR Metaconfidence: high
  • Poolside exists to close the gap between models and human intelligence.
    #49 — Jason Warner & Eiso Kant, Poolsideconfidence: high

Benchmarks are cultural artifacts that steer the whole field

  • benchmarks are just memes that shape the most powerful tool ever created.
    #253 — Alex Duffy, Every.toconfidence: high
  • number one most painful thing about AI engineering today? And evaluation topped
    #137 — Barr Yaron, Amplifyconfidence: high

Open source is a strategy, not charity

  • you can take, you can download, you can run in your own infrastructure, your own devices.
    #9 — Omar Sanseviero, DeepMindconfidence: medium
  • why we do open source and how we do open source
    #528 — Devendra Chaplot, Mistral AIconfidence: high
  • building something to empower and accelerate the amazing work being done by the open source community
    #480 — Kathleen Kenealy, DeepMindconfidence: high
  • we have a dream of building a journalist model and generalist agent
    #400 — Junyang Lin, Alibaba Qwenconfidence: high