← All chaptersFrom Copilot to Colleague
Visual Guide

Chapter 09 · 12 min read

The AI-Native Organization

How teams and incentives change when AI becomes part of the workforce.

EVIDENCE OF SOURCE · CHAPTER 09
CHAPTER 09/2,834 words/Drafting

The biggest gains from AI do not come from giving individuals better tools. They come from redesigning the organization so delegated machine work can compound instead of colliding.

That distinction matters because many AI adoption stories still start too small. Someone buys a seat. A few engineers get faster. A product manager drafts more quickly. Support summarizes tickets with less effort. A consultant uses AI to speed up research. These changes are real, and sometimes valuable, but they do not yet amount to an . They are tool-use stories. They describe local acceleration, not institutional redesign.

An begins later, at the point where the work itself changes shape. Execution gets cheaper. Exploration gets faster. More people can create artifacts that previously required specialist intermediaries. Individual operators can direct several agents in parallel. Teams can investigate more options before committing. And almost immediately, a new set of bottlenecks appears. Review load rises. Duplicate work spreads. Private agent workflows drift out of sync with shared priorities. Standards become harder to enforce informally. Managers can no longer assume that the org chart maps cleanly to who is able to produce what.

This is the deeper lesson of the corpus. AI does not merely accelerate output. It reorganizes where scarcity lives.

Scarcity moves upward: from typing to judgment, from production labor to orchestration, from drafting to deciding which draft deserves trust, from isolated productivity to system throughput. Once delegated work becomes real, organizational design becomes the macro-scale version of design.

A Monday morning scene from the software factory

Figure 09.1/Where scarcity movesCLICK TO ENLARGE

Picture a Monday morning at Meridian, which by now has gone well past casual AI adoption.

Over the weekend, engineers launched subagents against backlog items. Product used internal tools to spin up three alternative flows for a new onboarding step. Support discovered a recurring customer complaint and had an agent draft both a help-center update and a proposed product fix. A staff engineer woke up to twelve pull requests, four synthetic eval reports, two security review requests, and a dashboard showing that an internal automation quietly touched three systems nobody had discussed together on Friday — any one of which could be hiding another admin-override regression, the throttle on the wrong path from Chapter 4, now buried under a weekend of green checkmarks.

Nobody in this scene is doing anything obviously reckless. In fact, everyone is being productive.

And that is the problem.

The organization is no longer starved for output. It is starved for coherence. Which of these changes matter? Which ones are duplicates? Which artifacts reflect the current strategy and which ones reflect a private interpretation of it? Which outputs are safe to merge, safe to trial, safe to ignore, or dangerous to trust? Who is even responsible for saying so when authorship is now half human, half machine, and spread across several functions?

That is the scene this chapter has to explain. It is where the stops being just a repo pattern and starts becoming a company problem.

Buying seats is not the same as redesigning work

Figure 09.2/Alignment debtCLICK TO ENLARGE

The easiest way to misunderstand AI transformation is to measure adoption by access rather than by whether work is routed differently. A company can purchase licenses, encourage experimentation, and still remain structurally unchanged: individuals become a little faster, but decisions, approvals, and responsibility flow through the same assumptions the organization held before. Count the seats and AI looks transformative; ask whether a single approval path has changed, and it often has not. Until the operating model bends, AI is helpful but not yet formative.

The more interesting threshold is when AI use becomes sufficiently widespread that the operating model itself starts to bend. Dan Shipper, building the AI-native company Every, puts the discontinuity as a deliberate provocation: “There is a 10x difference between an organization where 90% of engineers use AI versus one where 100% do.” The last ten percent is where compounding effects appear. Once usage becomes pervasive, local shortcuts stop being the main story. Shared expectations shift. Documentation quality matters more. Examples, standards, prompts, policies, and internal tools start acting like leverage multipliers rather than optional aids.

This is where the returns as an organizational case, not just a technical one. Earlier chapters showed a team turning a repo into an environment agents could use. Chapter 9 asks what happens after that. Someone has to own the rules. Someone has to decide which patterns become standard. Someone has to arbitrate between local experimentation and shared reliability. The workplace built for agents becomes, in effect, an institution with governance.

That is the first mark of an : it stops treating AI as a personal productivity trick and starts treating delegated machine work as part of how the company itself operates.

Cheaper execution shifts value toward judgment and throughput design

Figure 09.3/Broaden who createsCLICK TO ENLARGE

When output gets cheaper, not all work becomes equally valuable. This is the subtlety many simplistic productivity narratives miss: when generation is nearly free, the scarcity does not vanish but relocates — from making things to deciding which of them deserve trust. The rule that follows is easy to state and easy to forget: more output per person is not the same as more throughput, and an organization can produce far more while moving slower.

More things can be attempted. More variants can be generated. More tasks can be pushed into motion. But the organization still has to decide what matters, which options deserve review, which outputs are trustworthy, and where limited expert attention should go. The result is that judgment, architecture, and throughput design become even more important as raw execution becomes cheaper.

This creates new pressure on leadership roles. Product managers have to frame work more clearly because ambiguous goals produce more low-value output, not less. Senior engineers have to encode standards and review paths rather than merely embody them personally. Managers have to think in terms of queue health, bottlenecks, and quality systems, not only staffing plans. Internal platform teams become strategic because they decide whether speed compounds or fragments. Justin Reock, working on engineering leadership at DX, reframes the manager’s job as allocating judgment and attention, which stay scarce, rather than production capacity, which is now abundant. A useful test is which scarce resource each ritual rations: a standup that reports how much got produced rations the abundant thing; one that surfaces which decisions are unmade rations the scarce one.

The research on developer productivity helps here mainly as a caution. Local speed gains do not automatically improve system throughput. An engineer can open more pull requests and still make the organization slower if review queues clog, priorities scatter, or trust in the output declines. Nick Arcolano’s analysis at Jellyfish, built on some twenty million pull requests, shows the failure mode at scale: output volume rises and the dashboards light up green while the real constraint — whether the organization can review, integrate, and trust that output — goes unmeasured until it breaks. AI-native advantage therefore cannot be measured only by artifact volume. It has to be measured by the outcome — rework rate, the share of generated work that ships unreverted, time in the review queue — which is to say by whether the institution converts cheaper generation into more trusted completed work.

That is Chapter 4 at organizational scale. Evals were the control system for agents. Review capacity becomes the control system for organizations using agents.

Broader creation works only when paths to ship stay constrained

Figure 09.4/Review is the new bottleneckCLICK TO ENLARGE

One of the most provocative in the corpus is that people outside traditional engineering roles should increasingly be able to ship meaningful changes. Lisa Orr at Zapier states it as a deliberate provocation: “at Zapier we are empowering our support team to ship code.” That is directionally right. As execution becomes easier, more people can participate in creation. Support can encode recurring fixes. Product can prototype flows directly. Domain experts can shape automations without waiting for every translation layer. Organizations that refuse this broadening will leave leverage on the table.

Broader creation becomes durable only when paths to ship are constrained.

This is another place where the book’s earlier chapters should echo loudly. You cannot safely widen authorship without strengthening , specs, evals, permissions, and review. Otherwise democratized creation simply means democratized breakage. The organization becomes noisier, not more capable.

The strongest form of this argument is pro-scaffolding, not anti-democratization. More people should be able to create because the system around them makes safe creation easier. The right internal platforms give non-specialists narrow, high-leverage, well-governed ways to contribute. Templates, policy rails, staged approvals, sandboxed environments, and reusable agent workflows let broader participation coexist with stronger operational discipline.

This is where the becomes a company pattern. The “factory” is not just a coding stack for engineers. It is the broader environment that packages institutional judgment so more people — and more agents — can work inside it safely.

Review becomes the organizational bottleneck

As generation gets cheaper, validation gets more expensive. If one person can now direct several agents, produce many more candidate artifacts, or explore a much wider possibility space before lunch, the organization’s existing review structures will break first. Pull requests pile up. Specs multiply. Drafts arrive faster than domain experts can inspect them. Managers lose visibility into what is real work versus local experimentation. The company starts drowning not in effort, but in options. Zack Proser, running parallel agents on WorkOS’s Applied AI team, states the inversion plainly: “agents are not the bottleneck now and I think that’s going to increasingly be the case, but we are.” Once you hand an agent verification criteria and the tools to meet them, it loops until it does, while the human attention that has to confirm the result “still degrades under load. It’s still the hard constraint.”

This is why Chapter 9 must keep explicit continuity with Chapter 4. The hard question is no longer how to make more output appear. The hard question is how to decide what deserves trust without making humans inspect everything line by line.

A mature therefore invests in layered review. Some outputs are rejected automatically. Some are checked against and evals. Some are sampled. Some are escalated because they touch high-risk paths. Some require domain sign-off because the consequences justify it. The point is not to force universal manual review but to create a review system proportionate to the risk and volume of delegated work. Because generation is cheap, a more trustworthy verdict can come from redundancy rather than a more expensive model: Aakanksha Chowdhery describes having the models “generate multiple responses and then do majority voting,” and Leonard Tang at Haize Labs has “weaker LLMs debate each other about what the stronger model is saying” to build judging systems that beat a frontier model at a fraction of the cost. The redundancy pays off only when the voters are genuinely independent, since one model sampled repeatedly cancels noise but not shared bias.

Eric Zakariasson’s software-factory framing matters here because it shows what management becomes when many workers, human and machine, are active at once. Someone needs a board that says what is in progress, what completed, what failed, and what now needs human attention. Without that roll-up layer, leaders do not manage a system. They manage a blur.

High-stakes domains make the same point even more sharply. In legal, tax, and compliance work, review was never optional. AI raises the stakes because it can create more candidate work than experts can casually supervise. So the organization either gets much better at triage, validation, and evidence surfacing, or it fails under the weight of its own increased productivity.

Alignment debt is the new invisible tax

Private agent workflows create a subtle organizational problem. They are often individually efficient and collectively incoherent. One engineer runs six agents on one interpretation of the problem. Another runs five agents on a slightly different interpretation. Product updates a draft in a private workspace. Support discovers a pattern and patches around it locally. Everyone feels faster. Then the organization discovers duplicated work, conflicting implementations, giant review queues with little context, and outputs that are technically impressive but strategically misaligned.

This is why Maggie Appleton’s framing is so important. Describing collaborative AI engineering from inside GitHub, she names the root cause precisely: “None of our current tools give teams a shared space to discuss plans, gather the right context, and work with agents as a collective.” The hidden cost is not only wasted effort. It is .

accumulates when execution fans out faster than shared understanding. Work happens, artifacts multiply, but the common plan, relevant context, and review surfaces do not stay synchronized. Eventually the debt comes due as rework, confusion, or an exhausting burst of manual coordination.

It names a distinctively AI-native failure mode. The problem is not merely that teams need to collaborate better in the abstract. It is that agent-amplified execution allows individuals to move so quickly in private that the old informal coordination mechanisms stop keeping up.

The remedy is not to ban private leverage. It is to move alignment earlier. Shared planning, visible decomposition, clearer context packets, common review spaces, and artifact roll-ups become more important, not less, when execution is cheap. In other words, the org needs the same thing the agent needed: a better control plane.

Roles blur, but responsibility cannot

AI-native organizations do change role boundaries. Support can do more technical work. Product can produce working artifacts. Designers can prototype logic, not just screens. Engineers can spend more time on specification, review, and systems design. Domain experts can shape workflows directly. All of that is real.

But blurred creation does not mean blurred accountability.

In fact, accountability has to become sharper precisely because authorship becomes more distributed. The operative rule is simple: every path you widen for creation obligates you to name, before it opens, who owns the production path, who owns domain correctness, who owns the security boundaries, and who decides which workflows stay human-gated. Widened creation without an assigned owner is not democratization; it is an accountability gap waiting to be discovered after something ships.

This is another reason internal platforms matter so much. Good platforms do not only make work easier. They encode the transfer points between creation and accountability. They define what can be done safely by many people and what must still pass through narrower ownership structures.

The company becomes a harness for its own agents

At organizational scale, those same requirements turn into company design. An AI-native company externalizes standards, examples, specs, policies, permissions, review rules, and approved workflows so that both humans and agents can operate inside them. It packages taste into templates, governance into platforms, and institutional memory into accessible systems. It creates broader paths to create and narrower, better-governed paths to ship. It treats management not as inspection of every artifact, but as design of the environment in which good artifacts are more likely to emerge. That is what it means for the company to become a for its own agents.

The phrase should not be heard as dehumanizing. The point is not that the organization becomes a machine but that it becomes better at converting distributed intelligence — human and machine — into dependable throughput. It makes judgment reusable. It makes oversight scalable. It makes delegation legible.

This is also where the High-Stakes Colleague quietly returns one last time. In expert domains, the institution does not win by replacing professionals with unrestricted systems; it wins by building workflows in which professional judgment is focused where it adds the most value. The same principle holds in software and in management. An does not eliminate people from the loop. It redesigns the loop so human attention lands on what is most consequential.

AI-native advantage is institutional, not merely individual

The strongest companies in an AI era will not simply be the ones with the most aggressive prompting culture or the highest number of tool seats. They will be the ones that learned how to convert cheap generation into trusted throughput.

That requires broader creation, tighter review, earlier alignment, stronger internal platforms, clearer authority boundaries, and operating models that treat delegated work as a first-class organizational material. It requires leaders to notice that speed alone is not the prize; the prize is compounding.

An is therefore not just a company where people use AI a lot. It is a company that reorganized itself so that machine-amplified work becomes cumulative instead of chaotic.

That is the outer-ring lesson that completes the book’s core spine. The technical question and the organizational question turn out to be the same. How do you build an environment in which delegated work deserves trust?

That question has a quieter twin: which parts of this answer outlast the churn of tools and interfaces? That is the subject of the final chapter.

21 claims · 81 source anchors

Evidence — Source Anchors

The important transition is from suggestion to delegated execution

Open in graph
  • from helpfulness to productive
    #206 — Joel Hron, Thomson Reutersconfidence: high
  • I think they need more
    #3 — Jacob Lauritzen, Legoraconfidence: high
  • most primitives the magic happens when you combine these things together
    #138 — Sam Bhagwat, Mastra.aiconfidence: high

Harness quality is a major determinant of coding-agent quality

Open in graph
  • a good harness is really operationalized around giving the model text at the right time
    #16 — Ryan Lopopolo, OpenAIconfidence: high
  • there's so much work that has been put in uh over the last you know 20 to 30 years around the automated validation and verification of software that you build
    #57 — Eno Reyes, Factory AIconfidence: high
  • instead of micromanaging, what I'm doing is I'm scaffolding and providing context.
    #190 — Eric Hou, Augment Codeconfidence: high
  • identifying problems with the code because if there's no problems then it's probably high quality code
    #179 — Josh Albrecht, Imbueconfidence: high

Agent-ready codebases are designed, not discovered

Open in graph
  • agents MD files an open standard
    #57 — Eno Reyes, Factory AIconfidence: high
  • context deficit as the biggest blocker.
    #190 — Eric Hou, Augment Codeconfidence: high
  • a garbage codebase you're going to get
    #621 — Matt Pocockconfidence: high

Evals are a control system, not just a test suite

Open in graph
  • improvement without measurement is limited and imprecise.
    #125 — Ido Pesok, Vercel v0confidence: high
  • We still want to build reliable scalable applications and that is still hard
    #184 — Samuel Colvin, Pydanticconfidence: high
  • eval to us it's actually the same problem from a from a systems perspective.
    #628 — Phil Hetzel, Braintrustconfidence: high
  • small CLI tool that we call eval tool
    #689 — Lawrence Jones, incident.ioconfidence: high
  • designed to allow agents to leverage our eval suite files.
    #689 — Lawrence Jones, incident.ioconfidence: high
  • classic benchmark maxing.
    #746 — Ara Khan, Clineconfidence: high
  • There are right ways to use them. There are wrong ways to use them.
    #746 — Ara Khan, Clineconfidence: high

Human oversight works best as an architectural layer, not an afterthought

Open in graph
  • There needs to be human interaction for approvals or other reasons and of course they need to be able to be uh able to run in parallel for efficiency
    #167 — Preeti Somal, Temporalconfidence: high
  • dial these agency dials far up.
    #206 — Joel Hron, Thomson Reutersconfidence: high
  • maintaining a factory would require you to have an overview of the processes you want your coding agents to go through.
    #629 — Eric Zakariasson, Cursorconfidence: high

The harness is evolving from a local loop into a staged software factory

Open in graph
  • getting to a place where you can build your own like software factory
    #629 — Eric Zakariasson, Cursorconfidence: high
  • unified agent harness that will manage
    #632 — Vaibhav Srivastav & Katia Gil Guzman, OpenAIconfidence: high
  • parallel agents working together to fix
    #42 — Robert Brennan, OpenHandsconfidence: high
  • The difference with missions is that we run features serially.
    #653 — Luke Alvoeiro, Factoryconfidence: high
  • Our longest mission ran for 16 days
    #653 — Luke Alvoeiro, Factoryconfidence: high
  • We just kind of gave each role its own kind of context window.
    #691 — Ash Prabaker & Andrew Wilson, Anthropicconfidence: high
  • it's no longer about the model or the agent. It's about the process.
    #743 — Vincent Koc, OpenClawconfidence: high

The context gap increasingly includes capability packaging and progressive disclosure

Open in graph
  • doesn't have to be loaded immediately to context.
    #683 — Pedro Rodrigues, Supabaseconfidence: high
  • specifically with progressive disclosure.
    #654 — Nick Nisi & Zack Proser, WorkOSconfidence: high
  • 49% reduction of the initial load.
    #625 — Sam Morrow, GitHubconfidence: high
  • rich interactive components that render directly in the chat.
    #747 — Marlene Mhangami & Liam Hampton, GitHubconfidence: high

AI-native advantage depends on organizational coherence, not output volume alone

Open in graph
  • you'll be able to assemble agent teams that can complete tasks orders of magnitude harder than what you can complete with a single agent today.
    #653 — Luke Alvoeiro, Factoryconfidence: high
  • observing their workflows, their pain points, co-designing solutions with them
    #693 — Eoin Mulgrew, 10 Downing Streetconfidence: high
  • maintaining a factory would require you to have an overview of the processes you want your coding agents to go through.
    #629 — Eric Zakariasson, Cursorconfidence: high

Coordination is the unsolved runtime primitive for multi-agent systems

Open in graph
  • the thing that's missing for me is coordination.
    #704 — Lou Bichard, Onaconfidence: high
  • through sort of state machines, you know, by building out workflows and effectively state machines
    #704 — Lou Bichard, Onaconfidence: high
  • They step on each other's changes. They duplicate work. They make inconsistent architectural decisions.
    #653 — Luke Alvoeiro, Factoryconfidence: high
  • we have the two agents basically negotiate what done actually means.
    #691 — Ash Prabaker & Andrew Wilson, Anthropicconfidence: high

Context engineering is a primary engineering discipline, not a prompt trick

Open in graph
  • picking up the right documents and answering those questions is a really cool use case.
    #100 — Ofer Mendelevitch, Vectaraconfidence: high
  • cool load generator that Kalen wrote that lets you configure agent swarms uh and agent subtasks with very specific SLOs's
    #104 — Val Bercovici, WEKAconfidence: high
  • connect the dots with graph technology and solve problems like context engineering
    #105 — Stephen Chin, Neo4jconfidence: high
  • the right agent in the future is going to be this system that decides what type of search
    #157 — Will Bryk, Exa.aiconfidence: high

Enterprise usefulness scales with working-set quality, not corpus size

Open in graph
  • about 73% of LM customers implementing use cases say that factual accuracy is their top challenge right now.
    #100 — Ofer Mendelevitch, Vectaraconfidence: high
  • how Harvey tackles retrieval, the types of problems there are and then the challenges that come up with that all with like retrieval quality, scaling, uh security,
    #154 — Calvin Qi (Harvey) & Chang She (Lance)confidence: high
  • how to build enterprise aware agents. How to bring the brilliance of AI into the messy complex realities
    #193 — Chau Tran, Gleanconfidence: high
  • you don't need a trillion at once, you need the right million.
    #756 — Kuba Rogut, Turbopufferconfidence: high

Enterprise MCP adoption converges on gateways, blessed platforms, and a root of trust

Open in graph
  • we think that the goal for a secure this for any security team is to is to bless one platform.
    #624 — Karan Sampath, Anthropicconfidence: high
  • challenges we've faced building and scaling our remote server, how we've overcome them,
    #625 — Sam Morrow, GitHubconfidence: high
  • if we continue this pattern for hundreds or thousands of agents, we've got a pretty big security problem on our hand.
    #150 — Jared Hanson, Keycardconfidence: high

AI-native advantage is an operating-model redesign, not a procurement decision

Open in graph
  • there's a 10x difference between an org where 90% of the engineers are using AI versus an org where 100% of the engineers are using AI.
    #65 — Dan Shipper, Everyconfidence: high
  • 80% of respondents say LLMs are working well at work,
    #137 — Barr Yaron, Amplify (2025 AI Engineering Report)confidence: high
  • It's about evolving from AI features sprinkled into the product to rethinking how you plan, build, and deliver value all through an AI lens.
    #199 — From Hype to Habit (AI-first SaaS)confidence: high
  • writing code has never been the bottleneck, right? We can in uh we can increase productivity a bit by helping with code completion, but our our biggest bottlenecks are elsewhere within the SDLC.
    #62 — Justin Reock, DX (acq. Atlassian)confidence: high

Broader creation requires tighter review and governance — they rise together or the first becomes a liability

Open in graph
  • at Zapier we are empowering our support team to ship code.
    #69 — Lisa Orr, Zapierconfidence: high
  • I'm going to make the case for the AI product manager. I'm going to argue that AI expertise is really important for this role.
    #162 — James Lowe, i.AIconfidence: high
  • all these skills that you're prioritizing don't necessarily need to be one person. They can be multiple people.
    #188 — Denys Linkov, Wisedocsconfidence: high
  • I'm going to talk to you today about how I believe AI is breaking how we hire technically.
    #207 — Beth Glenfield, DevDayconfidence: high
  • the challenge becomes who do I say no to?
    #743 — Vincent Koc, OpenClawconfidence: high

Activity-based metrics misread motion as progress in AI-augmented work

Open in graph
  • these are not productivity metrics. They're useful, but you cannot just kind of use them like maximize them to maximize developer productivity.
    #79 — Yegor Denisov-Blanch, Stanford (120k devs study)confidence: high
  • I do think that AI increases developer productivity, but there's also cases in which it decreases developer productivity.
    #195 — Yegor Denisov-Blanch, Stanford (100k devs study)confidence: high
  • just plain old PR throughput. How many pull requests does the average engineer merge per week?
    #101 — Nick Arcolano, Jellyfish (20M PRs)confidence: high
  • I'm going to talk about how we pay engineers. And we pay engineers like salespeople.
    #63 — Arman Hezarkhani, Tenexconfidence: high

Review capacity is the throughput limit of an AI-native organization

Open in graph
  • this talk uh is called uh one developer, two dozen agents, zero alignment. Uh this is the case for why we need collaborative AI engineering.
    #623 — Maggie Appleton, GitHubconfidence: high
  • you should have multiple different stages where you you plan it, you produce it, you review it and you essentially follow the whole uh SLC
    #629 — Eric Zakariasson, Cursorconfidence: high
  • every software engineer becomes a code reviewer as basically their primary job.
    #54 — Max Kanat-Alexander, Capital Oneconfidence: high

Alignment debt is the AI-native equivalent of technical debt

Open in graph
  • None of our current tools give teams a shared space to discuss plans, gather the right context, and work with agents as a collective.
    #623 — Maggie Appleton, GitHubconfidence: high
  • if we believe that all of our products are for like for all time going to be probabistic, then like we probably have to figure out how this world works.
    #160 — Ben Stein, Teammatesconfidence: high
  • you kind of like frontload uh the context to the agents either through like a plan or a long spec and then you send them off
    #629 — Eric Zakariasson, Cursorconfidence: high

Cheap generation raises the value of taste and judgment rather than lowering it

Open in graph
  • software fundamentals matter now more than they actually ever have.
    #1 — Matt Pocock, AI Heroconfidence: high
  • capable of doing everything um immediately
    #6 — Tuomas Artman & Gergely Oroszconfidence: high
  • intentionally designed to put friction
    #14 — Armin Ronacher & Cristina Poncela Cubeiroconfidence: high

Once agents go parallel and autonomous, the human's verification capacity — not the agents' generation capacity — is the binding constraint

Open in graph
  • agents are not the bottleneck now and I think that's going to increasingly be the case, but we are.
    #761 — Zack Proser, WorkOSconfidence: high
  • our attention is still, you know, in meatspace, if you will, and it still degrades under load. It's still the hard constraint, essentially.
    #761 — Zack Proser, WorkOSconfidence: high
  • we just want you to look at PRs that are ready for you in GitHub.
    #758 — Joshua Snyder, PostHogconfidence: high
  • instead of you reviewing changes, if the change is pretty easy, let's just approve it with an agent and deploy it behind a feature flag.
    #758 — Joshua Snyder, PostHogconfidence: high

The gap that kills agent PoCs is the evaluation gap — no defined, continuously-measured definition of success — not the choice of model

Open in graph
  • Second is the evaluation gap.
    #767 — Sandipan Bhaumik, Databricksconfidence: high
  • Evaluation is basically specification for your AI system.
    #767 — Sandipan Bhaumik, Databricksconfidence: high

Trustworthy judgment can be manufactured from cheap stochastic generation — sample-and-vote, multi-model consensus, and debate panels beat a single expensive call

Open in graph
  • the models to generate multiple responses and then do majority voting.
    #251 — Aakanksha Chowdhery, Reflection.aiconfidence: high
  • having LMS debate each other, having the weaker LLMs debate each other about what the stronger model is saying and seeing if that makes sense.
    #116 — Leonard Tang, Haize Labsconfidence: high
  • Second tier is a multimodel consensus. So we leverage a diverse range of models such as GBT4, claude and
    #093 — Alberto Romero, Jointlyconfidence: high
  • we using multi-agent debate to get reliable State evaluation instead of using single uh
    #451 — How to Improve Your Agents (lit review)confidence: high