How Engineering Teams Manage Fleets of AI Coding Agents at Scale

For most of the last two years, AI in the codebase meant one developer and one assistant. You typed, the model suggested, you accepted or rejected. That model is already starting to feel small. The new shape of the work is a team supervising a fleet of autonomous coding agents, each one researching, planning, and shipping changes while humans steer from a distance. The unit of management is no longer a keystroke. It is a queue of running agents.

This post is an operations explainer, not a buyer's guide. For recommendations on which products to pick, start with our best AI coding agents guide. Here we focus on the harder question: once agents can work on their own, how does a team manage them at scale without shipping a mess?

The anchor case study: Block's BuilderBot

The clearest production example to date comes from Block. The company built an orchestration layer called BuilderBot on top of its open-source agent framework, Goose. Engineers manage a fleet of coding agents from a single Slack thread. You tag @builderbot with a short description (a bug fix, a migration across services, a new feature), and the agent goes to work. Multiple teammates collaborate in real time, watching it research, plan, and implement while they steer.

The numbers are worth sitting with. BuilderBot executes more than 200,000 operations per day and merges roughly 1,500 pull requests per week. That is about 15 percent of all production code changes at Block. The company got around 12,000 employees using AI agents in two months. Goose itself was contributed to the Agentic AI Foundation (AAIF) under the Linux Foundation, and Block's integration work on it directly inspired its collaboration with Anthropic on the Model Context Protocol (MCP), now an industry standard for connecting agents to tools and data.

Block is the case study here, not a product to evaluate. It is useful because it surfaces every management problem a team hits when agents stop being assistants and start being a fleet.

Where the agents live: chat-ops versus the IDE

The first decision is where agents run. There are two camps, and most teams use both.

In-IDE assistants live inside the editor with the developer. Tools like Cursor, GitHub Copilot, Windsurf, and Codeium sit next to the cursor and respond in the loop. The human is always present, reviewing every diff as it appears. This is high-bandwidth, low-latency work for tasks where you want tight control.

Autonomous agents live somewhere else: a chat channel, a CI runner, a background worker. Block's choice of a Slack thread is deliberate. Chat-ops turns agent work into something a whole team can see and steer, with a durable transcript of what the agent decided and why. Tools designed to run on their own, like Devin, or terminal-first agents like Claude Code and Aider, fit this mode. They take a task description, disappear for minutes, then come back with a branch and a pull request.

The practical rule: use the IDE assistant when you want to drive, the autonomous agent when you want to delegate. They are not competitors so much as different gears.

You still merge through review

The single most important guardrail is also the most boring: humans still own the merge. None of the scaled setups let agents push straight to main. Block's 1,500 weekly PRs are pull requests, which means they pass through the same review surface as human-authored code.

Review is where a fleet stays accountable. An agent can open a hundred PRs a day, but a reviewer reading the diff catches the wrong abstraction, the silently dropped edge case, or the change that passes tests but solves the wrong problem. Keeping the agent's output in PR format also means it inherits everything your team already built around review: required approvals, code owners, status checks, discussion threads. The agent does not get a special lane.

Guardrails: tests, CI, and scoped permissions

Review catches intent problems. Automated guardrails catch the rest, and they have to be in place before you let agents run at volume.

Tests and CI become non-negotiable gates. If an agent's PR cannot pass the existing suite, it does not move forward, full stop. Teams that scale agents invest heavily in test coverage first, because the agent's output is only as trustworthy as the checks it has to clear. A green pipeline is the contract.

Scoped permissions are the second layer. An agent should have exactly the access it needs and nothing more: a single repo, a feature branch, read access where it does not need to write. The blast radius of a confused agent is bounded by what you let it touch. Code search tools like Sourcegraph Cody help here too, giving agents and reviewers a precise view of how a change ripples across a large codebase before it lands.

Tool access through MCP

An agent that can only edit text is limited. The leap to a useful fleet comes when agents can reach the systems around the code: the issue tracker, the docs, the database schema, the deployment logs. That is the problem MCP was built to solve.

The Model Context Protocol gives agents a standard way to connect to tools and data sources without a bespoke integration for each one. Block's work on Goose fed directly into its collaboration with Anthropic on MCP, and it is no accident that the company behind Claude and the company running a 1,500-PR-per-week fleet converged on the same protocol. At scale, you do not want every agent reinventing how it talks to your internal services. You want one interface, governed in one place, so that granting or revoking a capability across the whole fleet is a single change.

Observability: measuring what the fleet ships

You cannot manage a fleet you cannot measure. Once agents are producing a meaningful share of your commits, you need numbers that tell you whether the output is good, not just abundant.

The two metrics that matter most are PR merge rate and revert rate. Merge rate tells you how often an agent's work is good enough to ship: a low rate means the agent is generating noise that humans have to clean up. Revert rate is the honest signal of quality, because it counts the changes that got merged and then had to be undone. A fleet that merges a lot but reverts often is not saving anyone time. Block's framing of its output as a percentage of production changes (about 15 percent) is itself an observability discipline.

Beyond those two, teams track time-to-merge, cost per task, and how often an agent gets stuck and needs a human to take over. Treat the fleet like any other production system, with dashboards and alerts, not a black box you hope is working.

When to use an autonomous agent versus an in-IDE assistant

The management question that ties all of this together is task routing. Not every job belongs to an autonomous agent.

Reach for an autonomous agent when the task is well-scoped, repetitive, or wide: a mechanical migration across dozens of services, a batch of dependency bumps, a clearly specified bug with a reproducer. These are the jobs where a fleet shines, and where Block points BuilderBot. Reach for an in-IDE assistant like Cursor or GitHub Copilot when the work is exploratory, ambiguous, or architecturally sensitive, where you want a human in the loop on every line. Our best AI coding tools guide breaks the options down by use case.

The takeaway

Managing a fleet of AI coding agents is mostly a return to old engineering discipline applied to a new kind of contributor. You gate on review, you gate on tests, you scope permissions tightly, you give agents controlled access through a standard like MCP, and you measure output with merge and revert rates. Block's BuilderBot shows the ceiling is high: 1,500 PRs a week, 15 percent of production changes, built on rails any team can recognize. The agents got faster. The job of keeping them honest stayed the same.