⏱️ Reading time: 10 min
At the Claude Code team, Claude writes and reviews about 90% of new code, according to Addy Osmani on Anthropic’s official blog. At that volume, the question stops being how much code the AI writes: it becomes how AI code verification actually works behind that code, without a human reviewing every line before it merges.
📑 En este artículo
- What AI verification loops are in Claude Code
- From one supervised agent to hundreds in parallel
- The three types of tests the team recommends building
- What this means for a team already running agents in production
- Comparison: other Claude Code skills and what they don’t say about their own verification
- When this approach isn’t enough
- Frequently Asked Questions
- References
What AI verification loops are in Claude Code
A verification loop is the mechanism that gives the agent concrete evidence of whether its work actually worked, instead of accepting its own claim. According to the Claude Code team, this is what makes it possible to delegate more work to a model: without verification loops, you can only run one or two agents at a time, because everything has to be reviewed by hand.
With a loop that produces evidence (a real run, a comparison of results, a verdict against a rubric) teams can scale to dozens or hundreds of agents working in parallel. The difference doesn’t depend on the model: it depends on whether something confirms the result without relying on the agent’s own claim that it works.
From one supervised agent to hundreds in parallel
The team itself measured the effect of adding a separate verifier. In internal tests on structured file generation, a grader that evaluated Claude’s work against a rubric improved task success by up to 10 points over a standard prompting loop, with the largest gains on the hardest problems, according to Anthropic.
What used to be expensive to verify (tests, benchmarks, specs, real end-to-end runs) is now cheap to ask an agent to produce. Like code itself, an agent generates tests and benchmarks quickly; once generated, they run like any other software, so they can be repeated many times without spending extra tokens on each run, the team explains.
The loop can also close backward. If an agent repeatedly fails a test that checks logs don’t leak user data, that pattern turns into a hook or an explicit rule in the repository’s CLAUDE.md, which keeps the same mistake from repeating on the next run.
⚠️ Watch out: a flaky test teaches the agent to just rerun it instead of fixing the code. A slow test reduces how many tests it can run per batch, which makes it harder to pinpoint what broke. That’s why heavy tests should run right before committing a change, not on every iteration.
The three types of tests the team recommends building
Out of every possible type of test, the Claude Code team highlights three for their ratio of cost to real evidence: end-to-end, property-based, and differential. Each one answers a different question about the code the agent wrote.
flowchart TD
A["Agent writes code"] --> B["E2E, property, or differential test"]
B --> C{"Passes with evidence?"}
C -->|"Yes"| D["Change gets merged"]
C -->|"No"| E["Hook or rule in CLAUDE.md"]
E --> A
End-to-end tests: the check you can’t fake
If a team has to pick just one type of test, the team recommends end-to-end: they describe real user flows, like adding a product to the cart, paying, and reaching the confirmation screen. They’re hard to game because the whole feature either works or it doesn’t; they have a definitive success criterion and can say exactly what broke, according to the article.
The difference from a unit test is concrete. A unit test might show something like “Error: Timeout after 30 seconds,” while an end-to-end test can show a screenshot with the exact description of what failed. Unit tests are faster and less flaky, but you can pass a unit test and still break the feature, because neither the agent nor the test understands the full task.
For this to work, the agent needs “hands and eyes” on the real system: a skill that can start, drive, and inspect the application using tools like CDP, simulators, or API harnesses. A feature map works both as a guide and as a grading rubric, since it typically lists features from the user’s point of view, with routes, keyboard shortcuts, selectors, and expected results.
Property-based tests: thousands of attempts to break a rule
A property-based test starts from a claim you believe can’t fail, and uses a library (fast-check for JS/TS, Hypothesis for Python, proptest for Rust) to generate thousands of inputs that try to prove it wrong. The Claude Code team lists four properties that show up in almost any project: round-trip (encoding and then decoding something returns the same value), invariants (something that must always hold, like a balance that’s never negative), idempotency (running a migration or a retry twice has the same effect as running it once), and equivalence (an optimized version produces the same result as a simple but correct one).
These properties are especially useful in rewrites, when the original spec has drifted away from actual production behavior and stops being a reliable source for verifying the new code.
Differential tests: the old system as referee
Differential tests perform best in large migrations where functionality is expected to stay the same: the old and new systems run on the same inputs and the results get compared. Any difference is a bug or a behavior change that needs review. The key, according to the team, is to use randomly generated inputs or a sample of real production traffic, normalize irrelevant factors like whitespace, and keep a list of already-approved changes for cases where the difference is intentional.
What this means for a team already running agents in production
For a platform or backend engineer at a startup already running Claude Code, Cursor, or a similar agent against a real repository, the bottleneck for parallelizing isn’t usually the model: it’s whether a verification loop with real evidence runs against every change before it merges.
The concrete decision before raising the number of concurrent agents is to audit two things. First, whether every critical product flow has at least one real end-to-end test, not just a unit test, and whether the agent has “hands and eyes” on the system (via CDP, a simulator, or an API harness) to run it. Second, whether recurring failures have already become an explicit rule in CLAUDE.md or a hook, instead of still depending on someone noticing them by hand each time.
Without those two elements, adding more agents in parallel multiplies the volume of unverified changes, not the actual delivery speed.
Comparison: other Claude Code skills and what they don’t say about their own verification
Searching “Claude Code” on GitHub, beyond this article, turns up a curious result: several of the highest-starred projects built on Claude Code are skills for tasks very different from programming. None of the three that showed up in this research include, in the public description the research gathered, an explanation of how they verify their own output, unlike the detail the Claude Code team publishes about its own.
| Project | What it does | Stars (as of 2026-11-17) | Does the research mention its verification? |
|---|---|---|---|
| geo-seo-claude | SEO/GEO skill for Claude Code: citability scoring, AI crawler analysis, brand authority | ~11,700 | No |
| vibe-wise | Plugin for Claude Code or Codex that helps you learn to program while the AI writes the code | ~4,400 | No |
| ai-legal-claude | Legal assistant with 14 skills: contract review, risk analysis, NDAs, compliance | ~1,800 | No |
None of the three is a direct competitor to end-to-end, property-based, or differential tests: they’re skills for business tasks, not testing frameworks. But the comparison still holds, because thousands of developers install these skills without the publicly available information explaining how anyone verifies that the contract, the SEO report, or the exercise they generate is correct. The Claude Code team, at least in what it publishes, is the exception within its own ecosystem, not the norm.
When this approach isn’t enough
The Hacker News discussion around this article raised two objections worth adding before copying this pipeline as is.
User sashank_1509 argues that ten or twenty experienced programmers, working for a year, would end up with a better version of Claude Code, without the thousands of bugs that get fixed and broken again and without the trillions of tokens spent. That’s an opinion, not a verified data point, but it points at something real: the rubric and the three types of tests compensate for the agent’s lack of reliability, they don’t eliminate it. A repo with business logic that’s hard to formalize into invariants will struggle to get value out of property-based tests, precisely because that logic is the part nobody finished sorting out.
Duskdozer points to something more uncomfortable about incentives: deterministic scripts that run on your own CPU don’t cost tokens, so companies that bill by the token don’t have the same incentive to promote them over tests generated and graded by an agent. Before adopting a rubric-based grader for every change, it’s worth comparing its token cost against an equivalent deterministic script (a linter, an assertion, a hash comparison) that solves the same check without depending on the model.
There are also technical limits built into the approach itself. Differential tests only work when an old system exists to act as referee, so they don’t apply to a brand-new feature with no precedent. Setting up the “hands and eyes” infrastructure for end-to-end tests (CDP, simulators, API harnesses) carries a setup and maintenance cost that a small team may not have time to pay.
Frequently Asked Questions
What’s the difference between an end-to-end test and a property-based test?
An end-to-end test confirms that a complete user flow works from start to finish on the real system. A property-based test doesn’t test a flow: it generates thousands of random inputs to try to break a specific rule, like an encoding-and-decoding function always returning the original value.
When is a differential test better than an end-to-end test?
When there’s a migration or a rewrite where nothing is expected to change for the user. The old system acts as referee: it gets the same inputs as the new system, and any difference in the output is suspect.
What exactly does the grader or rubric from the Claude Code team do?
It’s a verifier separate from the agent that wrote the code: it evaluates the result against explicit criteria, instead of accepting the agent’s own claim that the task went well. According to Anthropic, adding that step improved task success by up to 10 points on structured file generation.
References
- How the Claude Code team thinks about verification: article by Addy Osmani on Anthropic’s blog (November 16, 2026) covering the three types of tests and the rubric-based grader case.
- geo-seo-claude: SEO skill for Claude Code cited in the comparison, with about 11,700 stars as of November 17, 2026.
- vibe-wise: learning plugin for Claude Code cited in the comparison, with about 4,400 stars as of November 17, 2026.
- ai-legal-claude: legal skill for Claude Code cited in the comparison, with about 1,800 stars as of November 17, 2026.
- Comment by sashank_1509 on Hacker News: community objection about the volume of bugs and tokens spent.
- Comment by duskdozer on Hacker News: objection about the economic incentive behind token-based tests.
Featured image: Foto de Danial Igdery en Unsplash
How each article is made (sources, testing and editorial criteria): How we work.
Did it work for you? Got a different error? Say so below: questions get answered and help the next reader.
Leave a comment
0 Comments