⏱️ Lectura: 10 min
Databricks ran the same model, at the same reasoning level, through three different code harnesses on its multi-million-line codebase: the one that won on quality and cost less was the simplest of the three, the minimalist Pi harness.
📑 En este artículo
The result contrasts with the dominant trend in the industry: while several companies keep adding more layers, more orchestration, and increasingly longer prompts to their coding assistants, Pi bets on the opposite. That contrast is at the center of a recent analysis by Earendil on the Databricks benchmark and an extension built by Shopify.
TL;DR
- Databricks tested Pi, Claude Code, and Codex on its multi-million-line codebase using the same model and reasoning level.
- Pi combined with Claude Opus 4.8 in xhigh mode achieved the highest pass rate and lowest cost among the three harnesses.
- Pi’s system prompt and tool definitions take up fewer than 1,000 tokens; the harness ships with only 4 tools out of the box.
- Databricks measured up to double the cost per task between harnesses running the same model, with equivalent quality.
- Pi sent roughly 3 times less context per turn than the harnesses it was compared against, according to the report cited by Earendil.
- Shopify built pi-autoresearch by asking Pi, in natural language, to create the extension from its own documentation.
- Shopify reported unit tests running 300 times faster and React component mounting 20% faster.
- Anthropic cut Claude Code’s system prompt by 80%, a sign that models now need fewer native instructions.
What happened: the minimalist Pi harness is born
Pi is an AI coding harness (the software layer that connects a language model to the editor, the terminal, and the rest of the environment) that ships with just 4 tools out of the box. Its system prompt and the definitions of those tools add up to fewer than 1,000 tokens total, according to Earendil’s analysis. The bet is that most of the work can be handled with the basics, and if a team needs something more specific, they build on top of it.
That approach came under scrutiny when Databricks published its own study, “Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebase.” The goal was to find out which coding agents perform best on real tasks and how that performance varies with price. To avoid the bias of public benchmarks, which Databricks says are already oversaturated, the team built its own task set based on the daily work of its own engineers.
The results left no doubt: “the harness a model is called from dramatically impacts cost and quality,” Databricks wrote, adding that “in many cases, simple harnesses like Pi performed better on our workloads.” Combined with Claude Opus 4.8 in xhigh reasoning mode, Pi achieved the highest pass rate of the three harnesses evaluated, at a significantly lower cost than Claude Code and Codex.
Context and history
The logic behind adding layers makes sense at first glance: AI made writing code cheap, so many companies started building increasingly larger tools in search of better performance. Longer prompts, more orchestration, more layers of abstraction. The problem is that this complexity also makes every run more expensive, because each layer adds tokens the model has to read on every turn.
Just a year ago there was a reasonable argument in favor of native harnesses (those built by the same lab that trains the model): the idea was that the model was trained around that specific harness and therefore performed better within it. That argument has weakened. Today’s frontier models understand a generic terminal environment well and know how to act within it without extensive native instructions.
One sign of that shift: Anthropic cut the system prompt of Claude Code by 80%, according to Earendil’s analysis. The industry’s focus is shifting from “how native is the harness” to “how well does it manage context to avoid redundancy.” Models need a clear interface to the environment, not a harness that wastes context repeating information.
Technical details and performance
The cost difference Databricks measured didn’t come from using a different model: they ran the same model, at the same reasoning level, through all three harnesses. Even so, “cost per task differed significantly, more than double in some cases, while quality stayed the same,” the team reported. Databricks calls this Pi’s “context discipline”: “Pi sent roughly 3 times less context per turn. It managed context better, keeping a tighter working set and finishing tasks in fewer runs.”
💭 Key point: Databricks sums up Pi’s advantage as “context discipline”: it’s not about having fewer functions for its own sake, but about not paying in tokens for functions the current task doesn’t use.
| Harness | Built-in tools | System prompt | Result in the Databricks benchmark |
|---|---|---|---|
| Pi | 4 | Fewer than 1,000 tokens | Higher pass rate, lower cost per task |
| Claude Code | Several, with additional orchestration | Larger, though recently cut 80% | Lower pass rate, higher cost than Pi |
| Codex | Several, with additional orchestration | Larger | Lower pass rate, higher cost than Pi |
That detail matters because an agent’s real cost doesn’t depend only on the model’s per-token price: it depends on how many tokens need to be sent each turn and how many turns it takes to finish the task. According to Earendil’s analysis, a stronger, more expensive model run with an efficient harness can end up costing less than the reverse combination. It’s the same pattern already seen at the model level (running complex flows on Haiku 4.5 turned out more expensive than on Sonnet 4.6 when code execution was involved, because the agent needed more turns to complete the task), but now it repeats at the harness level too.
How to try it
Pi isn’t just minimalist: it’s also self-extensible. Instead of the provider trying to anticipate every possible workflow and pack it in out of the box, Pi reads its own extension documentation and builds a new workflow when asked in natural language. That’s how Shopify built pi-autoresearch, as David Cortés describes on Shopify’s engineering blog: he asked the agent directly to create the extension, and the harness started building it from its own documentation.
Pi, create an extension for Autoresearch: I want an autonomous loop
that optimizes a measurable target (build time, test speed,
render time) by trying changes, measuring the result, and
automatically discarding regressions.pi-autoresearch is an autonomous optimization loop: when you ask for a change, it runs experiments to find out what works and what causes regressions. As long as the goal is measurable, the system can keep discarding regressions and improving on its own. Shopify reported unit tests running 300 times faster, React component mounting 20% faster, reduced build times across several projects, and even performance improvements in pnpm.
Before adding a tool to your own harness, it helps to measure how much fixed context it adds. A reproducible method: count the tokens in the system prompt and tool definitions with a tokenizer.
import tiktoken
encoding = tiktoken.get_encoding("cl100k_base")
with open("system_prompt.txt") as f:
prompt = f.read()
tokens = encoding.encode(prompt)
print(f"Tokens in the system prompt: {len(tokens)}")This same method (counting tokens in the base prompt and the tool set) is what allows you to compare the context discipline of two harnesses before choosing one for a team: the less fixed context each turn carries, the more token budget remains available for the project’s actual code.
Impact and analysis
Databricks’s finding about the minimalist Pi harness separates two variables that used to be lumped together: the model and the harness. Until now, much of the conversation about coding agents revolved around which model to use. The study shows that the choice of harness can weigh as much as that decision, even when running exactly the same model.
flowchart TD
A["Model: Opus 4.8 xhigh"] --> B["Minimalist Pi harness"]
B --> C["4 base tools"]
B --> D["Extension: pi-autoresearch"]
D --> E["Autonomous optimization loop"]
C --> F["Task completed"]
E --> F💡 Tip: before adding a new tool to your own agent, ask yourself whether it truly earns its place: every extra tool definition is fixed context sent on every turn, even in tasks where it isn’t used.
There’s also a reading about extensibility. Pi doesn’t compete by offering more built-in functions, it competes by offering less friction to build your own. That shifts the responsibility of anticipating the workflow from the provider to the team using it, something that worked well for Shopify and Databricks, but that can be a real disadvantage for small teams without the time or engineering capacity to build their own extensions.
What’s next
Earendil also mentions that local models are advancing fast and describes them as very promising. If that trend holds, a lightweight harness like Pi, which doesn’t depend on heavy orchestration, has an additional advantage: less fixed infrastructure to port when the model runs on a team’s own hardware.
The other development worth watching is whether more labs repeat the prompt cut Anthropic made with Claude Code. If frontier models keep improving their ability to operate in generic terminal environments without extensive instructions, the advantage of native, heavy harnesses should keep shrinking, and studies like Databricks’s, which isolate the harness as a variable, will likely become more common.
📖 Summary on Telegram: View summary
Try it yourself: measure how many tokens your own code harness’s system prompt and tool definitions take up with the tiktoken snippet above and compare that number against the fewer than 1,000 tokens reported for Pi.
Frequently Asked Questions
What exactly is the minimalist Pi harness?
It’s a software layer for AI-assisted coding that connects a language model to the development environment using only 4 built-in tools, with a system prompt of fewer than 1,000 tokens, as described by Earendil.
Why does Databricks say the harness matters as much as the model?
Because it ran the same model, at the same reasoning level, through different harnesses, and found cost differences of more than double between them with equivalent quality.
What is pi-autoresearch?
It’s a Pi extension built by Shopify: an autonomous loop that tries changes, measures the result against a measurable goal, and automatically discards regressions.
Which model did Databricks test the Pi harness with?
The study tested it alongside Claude Opus 4.8 in xhigh reasoning mode, the combination that achieved the highest pass rate at the lowest cost among the harnesses evaluated.
What does it mean for Pi to be self-extensible?
That the harness itself reads its extension documentation and builds a new workflow when asked in natural language, without the provider having to anticipate every use case out of the box.
What’s the main downside of a minimalist harness like Pi?
That it shifts the responsibility of building the tools each team actually needs onto that team, something that worked for Shopify and Databricks but requires in-house engineering capacity.
References
- Earendil: Pi, Minimal and Performant: the analysis comparing the Pi harness against Claude Code and Codex using the Databricks study.
- Databricks: author of the study “Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebase” cited in Earendil’s analysis.
- Shopify Engineering: the blog where the construction of the pi-autoresearch extension was described.
- Claude Code (Anthropic) on GitHub: the harness whose 80% system prompt cut is mentioned as context for the shifting trend.
📱 Enjoy this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.
Imagen destacada: Foto de Bernd 📷 Dittrich en Unsplash
0 Comments