Why is 'Pi,' an AI coding tool with streamlined functionality, so high-performing? Its four basic functions help keep AI costs down.

Pi, a coding harness that entrusts code modification and command execution to AI, aims to improve processing costs and work efficiency by intentionally limiting its standard functions. EARENDIL, the developer of Pi, explains why its simple design is an advantage, based on performance verification using Databricks and examples of extensions developed with Shopify.

Pi, Minimal and Performant | EARENDIL

https://earendil.com/posts/pi-autoresearch-and-databricks/

AI coding tools are constantly being updated with new features such as work planning and the ability to coordinate multiple AIs. However, as the number of features increases, so do the instructions and tool descriptions sent to the AI each time. Increased input of information can lead to higher API usage costs, and important instructions may get buried in a large volume of text.

Pi adopts the policy of 'equipping only the bare minimum of tools as standard.' The only standard functions available to the AI are four types: 'read' to read files, 'bash' to execute commands, 'edit' to modify files, and 'write' to create files. The system prompt and tool definitions combined are kept to less than 1000 tokens.

To ensure that the number of features is not reduced but the range of applications remains broad, Pi allows users to add necessary functions later as extensions. Features such as sub-agents for running multiple AIs, work planning functions, and connectivity to external services are not included by default; only users who need them can install them.

In validation conducted by Databricks using a real codebase of millions of lines, the cost per task could vary by more than double depending on the harness used, even with the same AI model. Under conditions where response quality was comparable, Pi reportedly sent about one-third the context of its comparison counterparts per interaction, completing tasks in fewer trials.

The following graph shows the average cost per task on the horizontal axis and the success rate of the task on the vertical axis. The red dots and dotted lines indicate Pareto frontiers with particularly excellent cost-success rate combinations. Several configurations using Pi are included, demonstrating high success rates at relatively low costs.

As an example demonstrating scalability even with limited core functionality, 'pi-autoresearch,' developed by Shopify engineer David Cortez, is also introduced. pi-autoresearch is an extension that automatically repeats the process of changing, measuring, adopting, or discarding code based on defined evaluation metrics such as processing speed. By reverting changes that fail to improve performance, it allows for continuous trial and error while improving performance.

The following is a screenshot of pi-autoresearch running, repeatedly making code changes and measuring performance. Improved changes are classified as 'keep,' and worsening changes as 'discard.'

Other examples include Shopify seeing unit tests become 300 times faster and React component mounting processes becoming 20% faster.

The developers of the Pi explain that as high-performance AI models have become capable of understanding terminal operations and development tasks, the need for the harness to provide a large number of instructions is diminishing. They state that designing the system to add only the necessary functions, rather than including everything from the start, leads to a balance between low cost and high performance.