Google engineers talk about the '70% problem' of AI-assisted coding: Why aren't products improving even though AI has skyrocketed productivity?
Adi Osmani, Head of Developer Experience at Google Chrome and an expert on AI-assisted development, explored why AI has made engineers so much more efficient yet we haven't seen any noticeable changes to commonly used software products.
The 70% problem: Hard truths about AI-assisted coding
◆Two types of AI-assisted engineers
According to Osmani, developers who use AI to code can be broadly divided into two types: '
First of all, boostrappers are people who start new projects from scratch using AI tools such as 'Bolt', 'v0', and 'screenshot-to-code', and have the following characteristics:
-Start with a design or rough concept.
- Generate a complete initial codebase using AI.
- Create a working prototype in hours to days.
- Focus on rapid validation and iteration.
For example, Osmani saw one developer use Bolt to quickly turn a design from the design tool Figma into a working web app. It's not a production version yet, but it's practical enough to gather early feedback.
On the other hand, iterators are people who use tools such as 'Cursor,' 'Cline,' 'Copilot,' and 'WindSurf' in their daily development workflow, and Osmani believes that although they are not flashy, they have the potential to bring about great innovation.
Iterators are unique in that they use AI for the following purposes:
- Code completion and suggestions.
- Complex
Testing and documentation.
- Use AI as a ' pair programmer ' to solve problems.
Both can dramatically speed up development, but there are pitfalls and hidden costs, Osmani said.
Pitfalls of AI-assisted solutions
When you watch a senior engineer working with an AI tool, it may seem like magic, but if you look closely, you'll see that they don't just accept the AI's suggestions; they always do the following:
- Refactoring generated code into more compact and focused modules.
-Add processing for edge cases that the AI misses, i.e. extreme cases.
- Added comprehensive error handling.
- Enhancements to type definitions and interfaces.
- Examination of architectural decisions.
In other words, this work can be said to be 'applying the engineering wisdom cultivated through years of hard work to shape and limit the output of AI.' In other words, veteran engineers use AI to accelerate implementation while also mobilizing their own expertise to maintain the maintainability of the code.
Meanwhile, junior engineers often overlook these steps and simply accept the AI's output, resulting in what Osmani calls a 'house of cards' -- code that looks good but doesn't work.
◆ The '70% problem' in AI's learning curve
From these experiences, Osmani realized a seemingly paradoxical fact: AI tools are more useful to experienced developers than to novices.
Osmani isn't the only one to point out this paradox. Peter Yang, a writer who writes an AI newsletter, wrote in a social media post, 'As a non-engineer who has been coding with AI, I can honestly say that AI can help us do 70% of the things we want to do, but the other 30% is a pain.'
Honest reflections from coding with AI so far as a non-engineer:
— Peter Yang (@petergyang) December 1, 2024
It can get you 70% of the way there, but that last 30% is frustrating. It keeps taking one step forward and two steps backward with new bugs, issues, etc.
If I knew how the code worked I could probably fix it…
This 70% problem manifests itself as a game of cat and mouse where an AI tool finds a small bug in code, then when you try to fix it, something else breaks. Experienced engineers can spot the mistakes in the output of the AI tool, but beginners get caught in an infinite loop. What's more, beginners relying on AI often lack the ability to learn debugging skills or basic patterns.
◆How to interact with AI
Osmani isn't criticizing the use of AI tools in and of themselves: He says that the non-engineers he's seen who are the most adept at using AI take the time to understand the output of the AI, using it as a learning tool rather than a code generator.
Having seen many teams use AI, Osmani offers the following recommendations for those looking to get started with AI-assisted development:
Use AI for clearly defined, independent tasks, and build larger features incrementally by carefully reviewing every line of generated code.
- Split everything into small files, keep the interfaces between components clear, and document the boundaries of each module.
- Use AI to accelerate decisions rather than relying on it, maintain engineering standards, and don't leave code that doesn't feel right.
Summary
According to Osmani, it is not the speed of coding that has become the bottleneck in software quality, and he believes this is the reason why AI has not dramatically improved software.
Osmani concluded by saying, 'Remember, the goal is not to write more code faster, but to build better software. Clever use of AI can help us achieve this, but what 'better' means and how we achieve it is still up to humans.'
Related Posts:
in Software, Posted by log1l_ks