Designing for beginners in a beginner industry
Foundational research on how companies build AI agents. Even the best engineering teams turned out to be beginners at it, and that changed who the product was for.
TL;DR
Ask a staff engineer at a leading company how they build an AI agent, and in 2025 the answer was some version of: we’re still figuring it out. The product team knew how to support agents once they were running. It was less clear how to help the people still building them, even though customers kept asking. I led a foundational study to map, step by step, how teams build an agent. The main finding was that building agents is new territory, even for the most sophisticated companies. Nobody had shared practices yet, and teams wanted tools made for the job. That shifted the product toward people who are still learning, and it surfaced a new feature opportunity.
The big picture
From “which features?” to “who is this for?”
The original question was about features: what does someone building an agent need to see? I pushed to widen it before anyone committed to a feature list, and I’m glad I did. Even strong engineering teams were beginners at building agents, because the whole industry was. That changed the question from which features to which audience, and a product for beginners in a beginner industry looks very different from one built for experts.
Where teams got stuck
Teams struggled most with three things: knowing whether their agent was doing a good job, keeping visibility once an agent went live, and working without any shared way to build and test. Everyone was inventing their own approach, which was slow and hard to hand off.
The gap nobody was filling
The clearest unmet need was help with the part everyone was doing by hand. That, plus the lack of any standard way to do it, is what turned a feature study into a new feature opportunity.
From finding to strategy
From a feature list to a product strategy
What I almost missed
I thought I had the wrong participants
I built the study around senior AI engineers at leading companies, which is the usual choice for research on a developer tool. A few interviews in, something sounded off: even staff engineers talked about building agents in tentative, trial-and-error terms. My first instinct was that I needed stronger participants. If I’d followed it, I would have missed the finding, because they weren’t experts for the simple reason that nobody was yet. Their uncertainty was the data. The study stopped being about our sample and became a map of where the whole industry stood, and that’s where the beginner-first strategy came from.
Before I share any study, I review the findings for accuracy and make sure each one is backed by the data. If it isn’t, it changes.
Impact