Kunal's Blog

We asked an AI to build the same feature 20 times.

Ten times, it got a one-line task: Add subscription billing. The other ten times, it could also read the full brief our team had written for that feature.

With the one-line task, it built the feature right on the first try 0 times out of 10. With the full brief, 10 out of 10.

Same AI. Same task. The only thing that changed was what it knew. And the way it went wrong without the brief is a lesson for anyone handing work to AI.

How we tested it

We used Claude Code, Anthropic’s AI coding assistant, and a small demo online shop we built for testing. Every try started from scratch, with the exact same instruction.

The one difference: in half the tries, the AI could also open the feature’s full brief in Stride. Nothing in the instruction told it to look. It looked every time.

To keep ourselves honest, we wrote an automatic tester before the first try. It checked 8 requirements in a real browser, and the AI never saw it.

It got the code right. It guessed the business.

The results split cleanly in two. Anything the code itself hinted at, the AI got right every time, from rejecting fake payment messages to giving members free shipping: 40 checks out of 40.

Anything that only lived in the brief, it had to guess. It got those right just 9 times out of 40.

Ten tries, one made-up name

Here’s the strangest part. All ten one-line tries called the plan “Orbit Membership”. The team had named it Orbit Plus.

The tries couldn’t see each other. Each one picked the most obvious name, so they all got it wrong in exactly the same way. These guesses weren’t random, and that’s what makes them easy to miss. The prices, meanwhile, were all over the place: $8, $5 and $9 a month.

One try even stopped to ask before building anything, because, in its own words:

…the code tells me how to build this but not what the business wants.

Claude Code, on one of the one-line tries

We told it that was everything in the task. So it guessed too: Orbit Membership, $5 a month.

When a card is declined

The brief had a simple rule: if a customer’s payment fails, they keep their perks for 3 days, then the perks pause.

Without the brief, the AI invented its own rule. Two, in fact. Six tries kept the perks going: customers still had them three and a half days later. Four tries switched the perks off within the hour.

Neither is what the business decided. Either could easily have gone live.

Feedback only fixes what it mentions

So we did what a reviewer would do. We told each try which requirements it had missed, word for word, and gave it up to three rounds to fix them.

Six tries fixed everything in one round. Four never did. And the four that never finished were the ones that had guessed the price right.

Here’s why. The tries with the wrong price were told “Shoppers can subscribe to Orbit Plus for $9 a month.” That one sentence fixed the price and, by accident, gave them the name.

The lucky four never saw that sentence. Their feedback said the account page should show “the plan”. Theirs did: Orbit Membership. So they hunted for a problem they couldn’t see, until one of them wrote:

I can’t find what QA is failing, and another blind change risks making things worse rather than fixing them.

Claude Code, after the third round

Better, and cheaper

Reading the brief took a little longer at the start. But counting everything until the feature was finished, the tries with the brief:

• used 32% less AI processing (measured in tokens)
• spent 34% less time working
• cost 21% to 34% less, depending on how quickly feedback came back

Guessing isn’t free. Wrong guesses have to be caught, explained and fixed, and some never get fixed at all.

It wasn’t a one-off

The day before, we ran the same test on a smaller feature: Add a wishlist. Same result: 0 out of 10 with the one-line task, 10 out of 10 with the brief.

Same kind of mistake, too. Every one-line try put the wishlist on a page of its own. The team had decided it belonged on the customer’s account page.

What to tell your AI

The AI was excellent at everything the code could tell it. Every mistake was a decision someone had made but hadn’t written down where the AI could see it. So before you hand work to an AI, make sure it knows:

1. Names. What is it called?
2. Numbers. Prices, limits, amounts.
3. Timing. Deadlines, grace periods, expiry dates.
4. Follow-ups. Emails, receipts, notifications.

A few honest notes

We make Stride, and we wrote the brief, so it’s no surprise the tries that could read it got it right. The surprise is how consistently the others got it wrong.

It’s also a small test: two features, one AI, 40 tries. You’d likely get the same benefit by pasting every detail into your prompt by hand. Stride just keeps those details in one place that your whole team, and the AI tools you connect to it, can read.

Where Stride fits

Stride keeps each task’s brief, requirements, designs and tests together, and lets AI coding tools like Claude Code, Cursor and Codex read them directly. Your AI has less to guess, and your team has less to re-explain.

How we ran the test

• The AI. Claude Code 2.1.286 with Anthropic’s Opus 5.5 model. A fresh session for every try: no memory, no custom settings, no web browsing.
• The instruction, word for word, every time: Implement ticket OST-2: "Add subscription billing".
• The difference. Half the tries could read the brief through Stride’s read-only connection for AI tools (MCP). All 10 did, in 3 to 8 lookups each.
• The tester. 8 automatic checks, one per requirement, run in a real browser against each finished build. Written and locked before the first try, proven on known-good and deliberately broken builds, and never shown to the AI.
• The rules. Written down before the first try and not changed. The tries alternated, one at a time, on 3 October 2026.
• Feedback. A fixed note listing each missed requirement in its own words, up to three rounds, in the same session.
• One rerun. A network outage interrupted the first attempt at the feedback rounds, so we undid them and ran them again. One undone attempt had started with a wrong note because the outage broke its scoring; we stopped it after two minutes and don’t count it. The first tries were not affected.
• Cost. As Claude Code reported it. Feedback went out only after all 20 first tries, more than an hour later (about six hours in the end, after the outage), so nine one-line tries paid about 50 cents each to reload their conversation into the AI’s short-term cache (the tenth found its cache still warm from the stopped attempt, whose cost isn’t counted). With instant feedback, the cost gap would be about 21% instead of 34%. Token counts are the same either way. Tries that never finished are counted at what they had used when we stopped them.
• The wishlist test ran on 2 October 2026 with the same rules and 5 requirements.