Blog, Pitang.
Jadley Felipe
Functional testing with AI: what changes with SDD and Spec Kit?
AI
In the latest posts here on the blog, Diego Luna explained what Spec-Driven Development (SDD) is and why developing with AI requires more than just good prompts. Later, he showed what we learned by comparing different SDD approaches, including Spec Kit, OpenSpec, and Claude Code's plan mode.
In this post, I want to dive deeper into a specific use case: creating automated functional tests with SDD.
I came to this topic through my own experience. On a project, I was generating functional tests with AI and, near the end, I switched to an SDD workflow. The gain was significant enough that I wanted to measure it more carefully. So I set up an experiment and presented the results at Intervalo Técnico, an internal meetup where we share studies and debate with colleagues from other teams. Now, I'm sharing what I learned here.
Why are functional tests good ground for testing SDD?
There are various types of software testing, and each one answers a different question:
a unit test checks whether a function does what it should;
an integration test checks whether different parts talk to each other properly, such as the back-end and the API;
a component test checks an isolated piece of the UI.
An end-to-end functional test, which was the focus of the experiment, verifies business behavior from the user's perspective. It spins up the entire application and goes through a full journey: a bot logs in as a customer, opens a ticket, and checks if it appears as open.
This type of test is the most expensive and slowest to maintain. On the other hand, it is the only one that guarantees that the user journey works from start to finish.
It is also full of implicit decisions. Which roles are involved? Which status transitions need to be covered? What should be done when the test finds a defect?
When these decisions aren't written down, the AI makes them on its own. This is exactly the problem that SDD aims to solve. That's why I thought functional tests would be a great place to see if it truly makes a difference.
How did I set up the experiment?
The idea was simple: run the same requirement twice. Once using only prompts, in a vibe coding style, and once with GitHub Spec Kit.
To keep the comparison fair, I kept everything else identical: the same project, the same feature, the same AI model (Claude Opus, with medium reasoning, running in Claude Code), and the same testing tool, Playwright. If anything changed in the outcome, I wanted to be certain that the cause was the method, not another variable.
The application: Pitang Desk
I used a personal project of mine, Pitang Desk, a help desk style ticketing platform. The back-end is in Node.js with NestJS and PostgreSQL, and the front-end is in React. I chose my own project so I would have total freedom to tweak it without depending on a real client.
The application has two roles: requester and agent. A ticket goes through several statuses: open, in progress, waiting on customer, resolved, and closed. Each role can only perform certain transitions, and the server rejects the others.
An important detail: the project was already "AI Ready." It had an agent instructions file, layer-by-layer rules, consistent code, and unit and component tests. This significantly influences the results, as I will show later.
The requirement: the ticket lifecycle
The tests needed to cover the ticket journeys: opening, servicing, conversation, internal notes, pausing and resuming, resolution, and closure. They also needed to cover isolation—meaning, ensuring that one customer cannot see another's ticket, neither in the list nor via a direct link.
I also defined a rule that applied to both sides: the tests do not alter the product. Any defect found is recorded, not bypassed.
I did this on purpose because AI tends to be "hasty." When it encounters an issue, it tends to fix it on the spot. In testing work, that's bad: the fix can mask the defect, and we lose the chance to understand what is actually happening.
How does Spec Kit organize the work?
Diego already introduced Spec Kit in the comparison of approaches, so I'll get straight to what matters here.
It's a GitHub tool that works with any agent: Claude Code, Copilot, Gemini, and others. It breaks down the work into stages, and each one generates Markdown files. The ones I used were Constitution, Specify, Clarify, Plan, Tasks, Analyze, and Implement.
The fact that everything turns into a file is more important than it seems. Every decision is recorded and can be reviewed by anyone, even those who don't read code.
What was the step-by-step process with Spec Kit?
Step 0: Constitution, the house rules
The Constitution defines the project's principles, similar to an agent instructions file. It is a distinct feature of Spec Kit, and most other SDD tools do not use this concept.
It matters because it's what the agent consults whenever it needs to decide something. If a rule isn't there, it will infer it.
Step 1: Specify, turning the requirement into a specification
Here, the requirement is turned into user stories and requirements.
Even before writing any code, the AI already raised a suspicion: perhaps the agent's action wasn't logging status changes in the history. It spotted a potential inconsistency just by reading the request.
This is the first sign of value in SDD. When the AI is forced to describe what it will do before doing it, gaps in the requirement surface early.
Step 2: Clarify, the questions no one asked
Clarify asks up to five multiple-choice questions, and each answer becomes a requirement in the specification. Since the project was already well-documented, the questions were few and spot-on.
One of them was decisive: “In terms of isolation, should we also check that the server rejects access, not just the UI?”
It sounds like a minor detail, but it isn't. If the test only verifies that the interface hides another customer's ticket, it guarantees nothing about security. Anyone could simply call the API directly using another user's session to see if data leaks.
With that answer, the test began validating both things: the UI displays "access denied" and the server responds with a 403 error, returning no data.
I also answered that the tests should cover all transition combinations per role, and that upon finding a defect, the test should record it and move on.
Here lies a core point of the experiment. In the direct prompt approach, these same decisions were also made, but by the AI, without anyone knowing. The difference matters because a test validating a decision that no one actually made still passes. It only reveals that it was validating the wrong thing after the problem has already reached production. In SDD, they were made by a person, consciously and on record.
Between steps: a browser for the AI
Before planning, I added the Playwright MCP to the project. With it, the AI gets a browser: it can navigate the application, take screen snapshots, and intercept network requests.
This changes the process considerably. The AI stops merely imagining how the application works and starts checking the specs against the running application.
It immediately found two flaws: the empty list was crashing, and the agent role was not recording status updates in the history. Without the browser, these issues would only have surfaced at runtime, after the tests were already written.
Step 3: Plan, technical planning
In the plan, the AI defined how the tests would be built:
three browsers at the same time, one per user, to simulate roles interacting in parallel;
data setup via API, which is faster and more stable than setting it up through the UI;
status verification in two places;
documentation of known defects, without trying to bypass them.
Step 4: Tasks, dividing the work
The requirement was broken down into 37 tasks, organized into 10 phases, and 17 of them could run in parallel. This division helps the AI work in smaller, verifiable chunks and allows progress to be tracked clearly.
Step 5: Analyze, reviewing before coding
For me, this is one of the most interesting stages of Spec Kit. Analyze reviews everything produced up to that point (specification, Clarify answers, plan, and tasks) looking for contradictions.
It found six inconsistencies among these artifacts. All of them were resolved before writing a single line of code.
Fixing an inconsistency in the specification costs a conversation. Fixing it after it has turned into code costs rework.
Step 6: Implement, execution
With everything defined, the AI implemented it. 36 out of the 37 tasks were completed. The only pending one was the commit, which was awaiting my authorization, as it should be.
Implementation took 12 minutes. The result was a suite of 23 tests that passed twice in a row and ran entirely in 18 seconds, thanks to Playwright's parallel workers. Running them twice matters because a functional test that only passes sometimes is useless.
There were three defects logged and zero lines altered in the product.
The tests covered very rich scenarios. One example is the internal note: the agent leaves a comment that other agents can see, but the customer cannot. The test opens both roles simultaneously and checks each screen. Other scenarios ensure that the customer sees the agent's name but never their email, and cover pausing and resuming, resolving, disagreeing, reopening, and closing.
And without SDD, what happened?
With the direct prompt, the AI didn't ask any questions. It went straight to implementation, reviewed its own work along the way, found a few defects, and delivered.
The outcome worked too: the tests passed and covered what was requested. But a passing test is not the same as a protecting test. The three business decisions that went through me in Spec Kit (checking server rejection, covering all transitions per role, recording the defect instead of bypassing it) were made here by the AI alone, and I would only find out which path it chose by reading the code. The difference lies in how that delivery was built and what remained documented.
Without SDD, the rules remained hidden in the code: in test names, in comments, and in a single file. In a small project, that's a minor nuisance. In a large project, or when a test breaks six months later, the AI—or the next developer—will have to sift through the code to figure out why each test exists and whether the rule it validates is still valid. That costs time and tokens, and sometimes costs the test itself: without knowing the intent, the tendency is to tweak the assertion until it passes.
With Spec Kit, the tests were organized by journey, and each requirement can be traced back to the test that validates it.
What do the numbers show?
I measured only the AI processing time, without counting the time I spent doing other things in parallel:
Time: 19 minutes with direct prompt and 29 minutes with Spec Kit, roughly 50% more;
Tokens: nearly five times more with Spec Kit;
Decision points with human participation: 1 with direct prompt and 12 with Spec Kit;
Business decisions confirmed by a person: none with direct prompt and 3 with Spec Kit;
Inconsistencies fixed before coding: 6 with Spec Kit;
Defects found before writing tests: 2 with Spec Kit.
Summing up the balance sheet: ten extra minutes of processing and about five times more tokens bought six inconsistencies and two defects resolved before any code existed, three business decisions confirmed by a person, and a suite that covers more than what was asked. A single one of those defects reaching production would cost more than that.
As for coverage, both approaches delivered 100% of what was requested. But I would say SDD reached about 130%.
With the direct prompt, the AI was literal: I asked for A, B, and C, and it did exactly A, B, and C, nothing beyond what I already knew was needed. With SDD, it challenged me. It tested the happy path and also what happens when the user does something wrong. These additions weren't random; they all made sense for the product.
There are also gains that don't show up in the numbers. One is the cleanup of test-generated data, even when execution is interrupted, which was suggested by Spec Kit. Another is versioned documentation that non-technical people can review. This allows, for example, validating a business rule with the client without showing a single line of code.
Is SDD equally worthwhile in every scenario?
Not to the same degree in every scenario. And that's precisely what I wanted to measure: where the extra cost easily pays for itself and where it outweighs the return.
Before the main experiment, I ran a smaller test: a simple holiday registry, a basic CRUD with date and description.
In terms of functional coverage, the result was almost a tie: both approaches delivered the same four journeys, with similar code structure. The difference showed up in the quality of what was delivered and in the cost.
SDD still brought gains: server-side validation, database cleanup, date collision detection in parallel tests, and documented caveats for me to decide on. But these were much smaller gains than in the ticket lifecycle.
The explanation is simple. The holiday registry had no decisions to be made. The requirement was already clear, the rules were few, and the feature was small. In that scenario, the extra cost of the full workflow is harder to justify. Still, it's worth noting: even here, SDD provided server validation and database cleanup, which the direct prompt did not deliver. The return was smaller, not nonexistent. The question shifts from "should I use SDD or not" to "how much of the workflow does this requirement deserve."
We see this outside the experiment as well. Here at Pitang, we have a project where SDD was heavily used at the beginning. At that stage, business rules were still being defined, the architecture was under construction, and every requirement brought decisions that needed to be recorded.
Over time, the project matured. The specifications, rules, and accumulated decisions became part of the repository itself, and the AI grew very familiar with the project's characteristics. Today, the team doesn't need to run the full workflow for every day-to-day requirement, as much of what SDD helped clarify is already written down and the agent knows where to find it. Requirements that introduce new decisions go through the workflow again.
This shows that the intensity of SDD is not fixed. It makes the most sense when there is still uncertainty. As the project matures, the knowledge produced during that phase becomes the context that enables lighter work. Rather than waste, the initial effort is precisely what makes it possible to skip the full workflow later. The tokens spent on the spec are not a sunk cost: they are the context that makes all subsequent requirements cheaper.
As Diego showed in the previous post, there are also middle grounds, such as plan mode, which plans the requirement at a much lower cost than a full SDD workflow.
When should you use SDD in test creation?
Based on the experiment, SDD yields the most when:
there are state and role rules, such as transitions, permissions, and isolation between users;
there are real decisions for the person operating the AI to make, not just tasks to execute;
other people will continue the work later. At Pitang Labs, switching between projects is frequent, and a repository with recorded specs and decisions lets newcomers connect their agent and find context ready to go;
the delivery needs to be auditable, because history ceases to be just the commit and begins to include all the decisions in the process.
It yields less when:
the feature is small, like a single screen without complex rules;
the requirement arrives exceptionally well-written;
it's a small prototype. A caveat here: many proofs of concept that come into the Labs are already of a size where SDD makes sense.
Note that "yielding less" is not "yielding nothing." The second case, in fact, is SDD done upfront: a requirement that arrives well-written is already a specification. And even when the full workflow isn't worth it, the alternative doesn't have to be a direct prompt. A plan mode or a lean Specify + Clarify can capture decisions at a fraction of the cost.
What did I learn from the experiment?
1. Make the project AI Ready
Project instructions, well-defined rules, and documentation make a difference in any approach. The more context the agent finds, the less it needs to infer, and the fewer doubts arise during the process.
2. Use SDD where there are decisions to be made
Changing a button's color from green to red doesn't require a specification. Restructuring a screen, altering a flow, or modifying permission rules does. The complexity of the decision is what justifies the extra cost. And in functional testing, almost every requirement carries decisions: which roles, which transitions, what to do with discovered defects. That's why, in this type of testing, SDD tends to be the rule, not the exception.
3. Give the AI a browser
I used Playwright's MCP, but there are other options, such as browser extensions for agents and computer use features. What matters is that the AI sees the UI, not just the code. That is what allowed finding defects before writing the tests.
4. Defects found are recorded, not bypassed
This applies to both SDD and vibe coding. Ask the AI to document the issue, not to fix it.
A practice I like is writing a test that passes with the bug. When the fix is deployed, that test will fail and serve as a reminder to update the test as well.
5. Favor instructions that work with any agent
A tip that came from Diego: Claude Code also reads the AGENTS.md file, a standard that Codex, Copilot, and Gemini also recognize. With a single instruction file, the project stays independent of the AI tool used.
Testing with AI also requires method
The experiment confirmed something this series of articles has been highlighting: SDD isn't about using more tools; it's about making conscious decisions.
With the direct prompt, the AI delivered tests that work. With SDD, it delivered tests that work and explain why they exist, who decided each rule, and what remains pending. In large projects that will change hands and require reliability, that difference is worth the extra cost. And that cost isn't wasted: specs become the context that makes subsequent requirements cheaper. That's why, at Pitang, we treat specification as part of the process of developing and testing with AI, not as an optional step. What we adjust from project to project is the weight of the workflow, not its presence. Tests that merely pass provide a false sense of security; tests that explain why they exist provide real security.
Here at Pitang Labs, we continue experimenting with tools and approaches to understand where each makes the most sense, and sharing what we learn.
