In the previous article, we explained what Spec-Driven Development (SDD) is and how the methodology fits into the way of working we are building at Pitang.

To develop with AI, someone needs to understand the problem, define the requirements, guide the implementation, and check whether the result satisfies the client. It is to organize this work that we use SDD in our agentic development pipeline. 

Here at Pitang, we have been testing different tools and approaches to discover how to apply SDD to demands of varying sizes. Part of what we learn circulates in the Technical Break, an internal meeting where someone studies a topic in depth and presents the results to colleagues from other teams, who ask questions, disagree, and share what they have seen in their own projects. 

It was there that I presented a study I conducted at Pitang Labs comparing SDD approaches in AI-assisted development tasks. I wanted to know if processes with more specification and planning steps deliver better results. To do this, I measured execution time, token consumption, and code quality. Below, I explain how I set up the experiments and what I found. 

How did we put different SDD approaches to the test?

I created a project management application with a Kanban board. I wanted something relatively simple, but with situations similar to those of a real-world system. 

I defined six tasks, distributed across three categories: 

• Bug fixes: a timezone error and a caching issue where the user retained access even after having their permission revoked. 

• New features: project archiving and notifications implementation. 

• Refactoring: changes to the existing code organization to facilitate maintenance. 

Some tasks had strict requirements, while others left room for interpretation. The request for notifications did not state which events should trigger an alert or who should receive it. This gap ended up being useful, because it showed what each approach did when a decision was missing. 

Which tools and approaches were compared? 

The study included Spec Kit, OpenSpec, Compozy, and Superpowers, in addition to other approaches based on instructions and planning stages. Each organizes the specification, clarifying questions, execution, and review in its own way. 

As a reference, I used Claude Code's planning mode, which I called the baseline. In it, the AI proposes a plan, which can be reviewed and adjusted before implementation. The result undergoes the same checks as the other approaches. 

The baseline is also a form of SDD, but leaner. The study therefore compares different levels of structuring within SDD itself. 

How did we evaluate the results? 

I used three criteria to compare the approaches: 

• Execution time: how long each approach took to complete the task. 

• Token consumption: the computational effort involved in interactions with the models. 

• Code quality: whether the implementation solved the problem, respected the requirements, and left intact what should not have been altered. 

Every delivery underwent type checking, linting, and testing. I also analyzed code changes through a blind review, without knowing which approach had produced each result. I conducted this review because an implementation can pass all automated checks and still alter a behavior that should have remained unchanged. 

Why did the experiment have three versions? 

The first round had 36 runs.

In the second, I struggled to control the execution conditions and models, which is why I discarded the numbers. The problems helped me redesign the experiment.   

In the third, I expanded the set of approaches, locked the tool versions, automated steps, and better separated planning and implementation. There were 162 runs, with three repetitions for each combination of task and approach. With the repetitions, it was possible to see how much each approach varies from one run to another. 

What did we learn from the experiments? 

1. The level of planning must match the complexity of the demand 

For small and medium tasks, the baseline achieved competitive quality and consumed less time and fewer tokens. More steps and more documents did not bring a proportional gain for this type of demand. 

An isolated fix, with a clear scope, can be handled well with a short specification, provided there is a plan and validation. In enterprise projects with many business rules, integrations, and dependencies—where decisions need to be maintained over time—consistent documentation and clear acceptance criteria make a difference. 

2. Clarifying requirements can prevent misguided implementations 

In the notifications task, some approaches paused to ask for clarification. Superpowers asked questions in some of the runs. The baseline, in this scenario, proceeded without asking. 

When the request has gaps, the AI fills them in on its own. The code may work and still do something different from what the client expected, and in a real project this turns into rework. Deciding which events trigger an alert and who receives it was up to people, and that decision should have been made before the agent started. 

3. Code that passes tests still needs to be reviewed 

Some implementations passed all automated checks, but introduced unnecessary alterations or changes capable of affecting existing behaviors. That is why technical review remains necessary, especially in systems that are already in production. 

4. Consistency matters just as much as the best result 

The results varied between repetitions. Some approaches fluctuated significantly, while others were more constant. To choose a tool, you must consider how it behaves across runs, alongside quality, time, and resource consumption. 

Is there one SDD tool that is best for all projects? 

From this study, it is not possible to choose a single winner. The baseline stood out on small and medium tasks, but the experiment used a single application, a single stack, and tasks that fit within a work session. It did not measure the value of documentation in long-term maintenance or the behavior of the approaches in large projects, which is where detailed specifications help preserve context and the decisions made. 

The choice depends on the case. For a one-off fix, a lean workflow is usually sufficient. For a critical system, it pays to invest more in documentation, planning, and verification. 

How do these insights contribute to software engineering at Pitang? 

Much of what we develop involves complex business rules, integrations, and requirements that change from client to client. In these projects, delivery must match what was agreed upon, and SDD is how we organize this in our agentic pipeline. Studies like those from Pitang Labs help decide when to use more or less structure. 

Presenting the study at the Technical Break to over 80 colleagues was one of the parts I enjoyed most. People asked questions and brought experiences from other projects, and I left with insights I wouldn't have reached on my own. It is the kind of exchange I value most at Pitang. 

Developing with AI requires method and criteria to apply it 

In the tests, the volume of specification that worked varied with the task. For one-off fixes, a short plan was enough. In critical projects, which the study did not cover, traceability and clear acceptance criteria help avoid costly ambiguities. 

With or without AI, it is up to those who develop to understand the problem and verify that the solution does what was asked. At Pitang, we continue experimenting, sharing what we learn, and refining our way of developing software with AI.