A bug count that climbs after every release is measuring the filing rate before it measures the product, and the next quarter’s budget is about to be spent on a guess. More automated tests, stricter review and a QA hire each prevent one origin of defect. Until last quarter’s escapes are tagged with their origin, nobody knows which one.
A climbing count is a filing rate first
Several things push a bug count up without the code getting worse. More users reach more corners of the product, so defects that were always there get found. A new reporting channel, a widget on the page or a portal for clients, makes filing easy enough that people file what they used to tolerate. The same defect arrives under three titles from three reporters because none of the reports carried enough to match them. And a backlog that is never closed makes the count cumulative, so it climbs by construction.
The first job is to separate three numbers that the tracker shows as one. Open tickets is a backlog measure. Tickets filed is a channel measure. Unique defects that reached production per release, each an escaped defect, is the only one that says something about the release. A count of tickets that rises after a feedback widget goes live measures the widget; a count of unique escapes per release that rises is the product getting worse, and only the second one deserves a quarter’s budget.
Every prevention spend fixes one origin
The candidate spends on an engineering manager’s list are each a fix for one kind of defect, and a defect has an origin whether or not anybody recorded it.
| Origin | What the ticket looks like | What prevents it |
|---|---|---|
| Requirement misread | Works as built, wrong as intended; the product owner says that is not what was meant | Acceptance criteria written as examples before coding, and a second reading by somebody who did not build it |
| Logic error in new code | Reproduces on a clean path through the new feature | Unit tests on the new code, and review |
| Regression | Used to work, broke in a change nobody thought was related | A regression pass chosen by risk, and characterization tests on the code being changed |
| Integration | Works in isolation, fails at the boundary with another service or a third party | Contract tests on the boundary |
| Environment or compatibility | Works in one browser, device or configuration and not another | A compatibility matrix named in the definition of done |
| Data or state | Works with test data, fails on the shape production holds | Test data that mirrors production shapes, and boundary values in the cases |
Read the manager’s list against the right-hand column. Test automation, in the form of a regression suite, funds the regression row. Stricter review funds the logic row. A QA hire funds the requirement row and the exploratory pass, and reaches the environment row only if the compatibility matrix is written down for them. None of the three touches the integration row, and a quarter spent on the regression row while the escapes are coming from a third-party boundary leaves the count exactly where it was, with a larger test suite to maintain.
Classify last quarter’s escapes before spending
The method is a tag and a table. Take every defect that reached production over the last few releases. For each, a person who knows the product records two things: the origin row it belongs to, and the check that should have caught it. That second tag is what turns a list of bugs into a list of gaps, because a defect that should have been caught by review and was not is a fact about review, and a defect no existing check would ever have caught is a fact about the checks.
Then read the distribution. The largest bucket is where the quarter goes. The number to watch as the spend lands is defect removal efficiency, the share of a release’s defects caught before it shipped, because it rises when more of a release’s defects are caught before shipping, and a channel that makes production reports easier to file pushes it down rather than up, so the rise that started the panic cannot flatter it.
The tagging is triage work, done at the moment a ticket is read, and it is the reason to insist that triage happens at all. Retrofitting it onto a year of closed tickets produces guesses dressed as data, since the person doing it no longer remembers the defect and the evidence that would have told them is gone.
Two origins need tests most teams have not written
Write more tests is the advice everyone gives, and it fails on two of the rows because the tests a team already knows how to write do not reach them.
The regression row is where the code nobody understands and everybody is afraid to touch shows up. Unit tests on new code do nothing for it. What does is a characterization test: a test written against the code as it is, recording what it does today, correct or not, so that a change shows what it altered. Nothing about the test asserts the behaviour is right. It asserts the behaviour is the same, which is the only question a regression is about.
The integration row is about boundaries, and the tests a team already has reach neither side of one. A unit test mocks the other side and proves nothing about it; an end-to-end test against a live copy of the other side is slow and fails for reasons that have nothing to do with the change. A contract test sits between them: each side tests against the agreed shape of the exchange, so a change on either side that breaks the agreement fails a test on the side that broke it. With a third party that will not run its half, the provider side of the check is a scheduled verification of the recorded contract against the live service, which catches the break after it happens rather than before. When the classification puts integration at the top, this is the spend.
The unknown bucket decides the first spend
Classification has a precondition, and it is the limit the whole plan rests on. A ticket can only be tagged with an origin if it carries enough to tell one origin from another. A console exception points at logic. A failed request with its status points at an integration. A report that reproduces on one browser and no other points at compatibility. A ticket that says the page broke points at nothing, and it goes into a bucket labelled unknown.
When unknown is the largest bucket, the first spend goes to how bugs are reported rather than to any prevention. A report filed through evidence capture carries the console log, the network requests with their statuses, and the browser, device and page path of the session that produced it, which is the same set of facts the origin table sorts on, so a triager can assign the row in the time it takes to read the ticket. Until the reports carry that, the origin table has a hole in it, and a hole in the table is a guess in the budget.
The tag cannot be back-filled for tickets whose evidence has gone, so the first quarter of classification measures only the tickets filed after tagging began, and the spending decision it supports is a quarter out. The distribution also moves once the largest bucket is addressed, so each decision is good for one quarter, after which the table is read again and the answer may be different.
