Bug priority is a decision you defend, not a score

Scoring matrices explain how to prioritize bugs without anyone owning the order. A defensible order is one where each deferral names its cost and its acceptor.

Five glass weights balance on a rail while one orange weight hangs below it

The methods offered for how to prioritize bugs end in a score: severity times frequency, divided by effort. The score produces an order nobody has to own. When a customer asks why their bug is deferred, the defensible answer is a sentence naming the cost of waiting and the person who accepted it, and no matrix writes that sentence.

A matrix produces an order nobody owns

The appeal of a scoring matrix is that it looks like arithmetic. Severity on one axis, frequency or reach on the other, effort as a divisor, and the backlog sorts itself. What the arithmetic hides is that every input was a judgement: somebody decided this bug is a three and that one a four, and the multiplication laundered those judgements into a number that nobody can be asked to defend. “It scored a six” is no answer to a customer, an account manager or the engineer who has to say why the thing they hit last week is still open.

The order a lead can defend has a different shape. For every bug that does not fit the sprint, there is a sentence: what it costs to wait, who bears the cost, and who agreed to it. Written down, that sentence survives the customer call, the retrospective and the day the bug turns out to matter. A score survives none of them.

Severity is a fact; priority is a decision

The two are conflated by every matrix that multiplies them, and the confusion is why the order keeps being argued. The difference between severity and priority is the difference between a property and a choice: severity is read off the evidence, from a written scale of severity levels, and moves only if the evidence does; priority is when, and it moves with the sprint, the launch date, the contract renewal and the number of people who have hit the bug this week. A severe bug affecting one account with a workaround can sit below a cosmetic one on the pricing page the week before a launch, and that is a correct order rather than a contradiction.

Keeping the two as separate fields on the ticket is what makes the order arguable at all: the severity field is where the evidence goes and nobody argues it, and the priority field is where the decision goes and everybody may. A board with severity and priority as two fields on the ticket keeps the evidence and the decision from overwriting each other, which is exactly what a blended score does.

The questions that produce a defensible order

Sorting by the answers to a fixed set of questions is what the matrix was trying to approximate. Asked directly, each question yields a reason that can be quoted later.

The questionAn answer that moves the bug upWhere the answer comes from
Is there a workaround?No, and the task cannot be completed another wayThe report, confirmed by reproduction
Who hits it?Every user on a common path, or one account whose contract names itThe route and the browser in the reports, and the account list
Did it work last release?Yes: it is a regression, and something specific changedThe release history
Does it damage data, or only display it wrong?Data: a wrong stored value outlives the fixThe evidence, and a look at what was written
Is it blocking other work?Yes: a fix or a feature is waiting behind itThe engineers, in the room
Is there a date attached?A launch, a demo, a renewal, an auditThe calendar, which the matrix never sees

Six questions, each with a source, and the source is the point. “Who hits it” is answerable from reports that carry the route and the browser, and unanswerable from reports that say “the page is broken”; a lead prioritizing the second kind is guessing reach, which is the input the matrix was most confident about. The date question is the one no score contains, and it is the one that most often reorders a sprint on a Thursday.

Deferral is the decision; fixing is the consequence

Reframe the sprint from “which bugs do we fix” to “which bugs do we defer, and who accepts each deferral”. The set that fits is then the residue, and every bug outside it has a named reason and a named acceptor: the account manager who agreed their customer can live with the workaround until next sprint, the product owner who accepted that the pricing page rounds wrong for another week.

That is what a deferral log is: a short list, written in the room, that the lead reads aloud when asked to defend the order. It also changes the conversation with the people doing the asking: an account manager who accepted a deferral last Tuesday does not escalate it on Friday, because the cost was theirs to weigh and they weighed it.

A zero-bug policy is the same idea run as a standing rule. Under it, a bug is fixed in the current cycle, ahead of feature work, or closed with a written reason, and nothing sits in between. A team that cannot run it in full still gets its premise: a backlog of deferred bugs with no acceptor is a list of decisions nobody made, and each entry is a reporter who was told “soon” by nobody in particular.

A deferral expires, and the calendar does not warn you

A deferral is correct on the day it is made and decays from then on. The account that accepted the workaround grows, the launch the cosmetic bug was deferred past arrives, and the regression that affected nobody last week is on the common path after Monday’s release. None of that changes the score, and all of it changes the order.

So the deferral log gets re-read every sprint, by the lead, against the calendar, and that is time the lead spends rather than time a tool saves. Each deferral is checked against its acceptor: does the reason still hold, and does the person who accepted it still accept? A deferral whose acceptor has moved teams or whose reason has lapsed is a bug that is open with nobody’s consent. That check is the whole method: severity read from the evidence, priority answered from the six questions, every deferral carrying an acceptor, and a re-read each sprint before the acceptor finds out the hard way.

Common questions

How do you prioritize bugs when there are too many to fix?

Set severity from the evidence against a written scale, then decide priority against the sprint by asking whether a workaround exists, who hits it, whether it is a regression, whether it damages data, whether it blocks other work and whether a date is attached. Every bug that does not fit is deferred with a named reason and a named acceptor.

Why is 'who hits it' the hardest question to answer?

Because it is only answerable from reports that say which page the bug happened on and in which browser, and most reports say the page is broken. A lead ranking those reports is guessing reach, which is the input a matrix presents as its most reliable. Reports that carry the page path and the browser answer it directly, and the account list supplies who the users on that path are.

Should you use a scoring matrix to prioritize bugs?

A matrix approximates the right questions but hides the judgement behind each input, so the resulting order cannot be defended to a customer or an account manager. Asking the questions directly, and writing the answer on each bug, gives an order with reasons that can be quoted later.

How do you defend a decision not to fix a bug this sprint?

With a deferral written at the time: what it costs to wait, who bears that cost, and who accepted it. An account manager or product owner who agreed to the deferral is the person the defence is addressed to, and a deferral they weighed themselves comes back as a question about whether the reason still holds rather than as an escalation.

How often should bug priorities be reviewed?

Every sprint, by the lead, against the calendar. A deferral is correct on the day it is made and decays as accounts grow, launches arrive and releases move a bug onto a common path. Each deferral is checked to see whether its reason still holds and whether its acceptor still accepts.

What should a deferral log record for each bug?

Three things, written in the room when the deferral is made: what it costs to wait, in the customer's terms rather than the engineer's; who bears that cost; and who accepted it, by name. The log is read aloud when the order is challenged and re-read each sprint against the calendar, so an entry whose acceptor has moved on or whose reason has lapsed is a bug open with nobody's consent.

The Notarus team

The people who build Notarus. We write up the decisions behind the product when the reasoning is worth more than the changelog line.

Keep reading

Reports with the evidence attached.

Start free, one script tag.

<script src="…/widget.js" data-key="pb_pub_…">