A QA lead handed a colleague’s test cases to review before a release reads them top to bottom and corrects the steps. That review finds typos. The finding worth having is the case that is missing, and nothing in the suite points at it; only the requirement does, which is why the review starts there.
The missing case is found from the requirement
A suite is a list of what somebody thought of. Reading it tells the reviewer what is covered and nothing about what is not, because an absent case leaves no mark in the document. Reviewers who read the cases first end up reviewing the writing: the steps get tightened, the titles made consistent, and the suite comes back looking better with the same holes in it.
The review that finds the hole runs the other way round. The reviewer reads the requirement, or the story and its acceptance criteria, and writes their own list of conditions before opening the suite: the limits the requirement states, the combinations of conditions that change the outcome, the states in which the same action means something different, and what must not happen. Then the suite is diffed against that list. A condition on the reviewer’s list with no case against it is the finding, and it is the one finding an author cannot make about their own work, because the author’s list is the suite.
That order has a cost. The reviewer has to know the requirement well enough to enumerate its conditions, which takes longer than reading twenty cases, and it is why a review by somebody who does not know the feature is a proofread with a different name.
What a review catches that a run never will
A run answers one question: does the build do what the case predicts. It says nothing about whether the prediction was worth making. The defects that live inside the case itself pass every run, which is how they survive until a release goes out with the case green.
| Defect in the case | What a run shows | What the reviewer asks |
|---|---|---|
| A case that cannot fail | Green, every release | Would this fail against the build from before the feature existed? |
| An expected result copied from the build | Green | Where in the requirement does this outcome come from? |
| Two conditions in one case | Green, or one failure nobody can attribute | Which condition does the verdict belong to? |
| Test data that lives in the author’s head | Green for the author, failed for everyone else | Could I run this with the account and data named here? |
| A precondition written as a step | Green, slowly | Does the run start where the case says it starts? |
| A negative condition with no case | Green | What does the requirement say must not happen? |
The first row is the one to check first, and it has a mechanical test: a case that would also pass against the build from before the feature is describing the build rather than the requirement. The second row is the same defect caught earlier. An expected result the author wrote by looking at the screen is the build acting as its own oracle, and it can be spotted because the requirement contains that sentence nowhere.
The fourth row is the one most reviews skip, because the reviewer knows the same accounts the author does. The test is whether the test data the case depends on is named in the case or in a fixture the case points at. A case that says “a valid code” was written for one afternoon.
A review has three verdicts, and one of them is removal
Most reviews only add. A reviewer who returns a suite with more cases in it than they received looks diligent, and a suite that only ever grows is how a test suite comes to spend its run time checking the same login form a dozen ways. The verdicts a review can give on a case are accept, rewrite and remove, and the third needs a reviewer because deleting a case is a verdict authors do not pass on their own work. Add, the fourth row below, is a verdict on the suite rather than on a case.
| Verdict | When it applies | What goes back to the author |
|---|---|---|
| Accept | The expected result traces to the requirement and a stranger could run it | Nothing; the case is recorded as reviewed against that version of the requirement |
| Rewrite | The condition is right and the case does not reach it | The condition, and which defect in the table above the case has |
| Remove | The case duplicates another, cannot fail, or predicts a requirement that no longer exists | The reason, because a case whose removal is unexplained gets rewritten from memory next quarter |
| Add | The reviewer’s condition list has an entry with no case | The condition only; the author writes the case, since the author maintains it |
The last row carries a rule about who writes what: a reviewer who writes the missing case has taken over authorship of the suite, and the next review of it is a review of their own work.
Read the last release’s failed runs before the cases
The cheapest evidence a reviewer has is what the suite produced last time. Every case that failed in the previous release and was filed left a ticket, and the ticket’s fate is a verdict on the case. A failure that was fixed was a case doing its job. A failure closed as irreproducible is either a defect that got away or a case with a weak expected result or an unreachable precondition, and a reviewer can usually tell which by reading what the ticket carried. Our QA checklist asks that a regression run’s failures be triaged as real defects or as test defects, and the test defects from the last round are the list a review opens with.
That reading is only possible when the failed run was filed with what the run saw. A ticket that gives the case number and nothing else cannot tell the reviewer whether the runner hit a real defect on an unstated browser or ran the case with the wrong data. A failed run filed through evidence capture on the page under test keeps what that run saw, the browser it ran on, the path it was on and the console and requests at the moment of failure, so a reviewer a month later can tell an unstated browser from wrong data without re-running the case.
A case whose failures keep closing as irreproducible has a pattern the reviewer should name: the same missing precondition, or the same data nobody else holds, or an outcome two runners read differently. Those cases go to the rewrite column first.
The review is an exit criterion, not a release step
A review scheduled the week before release finds the missing case after the time to write and run it has gone. What the reviewer then holds is a decision about risk, ship without the case or slip, that the author cannot make and the reviewer was not asked to make.
The way out is timing rather than speed. The review belongs at the point where the requirement is settled and the cases are written, as the exit criterion of test design and the entry criterion of the run, so that a missing case is a task on the board rather than a release decision. A team that reviews only before release will keep finding its gaps at the moment they cost the most.
The condition list from the first section has to outlive the review. A review leaves no artefact unless the reviewer writes one, and a review made against a requirement that has since changed is indistinguishable from no review at all. Recording which version of the requirement the suite was reviewed against is the only thing that lets the next reviewer diff instead of starting again, and it is the reviewer’s to record, because nobody else was in the room.
