A dashboard is being asked for because somebody wants a number that proves testing is working, and most testing metrics cannot supply it. Test cases executed, pass rate and coverage count what the testers did, and activity rises whether or not the release holds. The number that answers the question is measured in production, after the release.
A pass rate measures the test suite, not the product
Open a test execution report and the top line is a count of cases run and a count that passed. Read as a claim about the product, it says something narrow: the product agreed with the cases somebody wrote. A suite with no case that reaches the checkout reports a perfect pass rate on the day the checkout breaks. Coverage has the same shape one level down. It counts the statements or branches the tests reached, and a test that reaches a line and asserts nothing counts the same as one that would have caught the defect.
None of that makes the counts useless. They are useful to the people running the suite, as a record of what was run against which build. They fail as proof because they can be moved without moving the product, and a number on a sponsor’s dashboard gets moved.
| Metric | What it counts | How it rises with the product unchanged |
|---|---|---|
| Test cases executed | Cases marked run, by a person or a runner | Split each large case into several small ones |
| Pass rate | Cases whose assertion held | Quarantine the flaky ones, delete the ones that keep failing |
| Code coverage | Statements or branches a test reached | Add tests that reach lines and assert nothing |
| Defects found per sprint | Tickets filed | File cosmetic findings, file the same defect from two testers |
| Defects closed per sprint | Tickets moved to Done | Close as cannot reproduce |
Nobody on the team has to be dishonest for the right-hand column to happen. Each row is a reasonable local decision made under a target, and the sum of them is a dashboard that improves every quarter while the release does not.
The sponsor’s number is measured in production
The question behind the request is whether testing caught what mattered, and the only place that gets answered is after the release, by the users. An escaped defect is any defect that reached production without being caught, and the count of them per release is the closest thing to a verdict on testing that exists. Defect removal efficiency turns the same count into a ratio: defects caught before release, over all the defects the release turned out to contain.
Both have the property the activity counts lack. A tester cannot move them, because a tester does not control what happens in production. A team can only move them by catching more before release, which is the thing the sponsor wanted a number for.
They have a cost, and it is the reason dashboards drift back to pass rates. The count for a release is not complete on release day. Users reach the corners of a release over weeks, and a defect they find late still belongs to the release that shipped it. The verdict on this quarter’s testing is legible next quarter.
What can be shown this week without lying
Between the verdict and the activity counts sits a third kind of number: measurements of the conditions under which escapes go down. They do not prove testing worked. They show whether the work that makes it work is happening, and they are current.
The age of an unread finding is the first. A defect found in test and read after the release has escaped as surely as one nobody found, and the time between a report being filed and somebody in triage deciding what it is comes straight off the tracker’s timestamps. The share of findings closed as irreproducible is the second, and it measures the evidence rather than the testers: a finding whose console errors and failed requests are on the ticket gets reproduced or refuted, and one written from memory gets closed. Both numbers need every finding on one board, with its arrival time and its closure reason on it, which is what a board fed by reports filed from the page under test gives the person doing triage.
The third is the share of the risk list that had a pass run against it. A risk-based plan names the areas whose failure costs most, and a sprint under pressure skips some of them. A number that says which were skipped is worth more to a sponsor than any count of cases run, because it names the exposure rather than the effort.
How to lay the dashboard out
The layout carries the argument. The verdict sits on top, one release behind. The conditions sit under it, current. The activity counts go at the bottom or into the test report, where they were always meant to live.
| Panel | Number | Where it comes from | What it is |
|---|---|---|---|
| Top | Escaped defects per release, by severity | Production tickets tagged with the release that introduced them | The verdict, one release late |
| Top | Defect removal efficiency | Defects caught before release over all defects the release contained | The verdict as a ratio |
| Middle | Findings older than the sprint, still untriaged | Ticket timestamps | A condition, current |
| Middle | Share of findings closed as irreproducible | The reason written on each closed ticket | A condition, current |
| Middle | Risk items with no pass executed | The risk list against the test report | A condition, current |
| Bottom | Cases executed, pass rate | The runner and the test report | A record, never a target |
The last row belongs in the test report that closes a cycle, as a statement of what was run against which build. As a record it is fine and nobody should remove it. Promoted to a target it becomes the first table in this post.
The top panel is tagged by hand at triage
The top panel has a manual step in it. An escaped defect is only countable if somebody at triage tags each production bug with the release that introduced it and the check that should have caught it. That is a judgement about the product, made by a person who knows it, and it cannot be delegated to a rule in the tracker or to the reporter. If nobody makes it, the top panel stays empty and the dashboard falls back to the activity counts by default, which is how most of them came to show a pass rate.
So the answer to the request is a schedule rather than a number. The conditions can be shown this week. The verdict on this quarter’s testing can be shown next quarter, and only if triage does its half every day between now and then.
