Test suite

In software testing, a test suite is a named collection of test cases that are run together because they share a purpose: the regression suite, the smoke suite, the checkout suite. The suite is the unit that gets scheduled, reported on and marked passed or failed as a whole, and its membership is a decision about what one run is meant to prove.

The longer version

A suite exists because tests are rarely run one at a time. Grouping them gives a run a name, an owner and a verdict: the smoke suite passed, the regression suite has two failures. Membership follows purpose. A smoke suite holds the handful of cases that prove the build is testable at all; a regression suite holds every case that guards a previously fixed defect; a feature suite holds everything that exercises checkout. The same case can belong to several suites, and in most tools a suite is a tag or a folder rather than a copy.

The word is also used commercially for a bundle of products, and search results for it mix those in; the suite meant here is the one testers run. Inside testing, the neighbour it is confused with is the test plan. A plan says what will be tested, by whom, in which environment and against which criteria. A suite is the set of cases that carry out one part of that plan. A plan references several suites, and a suite can serve several plans.

What breaks a suite is accretion. Cases are added after every incident and never removed, until the regression suite takes an afternoon and nobody runs it before a release. Flaky cases, the ones that fail without the code changing, cost the suite its readers: after a few false alarms the team stops reading the results, and a genuine failure is dismissed with the noise. A suite has to be pruned and its flaky members quarantined for a pass to mean anything.

In practice a team keeps a small fast suite that runs on every change and a large slow one that runs before a release or overnight, and moves cases between them deliberately. The suite that gates a release is named in the test plan's exit criteria, so that 'the regression suite passed' is a checkable statement rather than a feeling.

Common questions

What is the difference between a test suite and a test case?

A test case is one check with a precondition, steps and an expected result. A suite is a named group of cases run together for one purpose, such as regression or smoke testing. The case is what passes or fails; the suite is what gets scheduled, reported on and quoted in the exit criteria.

What is the difference between a test suite and a test plan?

A test plan is a document: what will be tested, by whom, where, and against which entry and exit criteria. A test suite is a set of cases that carry out part of that plan. A plan names the suites it depends on; a suite does not contain a plan and does not know about scope or schedule.

How big should a test suite be?

Small enough that the team runs it when it is supposed to. A suite that gates every commit has to finish in minutes, so it holds the few cases that catch the most damage. A suite that runs before a release can be larger and slower. Size follows when the suite runs, not how many cases exist.

How are test suites organised?

By purpose first and by feature second. A suite is named for what one run of it is meant to prove: the smoke suite proves the build is testable, the regression suite proves fixed defects stayed fixed, the checkout suite proves one area of the product. In most tools a suite is a tag or a folder, so one case belongs to several suites without being copied.

How often should a test suite run?

As often as the change it guards happens. A small fast suite runs on every commit, because that is when a regression is cheapest to find. A large slow suite runs before a release or overnight, when its duration costs nobody a working afternoon. A suite that runs less often than the code changes is finding defects after other work has been built on them.

Bug reports that arrive ready to triage.

Collect replay, console, network and browser context with the Free plan.

<script src="…/widget.js" data-key="pb_pub_…">