Regression testing exists to catch the side effects of a change before users do. It runs after every fix, feature, refactor, dependency upgrade or configuration change, and the change decides how much runs. A one-line edit to a shared helper can justify the whole suite, while a copy change on one page may need one visual check.
What does regression testing cover, and what does it leave out?
The ISTQB glossary files regression testing under change-related testing and points it at “unchanged areas of the software”, and that phrase carries the whole scope. Regression testing covers what the change was not meant to alter: the checkout that shares a date helper with the invoice screen somebody edited, the layout a stylesheet tweak shifted on a page nobody opened. It leaves out two neighbours that also run after a change. The new or changed behaviour belongs to progression testing, checked against the new requirement, because the previous build is no reference for behaviour that was meant to change. The fixed defect belongs to confirmation testing, which ISTQB also names re-testing and which re-runs one ticket’s steps (regression vs retesting sets the two side by side). What regression testing hunts is the regression bug: behaviour that worked in an earlier build and stopped. Because the trigger is a change and not a calendar, every regression run starts from two questions, what changed and how far its effects can reach.
What are the types of regression testing?
Published lists of regression testing types disagree on both the names and the count, and no standard fixes either. The disagreement goes away once each type is read as an answer to those two questions: what triggered the run, and how much of the suite it re-runs. Sorted that way, five types cover the ground.
| Type | What triggers it | What it re-runs | Glossary term |
|---|---|---|---|
| Retest-all | A change with wide or unknown reach: a framework upgrade, a database migration, a release candidate | The whole regression suite | Test suite |
| Selective | A change with a known footprint, such as a commit touching a handful of files | The tests a code-to-test map links to the changed files | Test impact analysis |
| Prioritised | Any change where the time to the first failure matters, typically each pull request | The whole suite or the selected part, ordered so likely failures run first | Test case prioritization |
| Progressive | A changed requirement, so some existing tests now expect the wrong result | The existing tests, with the ones the change invalidated updated first; tests for brand-new behaviour stay outside the run | Progression testing |
| Visual | A change to styles, fonts, components or anything else that can move pixels | Screenshots of the captured pages, compared with an approved baseline | Visual regression testing |
Progressive regression is a trade convention, not an ISTQB term, and its edge is the one drawn above: updating an existing test keeps it guarding the area around the change, while a test written for the new behaviour itself is progression testing for that change, and counts as regression only once it has merged and a later change re-runs it. In practice, most of the other names on those lists map onto these five as well. Complete regression is retest-all. Partial and regional regression are selective runs whose scope somebody chose by hand rather than from a map. Unit regression is a selective run held at the unit level. Corrective regression re-runs the existing tests unchanged because the specification did not move, which is the ordinary case and the opposite of progressive.
The choice is made per change, never once per project. A pull request touching a pricing function gets a selective, prioritised run. The release candidate built from that sprint gets retest-all. A design-system update adds a visual run on top. A team that picks a single type for everything ends up with a suite that is either too slow to wait for or too narrow to trust.
How is regression testing different from smoke, sanity, retesting and functional testing?
Most of these comparisons mix three different axes. In ISTQB’s vocabulary, unit testing (which it calls component testing) and integration testing are test levels. Functional testing and change-related testing, the family regression and confirmation testing belong to, are both types of testing, filed by different criteria: functional by what is checked, change-related by what set the run off. A regression suite therefore contains functional tests at the unit and integration levels. Asking whether to do functional or regression testing is like asking whether to test the checkout or to test it after Tuesday’s deploy.
| Compared with | What it is | The question it answers | Where it meets regression testing |
|---|---|---|---|
| Smoke testing | A quick pass over a build’s most important paths | Is this build stable enough to test at all? | Runs first; a failed smoke test means the regression run never starts |
| Sanity testing | A narrow check of one area after a change to it | Did this change do what it claimed? | Sanity looks at the changed area, regression at everything around it |
| Retesting (regression vs retesting) | Re-running one defect’s steps after its fix | Is this bug gone? | A fix gets both: retesting for the ticket, regression for its side effects |
| Functional testing | A test type: does the system satisfy its functional requirements? | Does the feature do what the specification says? | Most regression tests are functional tests, re-run after a change |
| Unit testing | A test level: one component in isolation | Does this function or class behave? | The cheapest regression layer, where selective and prioritised runs work best |
| Integration testing (system integration testing) | A test level: the interactions between components or systems | Do the parts work together? | An integration suite re-run after a change is regression testing at that level |
The comparison people search for most, functional vs regression testing, has the shortest answer: a functional test is named for what it checks, a regression run for why it runs. A test written for a new shipping-fee threshold is progression work while that threshold is being built; once it has merged, the first unrelated change that runs it again makes it a regression test, and it is still a functional one. Non-functional checks such as performance and accessibility become regression tests the same way, when they are re-run after a change to confirm nothing degraded.
How do you build and keep a regression suite?
A regression suite grows from three sources. Every fixed bug gains a test that reproduces it, so the same defect cannot return quietly. Every progression test stays in the suite after it merges and starts guarding regression ground the first time a later change re-runs it. And any exploratory finding worth checking again gets written down as a case. Growth is the easy part. Keeping the suite small enough to run and sharp enough to catch something is where the design techniques earn their place.
| Suite decision | Technique | How it goes wrong |
|---|---|---|
| Which inputs get a test | Equivalence partitioning: one representative per class of input the system treats alike | Classes drawn from the code instead of the requirement miss the case the code forgot |
| Which configurations get a test | Pairwise testing: every pair of parameter values together at least once | A defect that needs three settings at once slips through |
| How one script covers many cases | Data-driven testing and keyword-driven testing | A data table nobody prunes fills with rows that test the same class twice |
| What the suite never executes | Statement coverage, a white-box measure | A line that runs with no assertion on it reads as tested |
| What runs on this change | Test impact analysis | A stale map selects nothing and the build passes |
| What runs first | Test case prioritization | A flaky test fails at random, scores as valuable and runs ahead of the tests that matter |
The last two rows are the daily mechanics. Selection gates the pull request with the tests a change can reach, prioritization puts the likeliest failures at the front, and the full suite still runs on a schedule and before release, because a selector that misses a dependency fails silently. Upstream of all of it, static testing catches part of the damage in review and analysis before any test runs.
The failure a suite cannot detect on its own is wear. The pesticide paradox is the principle that a set of tests repeated unchanged stops finding new defects, and a regression suite is repeated unchanged by design. The counters are deliberate: revise the cases each release, add a test for every escaped defect, and schedule exploratory testing in the areas the scripts touch lightly.
Part of every suite stays manual, and that part has its own problems. Manual regression testing is a filing problem covers where a manual pass loses its hours. Regression testing in agile needs a column covers why the pass lands on the last afternoon of a sprint. And a release that cannot trust its suite tends to stop change with a code freeze instead of checking it.
Which tools does regression testing need, by job?
Regression tooling is easier to choose by the job each tool does than by category labels, because each job fails differently and no single tool covers all four.
| Job | What the tool does | What it cannot tell you |
|---|---|---|
| Test runner | Executes unit, integration and end-to-end tests on each change, reports pass or fail, keeps artifacts from failures | Anything about a page, state or path nobody wrote a test for |
| Visual diff | Renders the captured pages, compares them with an approved baseline, flags changed regions for a reviewer | Whether the behaviour behind a page works, or how states and devices outside the captured set look |
| Test management | Holds the cases, which of them ran against which build, and the results of the manual pass | Why a failure happened; it records outcomes, not the state of the page |
| Evidence and reporting | Turns a regression a person found into a ticket carrying what was on the screen and in the browser | Whether the rest of the suite passes; it runs nothing |
Notarus belongs only in the last row: it is not a test runner and not a visual-diff tool, it executes no tests, and it compares no screenshot with a baseline. When somebody finds a regression on a live page, the report reaches the team with the screenshot they annotated, the session replay when one is available, the console errors and network calls from before the report, and the page URL, viewport and browser (evidence capture lists exactly what is kept).
The limit to plan around is that evidence is best effort: the project’s capture settings, the reporter’s browser and the widget’s performance budgets each decide part of what arrives, and a report can still be sent with none of it. That makes the evidence row a companion to a runner, never a replacement for one. A regression the suite catches already arrives with the runner’s own artifacts; the evidence row exists for the regressions a person catches first.
Related terms and guides
- Visual regression testing: screenshot comparison against an approved baseline
- Test impact analysis: choosing the tests a change can affect
- Test case prioritization: ordering a suite so likely failures run first
- Progression testing: testing the new and changed behaviour
- Pesticide paradox: why an unchanged suite stops finding defects
- Static testing: finding defects without running the code
- White-box testing: designing tests from the code’s structure
- Equivalence partitioning: one test per class of input
- Pairwise testing: covering every pair of parameter values
- Statement coverage: the share of statements a suite executes
- Keyword-driven testing: tests written as readable action keywords
- Data-driven testing: one script run over a table of cases
- Manual regression testing is a filing problem: where a manual pass loses its time
- Regression testing in agile needs a column: giving the pass a place on the board
