Canary testing
Canary testing is releasing a change to a small share of real production traffic first, and comparing how that group fares against everyone still on the current version before widening. If errors or latency rise, the change is pulled back with few people affected. If the numbers hold, the share grows.
The longer version
A router or a feature flag sends a small slice of requests, perhaps one in a hundred or a set of internal users, to the new version while the rest stay on the old one. The team watches a short list of signals for both groups: error rates, response times, failed payments, and for a web front end the errors and load timings reported by real user monitoring. The comparison is the point. A graph that looks acceptable on its own can still be worse than the control group beside it.
The slice has to be large enough to show a difference and varied enough to include the browsers and accounts that tend to break, so a canary made only of staff on fast laptops proves little. The pass criteria and the rollback trigger are agreed before the release, not argued over while a graph is climbing. A short watch period also misses faults that appear only at month end or for a rare account type, so its length should match the risk. And because a canary exposes real users to the change, it is a poor fit for one that cannot be undone for the people who got it, such as a data migration that rewrites records. A change like that is better rehearsed on a copy of production data, or in a dark launch that writes somewhere harmless.
Canary testing sits at the front of a phased rollout, the schedule that follows a good result, and it works with a feature flag, which can switch the new path off without a deploy. A router can do the same job at the infrastructure level by weighting traffic between two versions of a service.
Common questions
What is canary testing?
Canary testing is releasing a change to a small portion of live traffic and comparing it with the current version before rolling it out wider. The small group acts as an early warning, so a defect that only shows under real use reaches a few users instead of everyone.
Is a phased rollout the same as canary testing?
A phased rollout is the whole schedule of stages by which a release reaches all users, and canary testing is its first small step, with an explicit comparison against the current version. A rollout usually begins with a canary stage and continues to widen while the signals stay healthy.
How big should a canary group be?
A canary group should be big enough for a difference in error rate or latency to show up against normal variation, and small enough that a failure is tolerable. A site with heavy traffic can learn from a very small share, while a low-traffic product may need a longer watch period or a hand-picked group of accounts.
What metrics should you watch during a canary release?
Watch error rate, latency, saturation of the service and the business actions the change touches, such as sign-ups or completed payments, for the canary and the control group side by side. For front-end changes, errors and load timings from real user monitoring show what the browser experienced.
Where does the name canary testing come from?
The name comes from the practice of taking canaries into coal mines, where the birds reacted to dangerous gas before miners did. In software the small group of users on the new version plays that role, showing trouble before it reaches the rest of the user base.
Bug reports that arrive ready to triage.
Collect replay, console, network and browser context with the Free plan.