A code freeze is a regression test you skipped

A code freeze swaps regression evidence for calendar days. What a freeze still buys, which risks a trusted suite covers instead, and how the freeze shrinks.

Four upright glass slabs, three frosted and one clear, stand beside three stacked lens discs with an orange top disc

In software, a code freeze is the stretch before a release when merges stop, so the build that ships is the build that was checked. A freeze pays for that safety in calendar days, standing in for proof that a late change broke nothing. Where a regression suite is trusted to supply the proof, the freeze shrinks.

A freeze trades calendar days for regression evidence

The phrase also turns up in construction, where the code in question is a building code. In software releases the practice comes in three strengths. A feature freeze stops new features from entering a release and lets fixes keep flowing. A code freeze, sometimes called a hard freeze, stops nearly everything and admits only fixes someone approves one at a time. A change freeze reaches past the repository to deployments, infrastructure and configuration. Release management usually schedules the freeze next to the test window, the acceptance window and the deployment slot.

All three exist to stop the target moving. A release team needs to know whether a late change reached areas nobody meant it to touch, which is the question regression testing answers. A regression run answers it by checking. A freeze answers it by forbidding the change, so the last check stays valid while nothing moves.

So the freeze is a regression test paid for in days instead of in runs. Regression testing in software testing sorts regression work by its trigger, a change, and by how much of the suite each change re-runs. A freeze removes the trigger instead of running anything. As protection against regressions, that is reasonable only when the team does not trust what a run would tell it.

Some freezes buy what no test suite can

Several things a freeze buys are sound, and evidence about the code supplies none of them.

The clearest is peak trading. An online shop heading into Black Friday faces a risk unrelated to regressions: any deploy can go wrong for operational reasons, such as a migration that locks a busy table, a cold cache or a configuration typo. A green suite says nothing about any of them, and an hour of degraded checkout costs the most in the busiest hours of the year, so fewer deploys means fewer chances to fail at the most expensive moment. Holidays raise a related problem from the other side. The engineers who would read a failure and roll it back are away, and a rollback plan still needs somebody to trigger it, so matching the rate of change to the people available is a staffing decision.

Some audits, certifications and financial closes add a requirement of their own. Where the reviewer asks for a stable system across the review period, or for every change inside it to be approved one by one, a freeze is the simplest way to comply, and evidence that a change was harmless answers a different question. Others accept approved, tracked changes, and there the freeze is a convenience rather than an obligation.

Mobile releases carry a reason of their own as well. A build that has passed store review cannot be recalled from the devices that already installed it, and a corrected build waits on another review before it reaches anyone. Freezing the release branch for that build mostly pays for the review lead time.

None of these is a gap in regression evidence; each concerns the cost of any change at a particular moment, or the people and paperwork around it. Those freezes keep their full length. The one this post argues with is the routine freeze: the few days held before every ordinary release, every sprint, to let things settle.

Every routine freeze risk has an evidence counterpart

Taken one at a time, most risks a routine freeze is said to cover have a counterpart that produces evidence instead of waiting for it.

Risk the freeze coversHow the freeze covers itEvidence that covers it without stopping merges
A late merge breaks an area nobody re-checkedNo late mergesTest impact analysis picks the tests each pull request can reach; the full suite runs on the release candidate
The shipped build is not the tested buildNothing changes after the last testBuild once, promote that artifact, record the test results against it
A regression in an area the suite has no test forDays of people using staging and hoping to noticeAn exploratory pass scoped to what changed, and a new test for each bug it finds
A regression that reaches production anywayFewer changes, so fewer chancesA feature flag to switch the change off, and a rollback with a named trigger
Operational risk of any deploy at peak tradingNo deploysNone at the code level; the freeze fits
Nobody available to respondNo changes while people are awayNone; a staffing question

The last two rows belong to the previous section, and the freeze keeps them. In the first four the freeze stands in for something a pipeline could produce on every change, which is the premise continuous delivery is built on. Each counterpart costs effort to set up. None of them requires holding merges back.

The third row is the weak spot in this argument. A suite checks only what somebody wrote a test for, and people actively using a frozen release do find regressions the suite missed. What fails there is the method: a pass nobody scoped or recorded, ended by the calendar instead of by coverage of the risky areas.

A quiet frozen branch proves less than it seems to

A freeze detects nothing by itself. A regression merged the day before the freeze is in the candidate on the first frozen day and still there on the last, unless somebody went down the path it breaks. A quiet week means nobody reported anything, which means “nothing broke” only for the paths people happened to take. Release readiness asks for evidence, and a calendar page turning is not any.

A freeze can be spent in one of two ways. Spent testing, scoped by what changed in the release and recorded as it goes, it is a manual regression window and should be planned as one. Spent waiting, it is a regression test skipped and paid for anyway. A freeze spent testing also inherits the costs manual regression testing is a filing problem describes, where the hours go into reporting each finding more than into finding it.

The system also keeps moving while the code stands still: browsers update themselves, third-party scripts on the page change on their vendor’s schedule, and production data grows into shapes the test data never had. A frozen repository is a smaller target than a frozen system.

Where the freeze stops merges to the main branch as well as the release branch, development carries on behind it in feature branches. They pile up and merge together in the first hours after it lifts, which makes the thaw the largest batch of changes in the whole cycle. When something breaks then, several changes are suspects at once, the exact situation that running regression checks per change was designed to avoid.

The freeze shrinks as fast as the suite earns trust

Treat each freeze as an experiment that produces data. Every defect found during it gets recorded, and for each one the team asks whether a test should have caught it earlier, and at which level. The answer becomes a test, written at the lowest level that reproduces the failure. Bugs that slip past the freeze as well and surface in production are escaped defects, and they get the same treatment, so the next one like either should meet a run instead of a calendar.

Over a few releases the question changes from “how long is the freeze” to “what did the freeze catch that the pipeline did not”. When the answer stays empty release after release, the routine freeze shortens, and the go/no-go decision rests on the suite’s results more and on elapsed time less. Peak-season and holiday freezes stay at their full length throughout, as do audit freezes wherever the reviewer requires one, because no amount of suite trust changes what they are for.

What a freeze finds, it mostly finds through people clicking through the release candidate, and their notes are the data the next freeze decision reads. “Checkout looked wrong on staging yesterday” supports no decision at all. With the Notarus widget on the staging site, the person testing circles the problem on the page in front of them, and the console errors, network calls, page path, browser, device and viewport go onto the ticket without anyone remembering to add them (evidence capture lists the closed set it keeps). Two limits matter for a freeze review held a sprint later: some reports arrive with gaps, since what gets recorded depends on how capture is configured and on the tester’s browser, and captured logs are kept for thirty days while the ticket itself stays. A review that reads the findings before the logs expire can tell which of them a test would have caught, and that count is what the freeze length should follow.

What survives of the routine freeze after a few releases of this is the measured gap between what the suite checks and what the release needs, which is usually smaller than the calendar assumed.

Common questions

What is a code freeze in software development?

A code freeze is a period before a release when a team stops merging changes into the release branch, apart from fixes someone approves one at a time. The aim is that the build which ships is the same build that was tested, with nothing added after the last check.

What is the difference between a code freeze and a feature freeze?

A feature freeze stops new features from entering a release while bug fixes and polish keep flowing in. A code freeze is stricter and stops nearly every change, admitting only fixes approved individually. Many release schedules run the feature freeze first and the code freeze closer to the release date.

Can bug fixes be merged during a code freeze?

A code freeze usually admits fixes through an exception process: someone with release authority approves each one, and the fix is re-tested before it merges. Each exception is a late change, so the regression run around it matters more than usual, since there is little time left to catch what it breaks.

Why do online shops freeze code before Black Friday?

Many online shops freeze code before Black Friday because any deploy carries operational risk, even a correct one, and an incident costs the most during the busiest trading hours of the year. A peak-season freeze lowers exposure when the price of a failure is highest, which no regression suite can do.

How long should a code freeze last?

A code freeze should last as long as the checks it waits for, and no longer. A team that runs its full regression suite against the release candidate and scopes its manual pass to what changed needs a short freeze; one waiting for problems to surface on their own needs a longer one.

How can a team shorten its code freeze safely?

A team shortens a code freeze by recording every defect the freeze finds and adding the test that would have caught it earlier. When several releases in a row show the freeze finding nothing the pipeline missed, the routine freeze can shrink, while peak-season and holiday freezes stay.

The Notarus team

The people who build Notarus. We write up the decisions behind the product when the reasoning is worth more than the changelog line.

Keep reading

Reports with the evidence attached.

Start free, one script tag.

<script src="…/widget.js" data-key="pb_pub_…">