Wednesday, October 16, 2024

Just say "no" to code freezes

One of the more insightful conclusions I've reached in my career, if perhaps also one of the more controversial opinions, is that you should always say "no" to code freezes (at least in an optimal development methodology). This isn't always possible, of course; depending on where you are in influence and decision making, you may have only some, or effectively no, input into this decision. However, to the extent that you are prompted with this question and have some input, my advice is to always push back, and I'll elaborate on this below.

The case for code freezes

I've heard a number of different justifications for why people want code freezes, and these desires come in a few forms. Some examples:

  • We need a code freeze
  • We need to defer some code changes for a bit, just until...
  • We need to hold off on this going in because...
  • We have a release schedule where no changes go in during this period
  • etc.

Typically, the justification for the ask is achieving some desired amount of code stability at the expense of velocity, while some "critical" process goes on. The most common case for this is QA for a release, but there are also cases where critical people might be out, before holidays where support might be lacking, etc. In my experience, these asks are also almost always via management, not development, under the pretense that the operational change in necessary to coordinate with other teams and such.

Note that this is, de facto, antithetical to Agile; if you're practicing Agile software development, you're not doing the above, and conversely if you're doing the above, you're not doing Agile. I mention this as an aside, because this is one area where teams and orgs fail at Agile quite regularly.

The reasons this is bad

Any time you're implementing a code freeze, you are impacting velocity. You are also increasing the risk of code conflicts, discouraging continuous improvement of code, and likely increasing overhead (eg: resolving merge conflicts in ongoing work). Furthermore, this can create a strong incentive to circumvent normal workflows and methodologies, by introducing side-band processes for changes during a "code freeze", which can be even worse (eg: "we need to push this change now, we can't follow the normal QA methodology, because we're in a code freeze").

Side anecdote: in a previous company, the manager insisted on a three month code freeze before each release. During this time, QA was "testing", but since QA overlapped with sales support, this was also where all the sales support enhancement requests were injected into the dev queues, as "critical fixes for this release". In essence, the code freeze would have allowed this part of the business to entirely hijack and bypass the normal PM-driven prioritization mechanism for enhancements, and divert the entire dev efforts for their own whims, if not for some push back from dev on the freeze itself (see suggestions below).

Note that this ask is very common; in particular, short-sighted managers (and/or people with higher priority localized goals than overall business success) ask for these fairly frequently, in my experience. It's often the knee-jerk reaction to wanting more code stability, from those who have a myopic view of the overall cost/benefit analysis for process methodologies.

Alternatives and suggestions

To the extent that you have influence on your process, I'd suggest one of the following alternatives when a code freeze is suggested.

Just say "no"

The most preferable outcome is to be able to convince management that this is not necessary, and continue development as normal. This is unlikely in most cases, but I'll list it anyway, because it is still the best option generally, where possible. In this case, I'd suggest emphasizing that better code stability is best achieved by incremental improvements and quick turnaround fixes, as well as better continuous integration testing, and not code divergence. This argument is often not convincing, though, particularly to people with less overall development experience, and/or higher priority myopic goals. It may also not be feasible, given overall org requirements.

Create a branch for the "freeze"

The easiest and cleanest workaround, generally, is to appease the ask by creating a one-off branch for the freeze, and allowing testing/other to be done on the branch code, while normal development continues on the mainline. This is the closest to the Agile methodology, and can allow the branch to become a release branch as necessary. Note that this approach can often require ancillary process updates; for example, pipelines which are implicitly associated with the mainline may need to be adjusted to the branch. But generally, this approach is the most preferable when a freeze is deemed necessary.

Note that the typical drawback/complication with this approach is that developers will frequently be asked to make some changes in parallel in this scenario (ie: on the mainline and the freeze branch). In this case, I suggest mandating that changes happen on the mainline first, then are ported on-demand to the branch. Ideally, this porting would be done by a group with limited resources (to discourage demands for numerous changes to be ported to a "frozen" branch). For extended QA testing, this might encourage re-branching instead if many changes are needed, rather than porting extensively; this is also generally preferable if many changes are asked for during the "freeze".

Create a parallel mainline

This is a functionally identical to creating a branch for the "frozen" code, but can be more palatable for management, and/or compatible with ancillary processes. In essence, in this scenario, dev would create a "mainline_vNext" (or equivalent) when a code freeze for the mainline is mandated, and shift active development to this branch. When the code freeze is lifted, this would then become the mainline again (via branch rename or large merge, whichever is easier).

This approach, as with the above, also induces overhead of parallel development and merging changes across branches. But it satisfies the typical ask of "no active development on the mainline".

Exceptions, or when a real freeze might be necessary

I haven't seen many examples of this, but the one example I have seen is where a company has a truly real-time CI/CD pipeline, where any changes flow directly to production if all tests pass, and there is no mechanism to freeze just the production output, and disruptions to production operations will be catastrophic. In this specific scenario, it might be net positive to have a short code freeze during this period, if the risks cannot be mitigated any other way. In this case, the cost to the org (in productivity) might be justified by the risk analysis, and as long as the time period and process is carefully controlled, seems like a reasonable trade-off.

The ideal

Included just for reference: what I would do if I were dictating this process within a larger org.

  • Allow branches for QA testing and/or stability wants
  • Limit resources for merging into branches (QA or historical)
    • Ideally, have a separate support dev team for all historical branches
    • Encourage re-branching from mainline if merging ask requirements exceed allocated resources
  • Configure pipelines to be flexible (ie: allow release candidate testing on branches, production deployment from release branch, etc.)
  • Mandate no code freeze ever on the mainline (always incremental changes)
    • Solve any asks for this via alternative methods
  • Encourage regular and ancillary integration testing on the mainline (ie: dogfooding)

Anyway, those are my [current] opinions on the matter, fwiw.


No comments: