CAB in the Age of DevOps

The only reason most people and organizations have heard about the CAB (Change Advisory Board) is because it was described in the change management section of the ITIL® tomes, and then eagerly implemented as a quality control mechanism in large enterprises. For many non-IT Operations professionals, CAB is the only exposure they have had to ITIL, and it has become almost synonymous with it. It also means that ITIL itself has become synonymous with bi-weekly pain.

TL;DR

• CAB stands for ‘Change Advisory Board’, not ‘Change Approval Board’
• The CAB is not mandated in ITIL, nor is sending all RFCs to the CAB
• When auditors require CAB decisions, they confuse ends with means
• When managers require CAB decisions, there’s probably a trust issue
• Automation (for software changes) makes even auditors happier
• Software flowing through the CI/CD pipeline is not for the CAB
• Improving change management practices requires disintermediation

The Tale of the Three Confusions

As with many other things in IT, there is a significant difference between the theory and the practice of something, and in this case, we’re looking at change management. Although the theory (in ITSM) has been (partially) built on real-world practices, it is an idealised version of an attempt at an all-encompassing description that is meant to be tweaked to a particular situation — syrup and sparkling water in the same bottle, but unshaken.

This article is an attempt at bringing everybody on the same page so that we can start to collaboratively work on improvements to the change management practice. This is, in my opinion, a preferred approach over finding solace in mud-slinging and ‘off with its head’ types of calls to action, which tend to lead to cut-panic-rename-paste type initiatives in the enterprise.

There are three main areas where the confusion about the CAB and the change management practices in general have resulted in significant problems. I will introduce all three and then go into more detail.

First of all, there is a lot of confusion about what the letter ‘A’ in ‘CAB’ stands for. Secondly, there is confusion about which changes should “go to the CAB”. And thirdly, there is confusion about separating change management practices from release management practices.

‘A’ for ‘Allow’?

The idea of the Change Advisory Board, as the name would suggest, is to advise on the assessment of changes — mostly when it comes to high-risk changes, large-scale changes that go beyond the field of vision of a particular team and require coordination, and situations where we have conflicting priorities and need to make choices due to cost or timelines.

For various reasons, some of which I will cover below, the word ‘Advisory’ has morphed into ‘Approval’ in many enterprises. This has changed the CAB into a delay mechanism that’s allegedly there for quality reasons but often fails to deliver on this promise. The difference is far from being pedantic semantics:

Advisory: having or exercising power to advise
Approval: an act or instance of approving something

CABs in enterprises tend to be static — the same group of managers discussing all changes brought to them, regardless of the team/application/service, occasionally bringing in guest perpetrators to add colour to thick and dull RFCs (Requests for Change). This approach introduces a level of intermediation that loses a lot of detail and is prone to become a type of a performance — technically functioning, practically failing. Failing with advisory is one thing, failing with approvals quite another.

The practice of using RFCs in the enterprise feels more like out of an S&M handbook, rather than that of ITSM. The dynamics of how it has been implemented focus on painful scrutiny, not helpful advice. The RFCs are filled with information for the CAB, not with questions from and answers to the person or team seeking advice.

This is not to say that quality controls should not be put in place, but a Potemkin-type late-in-the-game mechanism is more likely to hinder than help the organization achieve its objectives. Error count minimization is achieved by avoiding change, rather than by improving the system’s resilience, and this, in turn, can often result in a need for big-bang changes when the system has exceeded its best-before date and is failing on multiple fronts.

Improving the practices here requires thoughtful and conscious disintermediation, where impact assessment and decision-making are brought closer to where the work is done.

Outdated practices and lack of trust

As I mentioned above, the CAB can be a useful component of change management for certain types of changes and a lot less useful for some others. Please note that I’m saying “can be”, by which I mean that CAB is optional, not mandatory. It should only be introduced if it is helpful, and its design, scope, role, and value needs to be reviewed on an ongoing basis.

Among others, there are two reasons for the ‘abusive CAB’ I’d like to discuss. The first has to do with auditing practices — the requirement to have every change documented, with approvals, and seeing the CAB as the only/best method for achieving this. While the first part of this requirement is rather reasonable — why wouldn’t we want to keep track of changes — the second part is an instance of preferring the “how” over the “why”. This requirement is often so ingrained that it feels like a law, and it eliminates other ways of working that would achieve the same outcome, but function differently.

Having the CAB sign off every change request might have been the only way the organization knew how to address the auditing requirement in the past. This was definitely not the advice in the ITIL guidance (see e.g. change authority and standard change), and it most likely achieved the objectives of risk management on paper only, but that might have been enough.

Expensive hardware, long procurement times, and even longer planning times made the looks-good-on-paper approaches rather popular. The world has changed, though, and provided there is enough support in the organization for investment in modern computing methods and practices, there is no need to continue with the risk management theatre.

Rather than relying on manual quality gates often managed by people who don’t know the context, we can utilize automation practices, where the detailed documentation of every change comes by default and in addition to improving the quality of the service, we also have much more reliable information to provide to the auditors.

Please note that the automation option mostly applies to software-based changes, which in the context of cloud computing includes parts of infrastructure management. But then, how reasonable is it to address individual software changes as part of the ‘traditional’ change management anyway? I’ll come back to this later.

The second common reason for the CABs being so popular yet misused is mistrust and the need to save one’s ass if something goes wrong. It can be a mechanism to impose control over teams of specialists — because how else would you know they are working on the right things exactly the way we want them to?

In this scenario, rather than utilizing the concept of change authority and letting the teams closest to the work make decisions, all of these decisions get escalated to the CAB level, where the team’s manager’s manager’s managers discuss and decide what changes are OK to go forward. While the decision-power is with the CAB, the person requesting the change is still made accountable. That is, when something does go wrong, the CAB can claim that they had requested for full and truthful information, and the reason for unexpected results must have been due to information being held or misrepresented. And, they explain, as soon as they check who that careless individual was, they’ll make sure this will never happen again …

The RAP battle

The third misconception has to do with changes vs. releases, and it also links to the challenge with the CAB’s scope. Change management looks at whether something should be changed, and when. Release management looks at whether the change package — whatever the contents — is ready to be released, and when. For simplicity’s sake, we can look at release management as part of change management in this context.

The way the CAB has been designed in many organizations for software-related changes is in reality as a Release Approval Process, and a funny one at that. Rather than assessing release readiness (which, again, is something that needs to be done, preferably in an automated manner with plenty of feedback), the RAP-CAB assesses the viability of the change when the work on that change has already started, or quite often, has already been completed.

The decision-making processes for changes (whether a change in software is needed) and for releases (how to best release the change) are disconnected and misaligned. In these organizations, the change management practice — although usually not called that — sits with “the business”, and “IT” is there to take orders once the decision has been reached, deliver what was intended (but rarely properly described), and ensure zero disruption to any services.

There are too many dysfunctions in that scenario to unpack and fit in this article, but what I would like to do is to propose a different lens to look at software changes, and perhaps remove some of the tension between the ITSM community and the DevOps community.

Extending the mandate

When we look at agile software development with continuous integration (CI) and continuous delivery (CD), then in most cases we can assume that the code that is injected into the CI/CD pipeline has been pre-approved in principle (that is, the developer has the mandate to work on it) and requires passing the (automated) quality gates before reaching the production environment.

This pre-approval does not mean that whoever asked for the development work to be done is 100% confident that these specific changes will deliver the expected outcomes, nor does it mean that the developer is 100% confident the code will flow through the quality gates with no issues. The feedback time for the developer is in minutes/hours, and for the customer in hours/days/weeks, depending on the design of the pipeline and the development methodology used.

If the customer has already decided that they want it, then that decision is effectively a CAB-decision, just not the CAB we usually think of. So, rather than even trying to manually assess every software release going through the pipeline in the CAB (or RAP) for whether it should be there, the teams responsible for change and release management should work together to design and improve the CI/CD pipeline and the automated quality gates.

Can we dismantle the CAB, then?

Whatever the design of your existing or planned CAB, the key principle to keep in mind is this — does it help to achieve expected business outcomes?

For change management to be effective, it cannot be limited to a pre-release control gate in IT. That is also to say that change management covers a lot more than just the CAB, and that there are probably quite a few other things in need of an improvement in the wider change management context in your organization.

When it comes to your CAB, then if it is a group of people not familiar with the details of the changes/releases they discuss yet making decisions and avoiding accountability, then this is not a CAB to keep; nor is it a CAB in the sense intended in the first place.

But if it is a group of people who can advise on planned changes, help with coordination and with prioritization, and are there to help the organization rather than their individual careers, then do keep it, but improve it, too.

If you want to assess the viability of changes, then this should be done by people who understand these changes. If you want to assess the viability of releases, then this should be done by people who understand these releases.

To increase quality, you need to disintermediate and automate.