Emergency Change Management: Please Stop The Drama

There’s a common change management quote that states something like: “Change is inevitable in any organization, but with change comes risk.” Although personally I prefer the Spider-man quote: “With great power comes great responsibility.” Which rather interestingly is equally true about change management (or change control/enablement if you’re now living in an ITIL 4 world).

This blog looks at emergency changes, which ITIL 4 describes as:

“…changes that must be implemented as soon as possible; for example, to resolve an incident or implement a security patch. Emergency changes are not typically included in a change schedule, and the process for assessment and authorization is expedited to ensure they can be implemented quickly. As far as possible, emergency changes should be subject to the same testing, assessment, and authorization as normal changes, but it may be acceptable to defer some documentation until after the change has been implemented, and sometimes it will be necessary to implement the change with less testing due to time constraints. There may also be a separate change authority for emergency changes, typically including a small number of senior managers who understand the business risks involved.”

Source: AXELOS, “ITIL Foundation: ITIL 4 Edition”

I’ll start by putting emergency changes into context.

The two sides of the change coin

Change is usually a positive thing, whether that change is:

• Reactive to resolve errors or to adapt to changing circumstances; or
• Proactive to provide new services.

But the reverse side is where uncontrolled or poorly managed changes lead to business-affecting incidents and problems and/or cause issues with corporate compliance initiatives.

It’s an interesting contradiction, especially when change management/control is considered one of the most used and most mature ITIL-espoused IT service management (ITSM) processes/practices after incident.

The cause in my opinion? The long shadow thrown by emergency changes.

Emergency change? Really?

Pop quiz:

• How many emergency changes does your organization have?
• Ever wondered how many of these emergency changes cause incidents?
• How many of these emergency changes really were emergencies?
• Does your organization confuse emergency and expedited changes?

There’s a big difference between emergency and expedited changes, and the need and delivery mechanism. But while they’re different beasts the respective volumes should both still be minimized, as both bring added business risks. Please check out Pink Elephant-er Troy DuMoulin’s great 2007 (but still relevant) blog on handling expedited versus emergency changes from which I borrow his definitions:

• Emergency changes – changes that are required to restore service due to an incident or a change that needs to be implemented quickly in order to avoid one.
• Expedited changes – changes that are required quickly due to a pressing need such as legal requirement or a business need but are not related to restoring service.

My real beef, however, is the volume of emergency or expedited changes organizations are still dealing with. Let’s collectively call them “fast-tracked” changes.

Fast-tracked changes: getting to an avoidable bad outcome quicker?

Independent of the definitions of emergency and expedited changes, an organization should not tolerate speed-of-change-related risks and incidents ultimately caused by a lack of planning by, or the inefficiency of, those involved in “creating” and “delivering” the change. They, like everyone else, need to realize that there’s a legitimate reason as to why there’s an agreed change management process and timeframe – to protect business operations and outcomes.

Based on my experience (and “ideal world” thinking), fast-tracked changes should be kept to a minimum. Less than 10% of all changes and as close to zero as possible – there’ll of course always be a need, but it should be closely monitored and managed.

Instead, an IT organization should ensure that their change management process (or change control/enablement practice) is as efficient as possible while still providing adequate protection to business operations. I’m sorry for stating the obvious here – sadly, sometimes this isn’t obvious to all.

So, aspire to better emergency change handling

Many see this as handling emergency changes better. But this is one-dimensional, the second dimension is reducing the volume of emergency (or expedited) changes. In my experience fast-tracked changes are often an indicator of a lack of IT operations influence or power, or a lack of ITSM maturity. It might also be a sign that a serious IT crisis is imminent.

If all the IT analyst firms out there are to be believed then fast-tracked changes are often 30-50% of all changes. This should be managed down to less than 10% over a planned timeframe. In terms of improving, a realistic target for the typical organization stuck at the 30-50% level should be < 20% within the first year of improvement and < 10% after the second year. It’s said that the best organizations are below 2%, but this is unrealistic for the average organization. One can also go deeper to look at the ratio of emergency to expedited changes. I once heard that this should be circa 1:2 when looking at fast-tracked changes. Sadly I can’t remember where this ratio is from, but I’m sure it was from someone clever and it sounds reasonable.

Finally, look beyond fast-tracked changes (although DevOps has now doubt impacted such historical stats for many organizations) Don’t ignore the changes that are bypassing or circumventing the formal change management process (or change control/enablement practice) and the risks that they pose.

So how is your change management? What advice would you offer to others? Please let me know in the comments.