How Do You Patch 100 Database Servers?

Voice of the DBA

In most of the organizations I’ve worked for or consulted with, patching was always a challenge. Patching hasn’t usually been given a priority and is often skipped when operations staff is busy. This has resulted in lots of un-patched, or slowly patched systems. I assume this is one reason Microsoft continues to release RTM-GDR patches because some people won’t patch at all unless there are critical fixes.

I also know that much of IT management sees patching systems like patching parking lots. Needs to be done, but tomorrow, after we do other important work today.

Patching isn’t easy, in fact, Allan Hirt says it was never easy, but these days we don’t get the downtime over a weekend to patch, and there is a desire to patch security issues immediately because of the potential reputational (more likely) or regulatory (less likely) risks. Also, we often need to patch dozens, if not hundreds or thousands, of systems.

So is there a way that most organizations do this? It’s interesting in the piece above that Allan notes that most of us have technical debt, and this debt consists of more than just code and systems. It’s also people, budget, politics, and more. This is even more of an issue if you didn’t write the software. Applications often limit patches or upgrades, and it can be hard enough to get applications certified on new platforms when you control everything. If you purchased something from a vendor, you might be bound by their timelines not your own.

So how do you patch a lot of systems? There are lots of ideas and potential solutions. I’m sure Allan covered some recently in his session at the PASS Data Community Summit. For me, this boils down to building part of a process, using it, evaluating it, and then rolling it out wider. This might mean I need two processes because some systems will lag behind others for various reasons. I’d probably spend a year or two slowly adding to this process and getting it better, all the while ensuring I used automation as much as humanly possible to process notifications, approvals, and actually deploy code.

Start small, experiment, test, evaluate, make changes. Always codifying things that I can. It’s a method that has worked for a long time.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.