Trouble in Patchland – the System Administrator Perspective

 The ongoing battle between Google and Microsoft recently heated up with a Google employee going public with details of an unpatched exploit in Windows XP only 5 days after notifying Microsoft of the vulnerability. There appears to be some politics (for want of a nicer word) going on behind the scenes of this episode, and there’s plenty of analysis of that out in the blogosphere. What interests me, though, is how this highlights the difficulty that patching poses for the system administrator.

Patches are unavoidable, unfortunately, and there is a conflict between the need to get a patch out quickly and the need to ensure that patches are fully tested before they break things. Vendors cannot test patches with every application out there, especially in-house software, and so there will always be a percentage of patches that break things. This conflict is one of the reasons that Microsoft started Patch Tuesday. They have received some criticism for no longer issuing patches as they become available, but I think that it’s actually a fairly good compromise. Large sites need a more predictable release cycle to deal with the flood of patches that have become a security reality. It would be nice of Microsoft could also deliver patches more quickly to smaller customers, but once a patch is released an exploit can be reverse engineered out of it leaving the others exposed. Once a month seems to work pretty well, all-in-all.

Now administrators have gotten in the cross-fire in what appears to be a battle between two behemoths. Dealing with patches is hard enough as it is, it’s unfortunate that this eruption could probably make things worse. Would we have been worse off had Microsoft dragged its feet in releasing a patch (which is what Google seems to claim would have happened had they not gone public) or could sites be affected in the time before the patch comes out? I honestly don’t know, but I tend to lean towards Google’s actions making things worse. Luckily these types of public disclosures don’t happen very often, so I really hope we’re not seeing the start of a trend.

In the end it still comes down to the lowly system administrator. Patches need to be tested and deployed, no way to get around it. Hopefully these kinds of issues can lead to better patching processes, but in the mean time it’s us admins that really on the line.