I am currently using Ivanti Patch for Windows Servers to patch our virtual demonstration and test environment and thought I would share a short blog post around some of the common misconfigurations that can be made when setting up the automated patching, this is not an exhaustive list and there are actually a surprising number of different ways to approach the patching of an environment depending on its complexity that are too numerous to list.
Without having already thought through and established a patching strategy for your environment, it’s very tempting to do a big bang and patch everything all at once, not to state the obvious but this is a really bad idea, so my first point is don’t do it.
At best a big bang patching strategy won’t work reliably and at worst you could wind up spending hours restoring from backups if things went really wrong in a perfect-storm type scenario.
So, Point 1 is no big bang patch deployments.
Patching domain controllers (DC’s) in an uncontrolled manner is asking for trouble and whilst you should be applying new critical and security patches in a timely manner, (within 14 days of release normally), don’t rush headlong into patching them immediately just on the off-chance there is a bug in the patch.
I’m referring to Microsoft patches here, you shouldn’t really be running anything on a DC application-wise purely from a security and stability of your DC perspective and if you are I would strongly advise moving applications workloads it off the DC – there are exceptions of course, Azure AAD Connect for example is often run on a DC, best practice would be not to do that but if your constrained for resources it’s something we see done a lot.
OK, back to patching, 7 days should normally be sufficient for any big problems to have been ironed out by Microsoft and the bugs removed and 14 days from a security point of view is the longest you should be leaving critical security patches as you are leaving your DC open to vulnerabilities that will be known by would-be hackers and exploit code.
Point 2 to note is that DC’s should be patched separately to the everything else, apart from anything, trying to patch any other machines whilst the DC is down can result in patch deployment failures due to name resolution issues.
Ivanti Patch for Windows relies on a back-end SQL database, when that database is not available then the Ivanti Patch console and its associated services become unavailable, in a large patch deployment having it go offline will result in a partial patch run with only some of the patches getting deployed – so patching SQL Servers should therefore be handled separately to the rest of your VM’s, provided that the only server being patched is the SQL server, the patches will be downloaded and scheduled for installation on the database server and Ivanti being unavailable during that time is not an issue provided that no other deployments are taking place.
Point 3 is therefore to handle patching of your SQL servers separate to the rest of your environment, do them on their own and give them plenty of time to complete, don’t try and stack other patch deployments on top of them, this may work some of the time but some patches will take longer to apply than others and you may find failed deployments when you come in the following morning. Also bear in mind any services that rely on the other databases hosted ono your SQL server as these will also be affected during the patching.
vCentre & Platform Services Controller Servers
Patching your vCentre Server is not a consideration if you are using vSphere 6.5 since you are more than likely using the vCentre Server Virtual Appliance deployed, but for vCentre installs that are still Windows server based and of course the associated Platform Services Controller then these need to be patched too.
Whilst the patch deployment can use vCentre as a means of targeting the VM’s to be patched the deployment still occurs over the network so vCentre being offline will not prevent patches deploying but it will prevent you from being able to add machines into Machine Groups.
Point 4 is something I would recommend rather than saying it is as important as the points I have detailed above but keeping the vCentre patching separate is again a sensible thing to do but it not as essentials as the other points.
For now the 4 points above should put you on the right track, how you patch the rest of your server and desktop estate I leave upto you but consider any dependencies that the servers may have on other services you provide and treat them accordingly, back-end application servers may cause outages to your external web-services and if they are resilient consider patching just half of your servers first and then do the second half separately so as not avoid an outage.
Happy patching! 😊