Happy New Year : Now Reboot all your Azure VMs (Updated)

 

Happy New Year, my fellow admins!

In a wonderful start to 2018, Microsoft has decided to enforce a reboot of customer VMs in several regions.

I’d recommend you check your Azure Service Health blade’s “Planned Maintenance” section if you run anything in Azure before January 9th 12:00 UTC to see if you are affected!

Text of the notice below:

Dear Azure customer,

Your Azure Virtual Machines (VMs) require an important security and maintenance update. The vast majority of Azure updates are performed without impact but, for this specific update, a reboot of your VMs is necessary.

A maintenance window has been scheduled starting January 10th 2018 (00:00 UTC) during which, Azure will automatically perform the required VM reboot. An affected VM will be unavailable for several minutes, as it reboots. For any VM in an availability set or a VM scale set, Azure will reboot the VMs one Update Domain at a time to limit the impact to your environments. Additionally, operating system and data disks will be retained during this maintenance.

You have one or more VMs that are eligible to initiate self-service maintenance proactively at any time between now and January 9th, 2018 (12:00 UTC). You can see which VMs are eligible for a self-service maintenance and initiate this step using the Azure Portal (Use the Azure Service Health link below). Also see the how-to guides for Windows/Linux VMs to learn more.

If you complete this self-service maintenance step before January 9th, your VMs will be marked as updated and will not be impacted by the scheduled maintenance window. If you initiate self-service maintenance, the temporary disk will not be maintained.

Interestingly, it appears that VMs deployed after November 2017 are not affected (at least for my subscription).

There’s been a lot of discussion in the places of the internet where admins like to lurk over the last few days of a rumoured vulnerability in Intel CPUs, for a full explanation of the issue I’d recommend checking out Python Sweetness’ blog here.

It may be a co-incidence, but I think it’s interesting that:

  • NT patches appeared in November
  • Azure Virtual Machines since November (for me at least) appear to be unaffected
  • This issue is rumoured to affect Hyperscaler virtualisation.
  • Remediation requires a reboot.

And, even more interesting, the “eligible self-service maintenance” Microsoft speaks of, is actually a Redeployment of the Virtual Machine, which suggests host migrations might be part of the background work that is going on in this “important security and maintenance update”.

I’m a little annoyed that this didn’t go out as an email, but I’m also going to go through and check my Service Health notification settings. I only spotted it by sheer luck.

I’m guessing the reason that there is little more information on this is that the issue is still under embargo, as there’s talk that the issue only affects Intel CPUs. 

Oh, and also, Intel’s CEO sold a bunch of his stock in about mid-december…… 

Good luck, and happy remediation!

Update 01:00 UTC 04JAN:

Well… this is completely irrelevant now as Microsoft runs through redeploying all Azure VMs. I’m guessing that Google published full details in response to Intel’s awesome press release.

Update 20:08 UTC 03JAN:

Have found out via Microsoft Support that the service health blade is horribly inaccurate and there’s actually 5 possible states for your virtual machine/s to be in:

  • Self-Service Maintenance Avaliable
  • Scheduled Maintenance Required – Stop/Deallocate -> Start Reccomended
  • Scheduled Maintenance Required – Redeploy recommended
  • Scheduled Maintenance Required – Redeploy can be considered
  • Already Updated

Apparently the only way to tell which VM is in which state is to run a report that only Microsoft Support can run, which means for those Azure admins without a current support agreement I wish you the best of luck! 🙁

I ended up logging a case due to “remediating” a virtual machine a couple of times to only have it pop back to “maintenance scheduled” status.

I’ve asked for further clarification on each status’s meaning, as well as when the Azure Service Health blade will be fixed so it’s actually worth reading the “Affected” list.