The best way to detect incorrectly stopped Azure VMs

Problem description: 'Stopped' and 'deallocated' states of an inactive VM

In clouds your VMs are temporarily leasing a part of the compute power of the hypervisor costs, and you pay money for that. It is obvious that you pay for them when your VM is powered on, but there are some cases where your VM may still consume money for сompute even if it is turned off.

The most annoying thing regarding charges for turned off VMs is the Azure’s difference between ‘stopped’ and ‘deallocated’ states of an inactive VM. The underlying mechanism is the following:

  1. When you stop your VM using Azure portal, it goes to “Stopped (deallocated)” state in the portal. This means that Azure has destroyed your VM instance on the hypervisor host and released all connected resources, like non-static Public IPs related to this. VM.You don’t pay compute costs for a VM in this state.
  2. When you stop your VM via Guest OS call, it goes to “Stopped” state in the portal. Unlike the previous case, the VM won’t be destroyed on its hypervisor host, so you still rent a part of the hypervisor and pay money for that. In most cases this is not what you want. Actually, the only scenario when you would like to have a machine in “Stopped” state is if you want to shut it down for some very short period of time and save non-static Public IPs connected to it.

How to detect not deallocated VMs

So, you want to scan the subscription for stopped, but not deallocated VMs. It can be done manually through the Azure portal, but as this task should be done periodically, Azure CLI is a much better way to perform this check. az vm list command is a very handy way to show VMs in some subscription (with –subscription parameter) and even to filter them by some condition and format output (using –query parameter). But machine state is not shown in this command output due to performance reasons. Therefore, we should use az vm show command with -d parameter set. The next thing to handle – we want to scan the whole subscription for such VMs while az vm show requires either the name of the VM or a list of the VM IDs as an input parameter. So, the first step is to make a quick az vm list query and format its output as IDs list. And at last, we filter output by VM’s powerState field, also noting that API response has a slight difference in VM state naming: “Stopped (deallocated)” in portal is shown as “VM deallocated”, while “Stopped” state is “VM stopped” in the API response. Polishing with some output formatting, and here is the command for detection of incorrectly stopped VMs: 
az vm show -d --ids $(az vm list --subscription --query "[].id" -o tsv) --query "[?powerState=='VM stopped'].{Id:id, ResourceGroup:resourceGroup}" --output table

You can execute this command in Bash console integrated into Azure portal or set up a periodical job to check your subscriptions.

👆🏻 Overlooked resources are contributed to a company cloud bill, and users don’t even expect that they’re paying for them.
💡 Find the ways of identifying and cleaning up orphaned snapshots to keep MS Azure and Alibaba Cloud costs under control → https://hystax.com/finops-best-practices-how-to-find-and-cleanup-orphaned-and-unused-snapshots-in-microsoft-azure-and-alibaba-cloud