Azure Burstable VMs - Will it Burst?

Will it Burst?

Microsoft has announced it's new B-Series burstable VMs. These VMs look like a great option for low priority, low requirement VMs. If you're anything like me, you will look after a bunch of virtual machines that don't require much CPU most of the time and therefore paying for a dedicated core or cores on Azure seems expensive.

B-Series VMs allow your machine to sit idle for most of the day but burst up to the full core allocation for short periods. These VM SKUs are very affordable compared to the other VM series.

How then, do you know if your VM is idle enough to run in a burstable SKU? You could carefully check VM performance metrics for every virtual machine you own - or you could run this script to get a quick answer and then dive a bit deeper for those machines that look good.

Check out the repo on GitHub and read the description below to find out more.

Azure-Burst-Check

Script to check if your VM is applicable to a burstable VM size. Good for ensuring your VM is idle enough, for enough time that it could run as a burstable machine.

The script will load the VM diagnostics for a specified period and will then do some rudimentary checks to see if it could run in a burstable VM SKU. This script only looks at CPU usage on the host and the memory size of the input VM. It does not take in to account disk or any other resource constraints. Ensure the burstable size shown fits your other requirements before resizing the machine.

Additionally, the script assumes 25% utilisation on a 4-core VM is basically saturating a single core and would be OK on a 1-core VM. The script also assesses all cores as equal, which is not the case in reality - an Av1-Series core is considerably slower than a Dv2-Series core.

The output will show all burstable VM sizes and an applicability rating for the input VM as "Good fit", "Possible fit" and "Poor fit".

The Throttled column shows the amount of data points that would have caused throttling of the VM due to the machine running out of credits. This is a little misleading since if the machine was throttled then it would take longer to do it's work and likely throttle for longer than the figure shows. I would advise if there is any significant (>5%) throttling then the VM size is unsuitable unless the VM is very low priority.

If you do decide to switch to a burstable VM, then I would suggest applying an alert to the new credit metric to show when the VM is running with low credits and increase the VM size accordingly.

Charts

Example VM that is unsuitable for B1s SKU - Out of credits

Example VM that is suitable for B2s SKU

Unable to create new files and folders on NAS share from HPE StoreOnce device. isFileExists

Issue

Unable to create new files and folders on NAS share from HPE StoreOnce device. May affect other windows SMB clone servers.

You see errors like the following in Windows Explorer:

Could not find this item
This is no longer located in <%3 Null:OpText>. Verify the item's location and try again.

There is a problem accessing \\server\sharename
Make sure you are connected to the network and try again.
sharename

You see errors like the following in Veeam Agent for Windows Backup log:

Error: The specified network name is no longer available. Failed to process [isFileExisits]

Cause

The Microsoft network client: digitally sign communications (always) policy setting is set to enabled

Resolution

Set the following policy to Disabled and reboot the machine.

Computer Configuration > Windows Settings > Security Settings > Local Policies > Security Options > Microsoft network client: Digitally sign communications (always)

Azure CPU Price to Performance Roundup

I have recently been running some benchmarks on Azure Virtual Machines... Lots of benchmarks!

Update: Script published on GitHub here! This is not a finished product and is a bit "hacky" - you have been warned! :)

In fact, I've written a script which will power up each VM type that is available to me on an MSDN subscription and make it run Cinebench R15. My MSDN subscription has the default cores per region limit and therefore the largest machine I was able to test was the D15_v2 at 20-Cores.

The script ran the benchmark 3 times to try to account for time based variance such as background processes. I tried to minimise background processes by disabling Windows Update and Defender on the machine.

The time taken to run through all of the machine types available to me serially and run Cinebench 3 times was not as long as I initially expected - about 24 hours for a full run. Due to this, I'm open to running other benchmarks that can be run from the command line and will output in some standard fashion. I might do a strawpoll if anyone is interested.

Keep in mind that Cinebench only tests CPU performance, so this will not be relevant for other machine uses such as GPU or Disk IO.

Price to Performance

Price to performance for each VM series and CPU type was calculated by dividing the Cinebench score (average of 3 runs) by the price per month of the Virtual Machine. The results are displayed as an average score of all the VM types in the series. 




From the results, you can see that the F series VM is by far the best performing per pound spent. The defunct G series VM is the most expensive for CPU.


Real world Cinebench R15 results vs Microsoft 'Azure Compute Unit' figures

I have normalised the Cinebench per-core, per-thread and ACU scores for each VM type and then averaged the normalised value to each VM series.




The best performing per-core virtual machine type is the H series, using the Intel Xeon E5-2667 v3 Haswell at 3.2 GHz. 

Looking at the results shows that almost all of the CPUs (relative to the H series) perform similarly to their respective ACU score which means that the ACU is a good benchmark to gauge relative CPU performance by.

The only outlier on this chart is the Dv3 series when looking at per-core values. Since this VM is using Hyper-Threading, there are 2 threads per core and therefore it performs significantly better relative to the other non-SMT virtual machine types. This is however reflected in the pricing of the VM and so the H series is still top dog on CPU price to performance.

A final note

I have looked at the variance on per core and per thread scores (divide multi core score by number or cores / threads,) and the variance on all the VM types in a series is very low. The only noticeable difference is the G series, where the Cinebench R15 multi core score does not scale linearly. Per core scores go down as the VM size increases. It's probably worth investigating single core benchmarks on these machines to see if there is some artificial limiting happening or if this is due to Intel Turbo Boost kicking in on the CPU.





Let me know if you think there might be a good benchmark to run on the entire series of Azure VMs?

Here is a Google Sheet with all of my results. 

Here is the script

Dave.

Windows Server Containers and Docker

I made a Windows Server Container with Docker tutorial over on GitHub.


Here are the instructions:



You can find the repository here.

Azure D-Series v3 Performance Comparison - Does Hyperthreading mean better price to performance?

Microsoft has just announced their new Dv3 and Ev3 Series VMs taking advantage of Hyperthreading on their Intel Xeon Broadwell CPUs. They suggest an up to 28% price reduction based on the Dv2 VMs with comparative vCPUs on each VM size.

I wanted to see how the new VM sizes compare to the older sizes and see if there is any price to performance benefits of the new machines. I've done some testing in these new VMs with Cinebench to see how the new VMs compare to the old VMs on the popular synthetic CPU benchmark.

Observations

While doing my testing, I noticed some interesting features of the new VMs.

Here are the Core and Thread counts for the comparative DS4 v2 and the D8s v3. Both VMs are shown as 8 "core" VMs in the marketplace. The new v3 VM has 8 Threads and only 4 cores, whereas the older v2 VM has 8 real cores.

D8s v3 - 8 virtual CPU VM	DS4 v2 - 8 virtual CPU VM

It's apparent from the above screen grabs from Cinebench that the hypervisor is presenting Hyperthreading up to the guest OS. 

While testing, I made note of the core and thread count of each VM and also the processor type on each VM. The new v3 VMs present their processors up to the operating system as hyperthreaded logical cores, while the v2 VMs present "full cores" to the OS.

Cores vs Threads

The new v3 VM types are all hyperthreading enabled VMs which means that for each pair of "virtual CPUs" there is a single underlying core on the physical processor in the server. This means that the machines will perform better per core, however they will not perform 2x better per core.

Below is a table of the VM types that I tested with their core and thread counts, and also the CPU type that was detected in each machine.

Server Type	Cores	Threads	CPU Type
DS2 v2	2	2	Intel Xeon E5-2673 v4 (1 socket, 2 virtual processors)
D2s v3	1	2	Intel Xeon E5-2673 v4 (1 socket, 2 virtual processors)
DS3 v2	4	4	Intel Xeon E5-2673 v4 (1 socket, 4 virtual processors)
D4s v3	2	4	Intel Xeon E5-2673 v4 (1 socket, 4 virtual processors)
DS4 v2	8	8	Intel Xeon E5-2673 v4 (1 socket, 8 virtual processors)
D8s v2	4	8	Intel Xeon E5-2673 v4 (1 socket, 8 virtual processors)

Testing Setup

I created a new VM on one of the sizes, waited for initial setup to complete and for the machine to become idle, then ran the multi-core and single-core benchmarks in Cinebench. Once I had gathered the results and some screenshots, I resized the same VM into a different model and ran the tests again.

I only had time to run a one single and one multi threaded test per VM type, but I hope to address this soon with full, multi run benchmarks.

Results

The results were not surprising based on the thread and core counts of the VMs. When testing normal desktop CPUs, I have seen similar results from Hyperthreaded and non Hyperthreaded CPUs. 

Below is the multi-core Cinebench score for each VM type that I tested. You can see that the equivalent VM type on the v3 VMs scores substantially lower then the v2 VM with the same number of virtual CPUs (threads)

The multi-core Cinebench score for each VM type tested, the top 2 machines are the 2-vCPU VMs, the second set are the 4 vCPU VMs and the last 2 machines are the 8-vCPU VMs.

You can see from the above chart that the equivalent v3 virtual machines score considerably lower that the v2 machines.

The next chart is something I put together to show the relative value of each machine type based on Cinebench score. It is calculated as (Multi-core Cinebench Score) ÷ (£ GBP per day cost to run the VM)



Cinebench scores weighted against cost to run the VM.

You can see that the v2 VMs score considerably higher than the v3 VMs in cost per cb score.

Here is my full spreadsheet of results.

Conclusion

Microsoft appears to be offering machines at a lower cost per core, however if you are looking for raw performance per £ spent, it's better to keep using the v2 D-Series VMs at this time.

Convert Azure Windows virtual machine license to Hybrid Use Benefit

EDIT: This functionality is now directly in the portal. Just select the VM, click configuration and you should now see the ability to switch license type.

Microsoft has recently announced their hybrid use benefit for Windows virtual machines. They claim a 40% lower price on Windows VMs in Azure which is certainly a good thing.

If you license your on premises hosts with Windows Datacentre and also run some Azure Windows VMs then you can use the license for both - at the same time!

"Each two-processor licence or each set of 16-core licences are entitled to two instances of up to 8 cores, or one instance of up to 16 cores. The Azure Hybrid Use Benefit for Standard Edition licences can only be used once either on-premises or in Azure. Datacenter edition benefits allow for simultaneous usage both on-premises and in Azure."

The only issue with this offer is that you need to enable the hybrid use at deploy time on your VMs. This isn't a problem for ephemeral VMs, but if you have permanent VMs that you want to save some money on, then you need to re-provision them.

It is possible to delete your VMs and then recreate them using the original VHD, with the hybrid use benefit enabled. I've written a script which will do just this and published it on GitHub here.

As always, DO NOT use this in production and make sure you have backups of your machines before you use the script!

Be aware that this will only work on some VMs, I've not yet worked out the prerequisites. The script *should* detect if it hasn't worked and put your VM back with it's original license if the ARM platform throws an error. To put this in perspective, I've run the script on 8 VMs, 1 of which didn't work.

Update: Should now work with managed disk VMs