Azure CPU Price to Performance Roundup

I have recently been running some benchmarks on Azure Virtual Machines... Lots of benchmarks!

Update: Script published on GitHub here! This is not a finished product and is a bit "hacky" - you have been warned! :)

In fact, I've written a script which will power up each VM type that is available to me on an MSDN subscription and make it run Cinebench R15. My MSDN subscription has the default cores per region limit and therefore the largest machine I was able to test was the D15_v2 at 20-Cores.

The script ran the benchmark 3 times to try to account for time based variance such as background processes. I tried to minimise background processes by disabling Windows Update and Defender on the machine.

The time taken to run through all of the machine types available to me serially and run Cinebench 3 times was not as long as I initially expected - about 24 hours for a full run. Due to this, I'm open to running other benchmarks that can be run from the command line and will output in some standard fashion. I might do a strawpoll if anyone is interested.

Keep in mind that Cinebench only tests CPU performance, so this will not be relevant for other machine uses such as GPU or Disk IO.

Price to Performance

Price to performance for each VM series and CPU type was calculated by dividing the Cinebench score (average of 3 runs) by the price per month of the Virtual Machine. The results are displayed as an average score of all the VM types in the series. 




From the results, you can see that the F series VM is by far the best performing per pound spent. The defunct G series VM is the most expensive for CPU.


Real world Cinebench R15 results vs Microsoft 'Azure Compute Unit' figures

I have normalised the Cinebench per-core, per-thread and ACU scores for each VM type and then averaged the normalised value to each VM series.




The best performing per-core virtual machine type is the H series, using the Intel Xeon E5-2667 v3 Haswell at 3.2 GHz. 

Looking at the results shows that almost all of the CPUs (relative to the H series) perform similarly to their respective ACU score which means that the ACU is a good benchmark to gauge relative CPU performance by.

The only outlier on this chart is the Dv3 series when looking at per-core values. Since this VM is using Hyper-Threading, there are 2 threads per core and therefore it performs significantly better relative to the other non-SMT virtual machine types. This is however reflected in the pricing of the VM and so the H series is still top dog on CPU price to performance.

A final note

I have looked at the variance on per core and per thread scores (divide multi core score by number or cores / threads,) and the variance on all the VM types in a series is very low. The only noticeable difference is the G series, where the Cinebench R15 multi core score does not scale linearly. Per core scores go down as the VM size increases. It's probably worth investigating single core benchmarks on these machines to see if there is some artificial limiting happening or if this is due to Intel Turbo Boost kicking in on the CPU.





Let me know if you think there might be a good benchmark to run on the entire series of Azure VMs?

Here is a Google Sheet with all of my results. 

Here is the script

Dave.

Azure D-Series v3 Performance Comparison - Does Hyperthreading mean better price to performance?

Microsoft has just announced their new Dv3 and Ev3 Series VMs taking advantage of Hyperthreading on their Intel Xeon Broadwell CPUs. They suggest an up to 28% price reduction based on the Dv2 VMs with comparative vCPUs on each VM size.

I wanted to see how the new VM sizes compare to the older sizes and see if there is any price to performance benefits of the new machines. I've done some testing in these new VMs with Cinebench to see how the new VMs compare to the old VMs on the popular synthetic CPU benchmark.

Observations

While doing my testing, I noticed some interesting features of the new VMs.

Here are the Core and Thread counts for the comparative DS4 v2 and the D8s v3. Both VMs are shown as 8 "core" VMs in the marketplace. The new v3 VM has 8 Threads and only 4 cores, whereas the older v2 VM has 8 real cores.

D8s v3 - 8 virtual CPU VM	DS4 v2 - 8 virtual CPU VM

It's apparent from the above screen grabs from Cinebench that the hypervisor is presenting Hyperthreading up to the guest OS. 

While testing, I made note of the core and thread count of each VM and also the processor type on each VM. The new v3 VMs present their processors up to the operating system as hyperthreaded logical cores, while the v2 VMs present "full cores" to the OS.

Cores vs Threads

The new v3 VM types are all hyperthreading enabled VMs which means that for each pair of "virtual CPUs" there is a single underlying core on the physical processor in the server. This means that the machines will perform better per core, however they will not perform 2x better per core.

Below is a table of the VM types that I tested with their core and thread counts, and also the CPU type that was detected in each machine.

Server Type	Cores	Threads	CPU Type
DS2 v2	2	2	Intel Xeon E5-2673 v4 (1 socket, 2 virtual processors)
D2s v3	1	2	Intel Xeon E5-2673 v4 (1 socket, 2 virtual processors)
DS3 v2	4	4	Intel Xeon E5-2673 v4 (1 socket, 4 virtual processors)
D4s v3	2	4	Intel Xeon E5-2673 v4 (1 socket, 4 virtual processors)
DS4 v2	8	8	Intel Xeon E5-2673 v4 (1 socket, 8 virtual processors)
D8s v2	4	8	Intel Xeon E5-2673 v4 (1 socket, 8 virtual processors)

Testing Setup

I created a new VM on one of the sizes, waited for initial setup to complete and for the machine to become idle, then ran the multi-core and single-core benchmarks in Cinebench. Once I had gathered the results and some screenshots, I resized the same VM into a different model and ran the tests again.

I only had time to run a one single and one multi threaded test per VM type, but I hope to address this soon with full, multi run benchmarks.

Results

The results were not surprising based on the thread and core counts of the VMs. When testing normal desktop CPUs, I have seen similar results from Hyperthreaded and non Hyperthreaded CPUs. 

Below is the multi-core Cinebench score for each VM type that I tested. You can see that the equivalent VM type on the v3 VMs scores substantially lower then the v2 VM with the same number of virtual CPUs (threads)

The multi-core Cinebench score for each VM type tested, the top 2 machines are the 2-vCPU VMs, the second set are the 4 vCPU VMs and the last 2 machines are the 8-vCPU VMs.

You can see from the above chart that the equivalent v3 virtual machines score considerably lower that the v2 machines.

The next chart is something I put together to show the relative value of each machine type based on Cinebench score. It is calculated as (Multi-core Cinebench Score) ÷ (£ GBP per day cost to run the VM)



Cinebench scores weighted against cost to run the VM.

You can see that the v2 VMs score considerably higher than the v3 VMs in cost per cb score.

Here is my full spreadsheet of results.

Conclusion

Microsoft appears to be offering machines at a lower cost per core, however if you are looking for raw performance per £ spent, it's better to keep using the v2 D-Series VMs at this time.

Microsoft Azure L-Series Local SSD Performance

I wanted to take a quick look at the new L-Series local SSD performance. It's worth noting that this storage is the non persistent, 'temporary' D: drive on the VM so should only be used for cache data that can be easily recreated.

According to Microsoft the local SSD performance is dependent on the machine type:

"The cache is subject to separate IOPS and throughput limits at the VM level, based on the VM size. DS-series VMs have roughly 4,000 IOPS and 33 MB/s throughput per core for cache and local SSD I/Os. GS-series VMs have a limit of 5,000 IOPS and 50 MB/s throughput per core for cache and local SSD I/Os."

https://docs.microsoft.com/en-us/azure/storage/storage-premium-storage

I did a couple of IOMeter tests to check sequential and random performance of the drive. I used 4 workers with 1 outstanding IO each on a 20 GiB test file. I tested both the L4 and L16 instance sizes.

L4, Sequential Read: Roughly 200 MiB per second (32 KiB, 100% Read, 0% Random,) Writing out the test file ran at just over 200 MB/sec.

L4, Random Read: gets roughly 20k IOPS

L4, Random Write: also gets roughly 20k IOPS

So it appears that the throughput is being throttled due to the very predictable performance figures being seen. Looks like the throttle for these VMs is set the same as the GS series VMs at 5,000 IOPS or 50MB/sec per core.

I have also performed the same tests on an L16 server which has 16 cores and 2.8 TiB of local SSD to see if they are throttled at the same level. I used the same tests as the L4 server with the exception of using 16 workers. Using only 4 workers on this VM resulted in significantly worse performance to 16 workers. Perhaps the throughput throttling is actually working at the core level?

In theory these results should be 800 MB/sec sequential reads or 80,000 random IOPS.

Writing the test file out was running around 620 MB per second (using 4 workers). Because of this, I've included a sequential write test with 16 workers since this is a bit worse than expected.

L16, Sequential Read

L16, Sequential Write

L16, 4k Random Read

L16, 4k Random Write

L16, 4k Random Write using only 4 workers 
results in significantly worse performance, although it doesn't scale linearly with worker count. We are still seeing 45,000 IOPS with this worker configuration. 

Charts

Conclusions

From the tests we can see that performance scales linearly with core count. An interesting note is that the worker count is very important to get full performance from the drive. This would translate to setting the correct thread count in SQL Server or similar.

The L-Series local SSD and cache appears to perform the same as the GS series VMs. It appears that Microsoft is delivering on the stated IO figures very well with low average latency of less than 1 ms.