Mar 13, 2020 15:00:00

Amazon, Google and Microsoft propose in their own cloud performance comparison report, the comparison policy after 2021 will be revised

Cockroach Labs , a developer of commercial database management systems, published in December 2019, `` Cloud report of 2020 '' comparing the performance of AWS, GCP, Microsoft Azure with benchmarks, inappropriate from Amazon, Google, Microsoft There were some suggestions, such as pointing out a simple comparison method. In response, Cockroach Labs has released a revised comparison method for reporting since 2021.

AWS, Azure, & GCP Respond to the 2020 Cloud Report | Cockroach Labs
https://www.cockroachlabs.com/blog/aws-azure-gcp-respond-to-the-2020-cloud-report/

A report published by Cockroach Labs in 2019 showed that Amazon's AWS performed well overall. You can find out more about the report by reading the following articles:

Results comparing the performance of Amazon, Google, and Microsoft clouds are now available-gigazine

After publishing this report, he received feedback from Amazon, Google and Microsoft on several comparison factors. For example, in the N1 instance used for GCP verification, specify the option `` --min-cpu-platform = skylake '' and point out that the CPU generation to be used should be limited.In Microsoft Azure, basically Pointed out that the `` network acceleration function '' which is enabled by default is disabled, pointed out that the physical distance conditions between virtual machines are not unified between clouds and communication delays are different, etc. It seems he received.

Cockroach Labs has responded and has publicly announced that the 2021 report will make several corrections to the comparison method. First, in the 2020 report, different versions of the OS were installed depending on the cloud, and the versions of the software used for the stress test were not unified. The version of the '

stress-ng ' command used for testing has been unified.

In the network, while using ' iPerf2 ' during the support period for bandwidth measurement, Azure will measure speed by TCP instead of ICMP which does not support acceleration.

Regarding storage, in the 2020 version, the numerical value of the combined storage of local connection and network connection was released, but it will be distinguished from the 20201 version, and it will make it easier to understand the difference between clouds. Benchmarking software will continue to use ' Sysbench ' to measure the performance under heavy load of online transaction processing . However, it is said to adjust the setting in consideration of Microsoft's point that 'benchmark tests that frequently perform fsync that synchronizes memory and storage like Sysbench are not effective'.

Regarding the results of TPC-C , it was a value under the condition where the ext4 write barrier was disabled, but it was pointed out that the state where the write barrier is disabled is at risk for the virtual machine and is inappropriate . However, all cloud companies have not disclosed information on the durability of failures due to the presence or absence of a light barrier, so comparisons are difficult.

Cockroach Labs states that it will work with Amazon, Google and Microsoft companies from the research stage, aiming to deliver more accurate reports to customers.