Dec 07, 2021 17:00:00

Fastly reveals verification results that uncover the lies of Cloudflare's '3 times faster than Fastly'

Cloudflare published

an article on its official blog on November 20, 2021, stating that its service is 196% faster than Fastly's edge computing platform, Compute @ Edge. In response, Fastly points out that Cloudflare's method of measuring performance is flawed and argues that it is a 'meaningless conclusion based on statistical lies.'

Lies, damned lies, and (Cloudflare) statistics: debunking Cloudflare's recent performance tests | Fastly
https://www.fastly.com/blog/debunking-cloudflares-recent-performance-tests

According to Fastly, the tests conducted by Cloudflare have the following six flaws.

-The nodes of the measurement service 'Catchpoint' used for measurement have been carefully selected, and it is suspected that the nodes that lead to convenient results have been extracted.
-Comparison between Cloudflare Workers , which officially supports JavaScript, and Compute @ Edge, which is still in beta with JavaScript support, running JavaScript.
-The Fastly account used by Cloudflare was a free account for trials, which has more limited functions than paid accounts and is not suitable for load testing.
-Since the test time was only one hour, it was difficult to incorporate the effects of unusual events and specific traffic patterns that occur every day into the test results, but it was easy to reflect random bias.
-Although the test code says 'I executed a function that just returns the current time', the sample code that returns a copy of the header of the received request is shown after that, and either description is incorrect. Therefore, the objectivity and reproducibility of the test are not guaranteed.
It makes sense to evaluate the 'time to receive the first byte (TTFB) ', that is, the server response time, in a test with little computational load, a modest payload, and no platform API. It is not a performance measurement.

Based on the above points, Fastly 'expanded the number of nodes from 50 to 673' 'extended the period from 1 hour to 1 week' 'measured with a paid Fastly account instead of a free trial account' 'compiled with Rust instead of JavaScript' We conducted a test to measure TTFB under the condition of 'using the WASM binary that was created'.

The result is below. Of the six regions, Fastly was shown to have shorter TTFBs, or better performance, in four regions except Asia and Africa. Especially in North America and Europe, it was about twice as fast, and in Oceania, it was about 10 times faster. Fastly also said that JavaScript is not used: 'We know that JavaScript support is important to many customers, but we are not happy with the performance of JavaScript in Compute @ Edge. That's why JavaScript support is still in beta. '

As Fastly points out, TTFB alone is not a good indicator of server performance.

Therefore, Fastly measured 'Network Round Trip Time (RTT) ' and 'TTFB' in the Asian region where TTFB was slower than Cloudflare, and graphed the median as follows. Of the two graphs, the left is the result of Fastly's standard service type, Varnish Configuration Language (VCL), and the right is the result of Compute @ Edge. In particular, in VCL, it was found that the cache hit is faster as the network RTT (blue) and TTFB (green) are almost the same.

From this point, Fastly concludes, 'It's absolutely impossible for Cloudflare Workers to be 196% faster than Compute @ Edge.' In addition, Fastly is working on a benchmark to evaluate performance in a more realistic environment, and plans to announce the benchmark result again when it comes out from a third party.