A Comparison of SPDY-over-SCTP and SPDY-over-TCP

Below are some prelimianry test results for SPDY-over-SCTP compared to SPDY-over-TCP. These results don't represent the full compliment of test conditions that I plan to cover, but they do represent some common scenarios, and are interesting. In choosing the test conditions, I referred to Jerry Chu's July 2009 IETF presentation [1] on tuning TCP for loss rate guidance. He reported 0.8% - 2.4% packet loss on Google front end servers, and I used 1% and 2%. I referred to the Akamai Q3 2010 report on the state of the Internet [2] for information on bandwidths. There's a lot of information there and I haven't yet had time to test a range of bandwidths at this point (let alone asymmetric bandwidths), but 5 Mbps seemed to be a good central point to start with and that is what I used. I also referred to Jerry Chu's IETF presentation [2] for data on RTT distribution. Jerry's data shows that Google's servers see 50% of connections with RTTs over approximately 70ms. I'd like to test RTTs of 20ms, 50ms, 100ms, and perhaps 200ms, but at this point I've only tested 100ms. Here are some details on the test conditions:

Here is some additional detail and a summary of the test results:

At 1% loss the average page load time is half a second better when using SPDY-over-SCTP. Four pages are more than 5% slower using SCTP, eight pages are more than 5% faster using SCTP, and the remaining 11 pages are within +/-5% of the TCP page load time. At 2% loss the average reduction in page load time is nearly 1.5 seconds, and SCTP provides a substantially better user experience than TCP. Only 1 page is more than 5% slower using SCTP, while 15 pages are more than 5% faster using SCTP. At 2.5% loss the improvement should be even more significant.

Below are more detailed results. The reported values are (average TCP page load time) - (average SCTP page load time).

The results are not uniform across the various test pages; some are very sensitive to loss while others are not, and some significantly benefit from SCTP while others do not. I plan to expand these results to cover other network conditions, and it will be important to include a larger set of test pages. A superficial review of the results doesn't reveal an obvious correlation between these results and such per-page attributes as number of SPDY sessions, number of connections or amount of data read. I also want to look into why certain pages gain much more than others from using SCTP. It may be that there are reasonable changes that would improve the performance of pages loaded over SCTP, and thereby provide a more uniform improvement with SCTP.

It is important to point out that lksctp's inability to bundle data with the COOKIE_ECHO impacts many sites by an additional RTT delay. Most of the test pages are loaded from multiple domains, and in many cases the completion time of the page depends on a connection other than the initial connection. In these cases, lksctp's inability to bundle data with the COOKIE_ECHO further delays the completion of the page by 1 RTT. I have looked at the way each of these sites loads and found that 17/23 experience an additional RTT delay. The table below shows the results from the above table adjusted for this additional RTT delay, for those sites where is is appropriate. These are the results I would expect if we were running our tests on FreeBSD or Mac OS X, or if lksctp were patched to allow bundling data with the COOKIE_ECHO.


References:


Contact: jtleight@udel.edu