Note: Stanislav Shalunov wrote an excellent article on TCP over WAN Performance Tuning and Troubleshooting. In this series of articles, I want to summarize his findings and provide tuning suggestions for popular operating systems.
Most operating systems today ship with TCP stacks optimized for either LAN (high speed, low latency) or broadband (low speed, high latency) traffic. In the case of statewide or nationwide networks, these settings fail miserably. For example, on a statewide network connected via an OC3, a FreeBSD server with default settings will only get about 13 Mbps. On a nationwide network connected via DS3, a Mac mini gets only 8 Mbps with default settings.
TCP uses the Slow Start and Congestion Avoidance algorithms to determine how many packets can be transmitted between the sender and receiver. This is usually expressed in terms of the congestion window. The congestion window is the maximum amount of data that can be transmitted without receiving an acknowledgment from the receiver. If an acknowledgment isn't received, TCP assumes there is congestion on the network and slows down the packet transmission. In a high-speed, high-latency network, the default window size in most operating systems causes TCP to scale back the transmission rate. With a larger window size, the receiver has a chance to acknowledge the incoming packets and the high-speed transmission is maintained.
To determine how much throughput can be expected, take the window size and divide it by the RTT in seconds. The chart below shows the calculations for various window sizes and and RTTs. In the following table, are throughput speeds are Mbps.
| Window Size | RTT 20ms* | RTT 60ms* | RTT 80ms* | Note |
|---|---|---|---|---|
| 8 KB | 3.3 | 1.1 | 0.8 | |
| 16 KB | 6.6 | 2.2 | 1.6 | Linux 2.6, Windows default |
| 32 KB | 13.1 | 4.4 | 3.3 | FreeBSD, Mac OS X default |
| 64 KB | 26.2 | 8.7 | 6.6 | |
| 128 KB | 52.4 | 17.5 | 13.1 | |
| 256 KB | 104.9 | 35.0 | 26.2 | |
| 512 KB | 209.7 | 69.9 | 52.4 | |
| 1 MB | 419.4 | 139.8 | 104.9 | |
| 2 MB | 838.9 | 279.6 | 209.7 |
*All throughput speeds are Mbps
It seems obvious that increasing the TCP window size is a necessary modification. Care must be taken to not increase the window size too large. If so, the system may run out of memory. A good rule of thumb is to set the window size to 1 MB.