Operating systems and network tuning

Network tuning can reduce Transmission Control Protocol (TCP) stack delay by changing connection settings and can improve throughput by changing TCP buffers.

Operating systems

A Windows system needs the least tuning while a Solaris system needs the most tuning. The following information pertains to each system specified, and might improve WebSphere® eXtreme Scale performance. You should tune according to your network and application load.

Windows

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\

Tcpip\Parameters

MaxFreeTcbs = dword:00011940

MaxHashTableSize = dword:00010000

MaxUserPort = dword:0000fffe

TcpTimedWaitDelay = dword:0000001e

Solaris

ndd -set /dev/tcp tcp_time_wait_interval 60000

fndd -set /dev/tcp tcp_keepalive_interval 15000

ndd -set /dev/tcp tcp_fin_wait_2_flush_interval 67500

ndd -set /dev/tcp tcp_conn_req_max_q 16384

ndd -set /dev/tcp tcp_conn_req_max_q0 16384

ndd -set /dev/tcp tcp_xmit_hiwat 400000

ndd -set /dev/tcp tcp_recv_hiwat 400000

ndd -set /dev/tcp tcp_cwnd_max 2097152

ndd -set /dev/tcp tcp_ip_abort_interval 20000

ndd -set /dev/tcp tcp_rexmit_interval_initial 4000

ndd -set /dev/tcp tcp_rexmit_interval_max 10000

ndd -set /dev/tcp tcp_rexmit_interval_min 3000

ndd -set /dev/tcp tcp_max_buf 4194304

AIX®

/usr/sbin/no -o tcp_sendspace=65536

/usr/sbin/no -o tcp_recvspace=65536

/usr/sbin/no -o udp_sendspace=65536

/usr/sbin/no -o udp_recvspace=65536

/usr/sbin/no -o somaxconn=10000

/usr/sbin/no -o tcp_nodelayack=1

/usr/sbin/no –o tcp_keepinit=40

/usr/sbin/no –o tcp_keepintvl=10

LINUX

sysctl -w net.ipv4.tcp_timestamps=0

sysctl -w net.ipv4.tcp_tw_reuse=1

sysctl -w net.ipv4.tcp_tw_recycle=1

sysctl -w net.ipv4.tcp_fin_timeout=30

sysctl -w net.ipv4.tcp_keepalive_time=1800

sysctl -w net.ipv4.tcp_rmem="4096 87380 8388608"

sysctl -w net.ipv4.tcp_wmem="4096 87380 8388608"

sysctl -w net.ipv4.tcp_max_syn_backlog=4096

HP-UX

ndd -set /dev/tcp tcp_ip_abort_cinterval 20000

Linux Tweaking

Raising netowrk limits for broadband under Linux
2003.04.01 16:03 by Philip
Keywords: TCP Window, WAN, TCP/IP, tweak, Linux

 

The TCP/IP parameters for tweaking a Linux-based machine for fast internet connections are located in /proc/sys/net/… (assuming 2.1+ kernel). This location is volatile, and changes are reset at reboot. There are a couple of methods for reapplying the changes at boot time, ilustrated below.

Locating the TCP/IP related parameters

All TCP/IP tunning parameters are located under /proc/sys/net/… For example, here is a list of the most important tunning parameters, along with short description of their meaning:

/proc/sys/net/core/rmem_max – Maximum TCP Receive Window
/proc/sys/net/core/wmem_max – Maximum TCP Send Window
/proc/sys/net/ipv4/tcp_rmem – memory reserved for TCP receive buffers
/proc/sys/net/ipv4/tcp_wmem – memory reserved for TCP send buffers
/proc/sys/net/ipv4/tcp_timestamps – timestamps (RFC 1323) add 12 bytes to the TCP header…
/proc/sys/net/ipv4/tcp_sack – tcp selective acknowledgements.
/proc/sys/net/ipv4/tcp_window_scaling – support for large TCP Windows (RFC 1323). Needs to be set to 1 if the Max TCP Window is over 65535.

Keep in mind everything under /proc is volatile, so any changes you make are lost after reboot.

There are some additional internal memory buffers for the TCP Window, allocated for each connection:
/proc/sys/net/ipv4/tcp_rmem – memory reserved for TCP rcv buffers (reserved memory per connection default)
/proc/sys/net/ipv4/tcp_wmem – memory reserved for TCP snd buffers (reserved memory per connection default)

The tcp_rmem and tcp_wmem contain arrays of three parameter values: the 3 numbers represent minimum, default and maximum memory values. Those 3 values are used to bound autotunning and balance memory usage while under global memory stress.

Applying TCP/IP Parameters at System Boot

You can edit /etc/rc.local, or /etc/boot.local depending on your distribution so the parameters get automatically reapplied at boot time. The TCP/IP parameters should be self-explanatory: we’re basically setting the TCP Window to 256960, disabling timestamps (to avoid 12 byte header overhead), enabling tcp window scaling, and selective acknowledgements:

echo 256960 > /proc/sys/net/core/rmem_default
echo 256960 > /proc/sys/net/core/rmem_max
echo 256960 > /proc/sys/net/core/wmem_default
echo 256960 > /proc/sys/net/core/wmem_max

echo 0 > /proc/sys/net/ipv4/tcp_timestamps 
echo 1 > /proc/sys/net/ipv4/tcp_sack 
echo 1 > /proc/sys/net/ipv4/tcp_window_scaling

Change the values above as desired, depending on your internet connection and maximum bandwidth/latency. There are other parameters you can change from the default if you’re confident in what you’re doing – just find the correct syntax of the values in /proc/sys/net/… and add a line in the above code analogous to the others. To revert to the default parameters, you can just comment or delete the above code from /etc/rc.local and restart.

Another method to reapply the values upon boot is to include the following in your /etc/sysctl.conf (adjust RWIN values as needed):

net.core.rmem_default = 256960
net.core.rmem_max = 256960 
net.core.wmem_default = 256960
net.core.wmem_max = 256960 

net.ipv4.tcp_timestamps = 0 
net.ipv4.tcp_sack = 1 
net.ipv4.tcp_window_scaling = 1

Notes:
Execute sysctl -p to make these new settings take effect.
To manually set the MTU value under Linux, use the command: ifconfig eth0 mtu 1500 (where 1500 is the desired MTU size)

Changing Current Values without rebooting

The current TCP/IP parameters can be edited without the need for reboot in the following locations:

/proc/sys/net/core/
rmem_default = Default Receive Window
rmem_max = Maximum Receive Window
wmem_default = Default Send Window
wmem_max = Maximum Send Window

/proc/sys/net/ipv4/
You’ll find timestamps, window scalling, selective acknowledgements, etc.

Keep in mind the values in /proc will be reset upon reboot. You still need to add the code in /etc/rc.local or /etc/boot.local in order to have the changes applied at boot time as described above.

Other TCP Parameters to consider

TCP_FIN_TIMEOUT
This setting determines the time that must elapse before TCP/IP can release a closed connection and reuse its resources. During this TIME_WAIT state, reopening the connection to the client costs less than establishing a new connection. By reducing the value of this entry, TCP/IP can release closed connections faster, making more resources available for new connections. Addjust this in the presense of many connections sitting in the TIME_WAIT state:

# echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
(default: 60 seconds, recommended 15-30 seconds)

Notes:
You can use any of the earlier described methods to reapply these settings at boot time.
Here is a quick way to view the number of connections and their states:

netstat -tan | grep ‘:80 ‘ | awk ‘{print $6}’ | sort | uniq -c

TCP_KEEPALIVE_INTERVAL
This determines the wait time between isAlive interval probes. To set:

echo 30 > /proc/sys/net/ipv4/tcp_keepalive_intvl
(default: 75 seconds, recommended: 15-30 seconds)

TCP_KEEPALIVE_PROBES
This determines the number of probes before timing out. To set:

echo 5 > /proc/sys/net/ipv4/tcp_keepalive_probes
(default: 9, recommended 5)

TCP_TW_RECYCLE
It enables fast recycling of TIME_WAIT sockets. The default value is 0 (disabled). The sysctl documentation incorrectly states the default as enabled. It can be changed to 1 (enabled) in many cases. Known to cause some issues with hoststated (load balancing and fail over) if enabled, should be used with caution.

echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
(boolean, default: 0)

TCP_TW_REUSE
This allows reusing sockets in TIME_WAIT state for new connections when it is safe from protocol viewpoint. Default value is 0 (disabled). It is generally a safer alternative to tcp_tw_recycle

echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse
(boolean, default: 0)

Note: The tcp_tw_reuse setting is particularly useful in environments where numerous short connections are open and left in TIME_WAIT state, such as web servers. Reusing the sockets can be very effective in reducing server load.

Kernel Recompile Option

There is another method one can use to set TCP/IP parameters, involving kernel recompile… If you’re brave enough. Look for the parameters in the following files:
/LINUX-SOURCE-DIR/include/linux/skbuff.h
Look for SK_WMEM_MAX & SK_RMEM_MAX
/LINUX-SOURCE-DIR/include/net/tcp.h
Look for MAX_WINDOW & MIN_WINDOW