Operating systems and network tuning
Network tuning can reduce Transmission Control Protocol (TCP) stack delay by changing connection settings and can improve throughput by changing TCP buffers.
Operating systems
A Windows system needs the least tuning while a Solaris system needs the most tuning. The following information pertains to each system specified, and might improve WebSphere® eXtreme Scale performance. You should tune according to your network and application load.
Windows
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\
Tcpip\Parameters
MaxFreeTcbs = dword:00011940
MaxHashTableSize = dword:00010000
MaxUserPort = dword:0000fffe
TcpTimedWaitDelay = dword:0000001e
Solaris
ndd -set /dev/tcp tcp_time_wait_interval 60000
fndd -set /dev/tcp tcp_keepalive_interval 15000
ndd -set /dev/tcp tcp_fin_wait_2_flush_interval 67500
ndd -set /dev/tcp tcp_conn_req_max_q 16384
ndd -set /dev/tcp tcp_conn_req_max_q0 16384
ndd -set /dev/tcp tcp_xmit_hiwat 400000
ndd -set /dev/tcp tcp_recv_hiwat 400000
ndd -set /dev/tcp tcp_cwnd_max 2097152
ndd -set /dev/tcp tcp_ip_abort_interval 20000
ndd -set /dev/tcp tcp_rexmit_interval_initial 4000
ndd -set /dev/tcp tcp_rexmit_interval_max 10000
ndd -set /dev/tcp tcp_rexmit_interval_min 3000
ndd -set /dev/tcp tcp_max_buf 4194304
AIX®
/usr/sbin/no -o tcp_sendspace=65536
/usr/sbin/no -o tcp_recvspace=65536
/usr/sbin/no -o udp_sendspace=65536
/usr/sbin/no -o udp_recvspace=65536
/usr/sbin/no -o somaxconn=10000
/usr/sbin/no -o tcp_nodelayack=1
/usr/sbin/no –o tcp_keepinit=40
/usr/sbin/no –o tcp_keepintvl=10
LINUX
sysctl -w net.ipv4.tcp_timestamps=0
sysctl -w net.ipv4.tcp_tw_reuse=1
sysctl -w net.ipv4.tcp_tw_recycle=1
sysctl -w net.ipv4.tcp_fin_timeout=30
sysctl -w net.ipv4.tcp_keepalive_time=1800
sysctl -w net.ipv4.tcp_rmem="4096 87380 8388608"
sysctl -w net.ipv4.tcp_wmem="4096 87380 8388608"
sysctl -w net.ipv4.tcp_max_syn_backlog=4096
HP-UX
ndd -set /dev/tcp tcp_ip_abort_cinterval 20000
Linux Tweaking
Raising netowrk limits for broadband under Linux
2003.04.01 16:03 by Philip
Keywords: TCP Window, WAN, TCP/IP, tweak, Linux
The TCP/IP parameters for tweaking a Linux-based machine for fast internet connections are located in /proc/sys/net/… (assuming 2.1+ kernel). This location is volatile, and changes are reset at reboot. There are a couple of methods for reapplying the changes at boot time, ilustrated below.
Locating the TCP/IP related parameters
All TCP/IP tunning parameters are located under /proc/sys/net/… For example, here is a list of the most important tunning parameters, along with short description of their meaning:
/proc/sys/net/core/rmem_max – Maximum TCP Receive Window
/proc/sys/net/core/wmem_max – Maximum TCP Send Window
/proc/sys/net/ipv4/tcp_rmem – memory reserved for TCP receive buffers
/proc/sys/net/ipv4/tcp_wmem – memory reserved for TCP send buffers
/proc/sys/net/ipv4/tcp_timestamps – timestamps (RFC 1323) add 12 bytes to the TCP header…
/proc/sys/net/ipv4/tcp_sack – tcp selective acknowledgements.
/proc/sys/net/ipv4/tcp_window_scaling – support for large TCP Windows (RFC 1323). Needs to be set to 1 if the Max TCP Window is over 65535.
Keep in mind everything under /proc is volatile, so any changes you make are lost after reboot.
There are some additional internal memory buffers for the TCP Window, allocated for each connection:
/proc/sys/net/ipv4/tcp_rmem – memory reserved for TCP rcv buffers (reserved memory per connection default)
/proc/sys/net/ipv4/tcp_wmem – memory reserved for TCP snd buffers (reserved memory per connection default)
The tcp_rmem and tcp_wmem contain arrays of three parameter values: the 3 numbers represent minimum, default and maximum memory values. Those 3 values are used to bound autotunning and balance memory usage while under global memory stress.
Applying TCP/IP Parameters at System Boot
You can edit /etc/rc.local, or /etc/boot.local depending on your distribution so the parameters get automatically reapplied at boot time. The TCP/IP parameters should be self-explanatory: we’re basically setting the TCP Window to 256960, disabling timestamps (to avoid 12 byte header overhead), enabling tcp window scaling, and selective acknowledgements:
echo 256960 > /proc/sys/net/core/rmem_default
echo 256960 > /proc/sys/net/core/rmem_max
echo 256960 > /proc/sys/net/core/wmem_default
echo 256960 > /proc/sys/net/core/wmem_max
echo 0 > /proc/sys/net/ipv4/tcp_timestamps
echo 1 > /proc/sys/net/ipv4/tcp_sack
echo 1 > /proc/sys/net/ipv4/tcp_window_scaling
Change the values above as desired, depending on your internet connection and maximum bandwidth/latency. There are other parameters you can change from the default if you’re confident in what you’re doing – just find the correct syntax of the values in /proc/sys/net/… and add a line in the above code analogous to the others. To revert to the default parameters, you can just comment or delete the above code from /etc/rc.local and restart.
Another method to reapply the values upon boot is to include the following in your /etc/sysctl.conf (adjust RWIN values as needed):
net.core.rmem_default = 256960
net.core.rmem_max = 256960
net.core.wmem_default = 256960
net.core.wmem_max = 256960
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
Notes:
Execute sysctl -p to make these new settings take effect.
To manually set the MTU value under Linux, use the command: ifconfig eth0 mtu 1500 (where 1500 is the desired MTU size)
Changing Current Values without rebooting
The current TCP/IP parameters can be edited without the need for reboot in the following locations:
/proc/sys/net/core/
rmem_default = Default Receive Window
rmem_max = Maximum Receive Window
wmem_default = Default Send Window
wmem_max = Maximum Send Window
/proc/sys/net/ipv4/
You’ll find timestamps, window scalling, selective acknowledgements, etc.
Keep in mind the values in /proc will be reset upon reboot. You still need to add the code in /etc/rc.local or /etc/boot.local in order to have the changes applied at boot time as described above.
Other TCP Parameters to consider
TCP_FIN_TIMEOUT
This setting determines the time that must elapse before TCP/IP can release a closed connection and reuse its resources. During this TIME_WAIT state, reopening the connection to the client costs less than establishing a new connection. By reducing the value of this entry, TCP/IP can release closed connections faster, making more resources available for new connections. Addjust this in the presense of many connections sitting in the TIME_WAIT state:
# echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
(default: 60 seconds, recommended 15-30 seconds)
Notes:
You can use any of the earlier described methods to reapply these settings at boot time.
Here is a quick way to view the number of connections and their states:
netstat -tan | grep ‘:80 ‘ | awk ‘{print $6}’ | sort | uniq -c
TCP_KEEPALIVE_INTERVAL
This determines the wait time between isAlive interval probes. To set:
echo 30 > /proc/sys/net/ipv4/tcp_keepalive_intvl
(default: 75 seconds, recommended: 15-30 seconds)
TCP_KEEPALIVE_PROBES
This determines the number of probes before timing out. To set:
echo 5 > /proc/sys/net/ipv4/tcp_keepalive_probes
(default: 9, recommended 5)
TCP_TW_RECYCLE
It enables fast recycling of TIME_WAIT sockets. The default value is 0 (disabled). The sysctl documentation incorrectly states the default as enabled. It can be changed to 1 (enabled) in many cases. Known to cause some issues with hoststated (load balancing and fail over) if enabled, should be used with caution.
echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
(boolean, default: 0)
TCP_TW_REUSE
This allows reusing sockets in TIME_WAIT state for new connections when it is safe from protocol viewpoint. Default value is 0 (disabled). It is generally a safer alternative to tcp_tw_recycle
echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse
(boolean, default: 0)
Note: The tcp_tw_reuse setting is particularly useful in environments where numerous short connections are open and left in TIME_WAIT state, such as web servers. Reusing the sockets can be very effective in reducing server load.
Kernel Recompile Option
There is another method one can use to set TCP/IP parameters, involving kernel recompile… If you’re brave enough. Look for the parameters in the following files:
/LINUX-SOURCE-DIR/include/linux/skbuff.h
Look for SK_WMEM_MAX & SK_RMEM_MAX
/LINUX-SOURCE-DIR/include/net/tcp.h
Look for MAX_WINDOW & MIN_WINDOW
Recent Comments