Performance Tuning in centos7

Tuned

In RedHat (and thus CentOS) 7.0, a daemon called “tuned” was introduced as a unified system for applying tunings to Linux. tuned operates with simple, file-based tuning “profiles” and provides an admin command-line interface named “tuned-adm” for applying, listing and even recommending tuned profiles.

Some operational benefits of tuned:

• File-based configuration – Profile tunings are contained in a simple, consolidated files
• Swappable profiles – Profiles are easily changed back/forth
• Standards compliance – Using tuned profiles ensures tunings are not overridden or ignored

Note: If you use configuration management systems like Puppet, Chef, Salt, Ansible, etc., I suggest you configure those systems to deploy tunings via tuned profiles instead of applying tunings directly, as tuned will likely start to fight this automation, overriding the changes.

The default available tuned profiles (as of  RedHat 7.2.1511) are:

• balanced
• desktop
• latency-performance
• network-latency
• network-throughput
• powersave
• throughput-performance
• virtual-guest
• virtual-host

The profiles that are generally interesting for database usage are:

• latency-performance
  

  “A server profile for typical latency performance tuning. This profile disables dynamic tuning mechanisms and transparent hugepages. It uses the performance governer for p-states through cpuspeed, and sets the I/O scheduler to deadline.”

• throughput-performance
  

  “A server profile for typical throughput performance tuning. It disables tuned and ktune power saving mechanisms, enables sysctl settings that improve the throughput performance of your disk and network I/O, and switches to the deadline scheduler. CPU governor is set to performance.”

• network-latency – Includes “latency-performance,” disables transparent_hugepages, disables NUMA balancing and enables some latency-based network tunings.
• network-throughput – Includes “throughput-performance” and increases network stack buffer sizes.

I find “network-latency” is the closest match to our recommended tunings, but some additional changes are still required.

T

Tuning a server according to specific requirements is not an easy task. You need to know a lot of system parameters and how to change them in a intelligent manner.
Red Hat offers a tool called tuned-adm that makes these changes easy by using tuning profiles.

The tuned-adm command requires the tuned package (if not already installed):

# yum install -y tuned

Tuning Profiles

A tuning profile consists in a list of system changes corresponding to a specific requirement.
To get the list of the available tuning profiles, type:

# tuned-adm list
Available profiles:
- balanced
- desktop
- latency-performance
- network-latency
- network-throughput
- powersave
- sap
- throughput-performance
- virtual-guest
- virtual-host
Current active profile: virtual-guest

Note: All these tuning profiles are explained in details in the tuned-adm man page.

To only get the active profile, type:

# tuned-adm active
Current active profile: virtual-guest

To get the recommended tuning profile in your current configuration, type:

# tuned-adm recommend
virtual-guest

To apply a different tuning profile (here throughput-performance), type:

# tuned-adm profile throughput-performance

cpu setting

tuned-adm profile throughput-performance
tuned-adm active
cpupower idle-set -d 4
cpupower idle-set -d 3
cpupower idle-set -d 2
cpupower frequency-set -g performance
# for more info /usr/lib/tuned/throughput-performance/tuned.conf

sysctl

kernel.numa_balancing=0

net.core.netdev_max_backlog = 300000
net.ipv4.tcp_sack = 0
net.core.netdev_budget=600
net.ipv4.tcp_timestamps=1
net.ipv4.tcp_low_latency=1
net.ipv4.tcp_rmem=16384 349520 16777216
net.ipv4.tcp_wmem=16384 349520 16777216
net.ipv4.tcp_mem = 2314209      3085613 4628418
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.core.somaxconn=2048
net.ipv4.tcp_adv_win_scale=1
net.ipv4.tcp_window_scaling = 1
#UDP buffer
net.core.rmem_max=16777216

Linux Kernel Tuning for Centos 7

tuned` should already be installed for Cent 7 and default profile is balanced.

tuned-adm profiles can be found in this directory

ls /usr/lib/tuned/
 
balanced/               latency-performance/    powersave/              virtual-guest/
desktop/                network-latency/        recommend.conf          virtual-host/
functions               network-throughput/     throughput-performance/

To see what the active profile is:

tuned-adm active

To activated tuned xxx

tuned-adm profile xxx

latency-performance

• latency-performance
  • Profile for low latency performance tuning.
  • Disables power saving mechanisms.
  • CPU governor is set to performance and locked to the low C states (by PM QoS).
  • CPU energy performance bias to performance.
  • This profile is the Parent profile to “network-latency”.

Activate tuned latency-performance for CentOS 7

tuned-adm profile latency-performance

For CentOS 7, the latency-performance profile includes the following tweaks

cat /usr/lib/tuned/latency-performance/tuned.conf

[cpu]
force_latency=1
governor=performance
energy_perf_bias=performance
min_perf_pct=100
 
[sysctl]
kernel.sched_min_granularity_ns=10000000
vm.dirty_ratio=10
vm.dirty_background_ratio=3
vm.swappiness=10
kernel.sched_migration_cost_ns=5000000

network-latency

• network-latency
  • This is a Child profile of “latency-performance”.
  • That this means is that if you were to activate network-latency profile via tuned, it would automatically enable latency-performance, then make some additional tweaks to improve network latency.
  • Disables transparent hugepages, and makes some net.core kernel tweaks.

cat /usr/lib/tuned/network-latency/tuned.conf

[main]
include=latency-performance
 
[vm]
transparent_hugepages=never
 
[sysctl]
net.core.busy_read=50
net.core.busy_poll=50
net.ipv4.tcp_fastopen=3
kernel.numa_balancing=0

throughput-performance

• throughput-performance
  • This is the Parent profile to virtual-guest, virtual-host and network-throughput.
  • This profile is optimized for large, streaming files or any high throughput workloads.

cat /usr/lib/tuned/throughput-performance/tuned.conf

[cpu]
governor=performance
energy_perf_bias=performance
min_perf_pct=100
 
[vm]
transparent_hugepages=always
 
[disk]
readahead=>4096
 
[sysctl]
kernel.sched_min_granularity_ns = 10000000
kernel.sched_wakeup_granularity_ns = 15000000
vm.dirty_ratio = 40
vm.dirty_background_ratio = 10
vm.swappiness=10

virtual-guest

• virtual-guest
  • Profile optimized for virtual guests based on throughput-performance profile.
  • It additionally decreases virtual memory swapiness and increases dirty_ratio settings.

cat /usr/lib/tuned/virtual-guest/tuned.conf

[main]
include=throughput-performance
 
[sysctl]
vm.dirty_ratio = 30
vm.swappiness = 30

virtual-host

• virtual-host
  • Profile optimized for virtual hosts based on throughput-performance profile.
  • It additionally enables more aggressive write-back of dirty pages.

cat /usr/lib/tuned/virtual-host/tuned.conf

[main]
include=throughput-performance
 
[sysctl]
vm.dirty_background_ratio = 5
kernel.sched_migration_cost_ns = 5000000

I/O scheduler

echo 'deadline' > /sys/block/sda/queue/scheduler

vim /etc/grub2.cfg

menuentry ‘CAKE 3.0, with Linux 3.10.0-229.1.2.el7.x86_64′
set root=’hd0,msdos1’
linux16 /vmlinuz-3.10.0-229.1.2.el7.x86_64 root= …. elevator=deadline
initrd16 /initramfs-3.10.0-229.1.2.el7.x86_64.img