Releasing cached memory in Linux && Set swappiness

Releasing cached memory in Linux

Under normal circumstances, modern Linux systems try to cache into memory disk data that is accessed often. Sometimes, we have that much memory in the system that our kernel keeps filling up the memory by caching every piece of data we access.

Other times, because of the swappiness factor, active data finds its way into the swap instead of the main memory. I have seen this behavior in a few systems hosting databases, specially running mysql, and it is a serious performance hazard.

In order to fix systems like this, we need to fix the swappiness, drop the caches and, swapoff and swapon the system swap.
Decreasing swappiness
Thus, the first thing to do is to decrease the overall swappiness. This tells the system how often to store data into the swap instead of the main memory.

By default, most common Linux systems have this value set to 60. However, this value can be modified easily, not only during boot time, but also in hot.

The higher the value of the swappiness, the more often the system will try to store data in the swap. On the other hand, the lower the value of the swappiness, the more often the system will try to store data into main memory.

The swappiness value falls into the range of 0 (try not to swap at all) to 100 (try to swap as much as possible).

To modify the swappiness in hot, simply echo the value you desire to /proc/sys/vm/swappiness. For example:

1
# echo “10” > /proc/sys/vm/swappiness
To have this done at boot time, modify /etc/sysctl.conf and set a value for the entry (create it if it does not exist yet) vm.swappiness. For example:

Code:

1
# Set swappiness
2
vm.swappiness=10
Dropping the caches
Once this is done, the next thing we can do is to clean up the caches of the memory. Unfortunately, this feature is only present since kernels 2.6.16. Most people should not have problem with this, but a few may run out of luck. In any case, let me remark that this operation is safe under most circumstances.

Dropping the caches is as simple as echoing a value between 1 (one) and 3 (three) to /proc/sys/vm/drop_caches. Once this operation is done, you do not need to echo a 0 (zero) back to /proc/sys/vm/drop_caches, the system will start caching again from scratch automatically. This means that you cannot actually tell the system to stop caching using this method.

The values that can be echoed do the following:

1 : Drops pagecaches
2 : Drops dentries and inodes
3 : Drops pagecaches, dentries and inodes
Example:

1
# echo “1” > /proc/sys/vm/drop_caches
Swapping off and on
Now comes the slow part of swapping off. You should normally know where your system is swapping, but just in case you are new to this, run the following command:

1
# grep swap /etc/fstab
Running this command will produce output similar to the following:

/dev/VolGroup00/LogVol01 swap swap defaults 0 0
Now that you have identified your partition, verify that its contents will fit into main memory by running the following command:

1
# free -m
This will produce output similar to the following:

total used free shared buffers cached
Mem: 8112 7823 288 0 12 6016
-/+ buffers/cache: 1794 6317
Swap: 1983 0 1983
Have a look at the value located in the swap row and the used column. If this value is higher than the value located in the mem row and the free column, it means that your swapped data won’t fit into your free main memory and, thus, the system will not allow you to.

Once you know that your swapped data fits into your free main memory, you can swapoff and swapon back your swap partition. Be very careful as this process can be very slow and cannot be normally stopped once it is running. You have been warned.

In my case, I would run something like the following:

1
# swapoff /dev/VolGroup00/LogVol01 && swapon /dev/VolGroup00/LogVol01
You may want to use something like nohup if you are connected to a remote machine and at the risk of losing the session. You can alternatively run the command into a screen as well.
How To Configure Swappiness
The operating system kernel can adjust how often it relies on swap through a configuration parameter known as swappiness.

To find the current swappiness settings, type:

cat /proc/sys/vm/swappiness
60
Swapiness can be a value from 0 to 100. Swappiness near 100 means that the operating system will swap often and usually, too soon. Although swap provides extra resources, RAM is much faster than swap space. Any time something is moved from RAM to swap, it slows down.

A swappiness value of 0 means that the operating will only rely on swap when it absolutely needs to. We can adjust the swappiness with the sysctl command:

sysctl vm.swappiness=10
vm.swappiness=10
If we check the system swappiness again, we can confirm that the setting was applied:

cat /proc/sys/vm/swappiness
10
To make your VPS automatically apply this setting every time it boots up, you can add the setting to the /etc/sysctl.conf file:

sudo nano /etc/sysctl.conf
# Search for the vm.swappiness setting. Uncomment and change it as necessary.
vm.swappiness=10

Try to add vm.swappiness = 0 to /etc/sysctl.conf and run sysctl -p then, this will instruct kernel to release memory occupied by the file cache more aggressively if a user application requires a memory region from kernel.
The swappiness parameter controls the tendency of the kernel to move processes out of physical memory and onto the swap disk. Because disks are much slower than RAM, this can lead to slower response times for system and applications if processes are too aggressively moved out of memory.

swappiness can have a value of between 0 and 100

swappiness=0 tells the kernel to avoid swapping processes out of physical memory for as long as possible
swappiness=100 tells the kernel to aggressively swap processes out of physical memory and move them to swap cache

The default setting in Linux is swappiness=60. Reducing the default value of swappiness will probably improve overall performance for a typical desktop installation. A value of swappiness=10 is recommended, but feel free to experiment.

To check the swappiness value use command: cat /proc/sys/vm/swappiness

To make a change permanent, edit the configuration file with your favorite editor:

vi /etc/sysctl.conf
and add following parameter to the end of the file like so:
vm.swappiness=10
Save the file and reboot.

Still, there are users who insist that, for example, a system backup should never force OpenOffice out to disk. They don’t care how quickly a system maintenance application runs at 3:00 in the morning,
but they care a lot about how the system responds when they are at the keyboard.

Swappiness should to be turned off

This one is pretty straightforward if you know about the Linux virtual memory system. Swappiness levels tell the virtual memory subsystem how much it should try and swap to disk. The thing is, the system will try to swap out items in memory even when there is plenty of RAM available to the system. The OS default is usually 60, which is a little aggressive IMO. You can see what value your system is set to by running the following command:

Since Couchbase is tuned to really operate in memory as much as possible. You can gain or at minimum not lose performance by just changing the swappiness value to 0. In non-tech talk, this tells the virtual memory subsystem of the OS to not swap items from RAM to disk unless it really really has to, which if you have sized your nodes correctly, swapping should not be needed. To set this, perform the following process use sudo or just become root if you ride in the wild west.

Make sure that you either have or modify your process that builds your OSs to do this. This is especially critical for public/private clouds where it is so easy to bring up new instances. You need to make this part of your build process for a Couchbase node.

Disable Transparent Huge Pages (THP)

Starting in Red Hat Enterprise Linux (RHEL) version 6, so this includes CentOS 6 and 7 too, a new default method of managing huge pages was implemented in the OS. It combines smaller memory pages into Huge Pages without the running processes knowing. The idea is to reduce the number of lookups on TLB required and therefor increase performance. It brings in abstraction for automatation and management of huge pages basically.  Couchbase Engineering has determined that under some conditions, Couchbase Server can be negatively impacted by severe page allocation delays when THP is enabled. Couchbase therefore recommends that THP be disabled on all Couchbase Server nodes

To disable this on a running system temporarily, run the following commands:

To disable this permanently, do the following:

Then copy the following into the bottom of /etc/rc.local.

If you do not want to reboot just yet, just use the echo commands above and know that the next time you reboot this will also be in place. Some people like to reboot if there is some change to such a file. Some do not.

Note: There is a different way to do this that you will find elsewhere and edits /etc/grub.conf. My problem with it is that it would get blown out with each and every kernel update in the future. What I propose is easier to manage in the long run and easy to put into something like Puppet module or Chef recipe to append to the end of rc.local when you boot strap a node.

THP is a great feature for some things, but causes problems with applications like Couchbase. It is not alone in this. If you go search the Internet for transparent huge pages, there are multiple documented issues from other DB and application vendors about this. Until something has been found to work with this, it is just best to turn THP off.