Writings on optimizing Linux

Introduction

One of the things I am always aiming for is maximum performance of systems in every situation. Years ago our customer started with virtualization, and one of our bigger systems did not run well. This was with CentOS-6.X based systems. There was a strange periodic bump in system load causing lagging with data presented on the operational screens used for vessel traffic management. Giving a VM more memory or CPU power did never solve this. Even external Linux specialists told us this was as it is and should be dealt with. Well, not in my world :-).

I started investigating what happend, used every possible tool and google to find out what caused this. In the end the load was caused by IO congestion at kernel level that was specific for a range of the Linux kernels, and could be solved by using different kernel startup parameters. With this al lot of other performance enhancements came out that stil can be used today at home or in professional situations and this page is a summary of that.

The IO congestion problem was solved by adding an parameter to /etc/grub.conf in the system startup line. The parameter in our situation was: nohz=off

Other adjustments we did here for speed and reliability improvement for Centos-6 where:

elevator=noop vmw_pvscsi.cmd_per_lun=254 vmw_pvscsi.ring_pages=32 cgroup_disable=memory

General System Optimization

Preload

Something that always can be done on Ubuntu based systems is install preload that makes systems run and start applications faster.

  $ sudo apt install preload
  $ sudo systemctl enable preload ; sudo systemctl start preload

Swappiness

Another tweak that can be done is making the system less aggressive in the use of the swapfile. The default for swappiness is 60, that can be set to a lower value so memory is later purged to the slower swapfile on disk. This can be done by adding a setting to /etc/sysctl.conf, personally I always do this at the end of the file.

   vm.swappiness = 10

You can activate this setting with: sysctl -p

Filesystem Optimization

disabling relatime / atime

One of the things that can be done is is lowering disk related actions by adjusting the way disc acces is handled. Although these days disabeling relatime or has less benefits, adding noatime to your mount options can still give some benefits. This can be done by changing the mount options given in /etc/fstab. Be careful when making changes here, this does not need to be done for the /boot entry.
Normally these entries can look like:

  /dev/mapper/system-root /   ext4    defaults

This can be changed to:

  /dev/mapper/system-root /   ext4    defaults,noatime

In disk heavy situations some other tweaks could be added like:

  /dev/mapper/system-root /   ext4                   defaults,noatime,nobarrier,data=writeback,journal_ioprio=4,commit=60

A reboot is required to make these settings active.

You can test disk speed acces with hdparm:

  $ hdparm -tT /dev/hda of hdparm -tT /dev/sda

Note: Setting atime or relatime to notime has benefits for flash memory devices like SSD, there lifetime that will improve highly from it due to lesser writes to it.

Optimizing disk throughput

When you are on a fast SSD drive, disk acces can be optimized by changing some kernel parameters. edit /etc/sysctl.conf and add the following:

   vm.dirty_background_ratio = 5
   vm.dirty_ratio = 10

Enable thes parapeters with sysctl -p or reboot.

Netwerk Optimization

If network usage is heavy, and you are within a fast network environment there are some network optimizations that can be done in the file /etc/sysctl.conf

These parameters can be used in a fast network environment:

  net.ipv4.conf.all.arp_ignore = 1
  net.ipv4.conf.all.arp_filter = 1
  net.core.rmem_default = 1048576
  net.core.rmem_max = 2097152
  net.core.wmem_default = 1048576
  net.core.wmem_max = 2097152

In a smaller network these parameters could be used:

  net.ipv4.conf.all.arp_ignore = 1
  net.ipv4.conf.all.arp_filter = 1
  net.core.rmem_max=524288
  net.core.wmem_max=524288
  net.core.rmem_default=524288
  net.core.wmem_default=524288

You can activate these settings with: sysctl -p

sources: IBM and Oracle

Rights Optimization

Systems based on RHEL like CentOS, Oracle Linux or Rocky Linux are build thight for security reasons. This can give problems and slowdowns of processing data on heavy used systems or systems that require a lot of open files or concurrent processes. To give yourself almost al the working sace you need adjest the file /etc/security/limits.conf and ad these settings (* is all users except root):

  *          soft     nproc          65535

  *          hard     nproc          65535

  *          soft     nofile         65535

  *          hard     nofile         65535

You can check your current limits with: ulimit -a or prlimit

The above settings give all users and processes almost there maximum requirements. On more current systems the value 65535 can be set to 102400.

The total number of open files can be found by:

  # lsof | wc -l

The number of open files by a specific user can be found by:

  # lsof -u <username> | wc -l

Sources: Rocky Linux Forum / dannyda.com / RedHat

Final Remarks

The systems we use some of these adjustments on are Rocky linux systems running Graylog and Elasticsearch. When you make adjustments, do this carefully and test if they work for your specific situation.