Performance Tuning & Monitoring

Introduction to Performance Tuning

We generally use an OS in its native configuration—i.e., without bothering much about how the system will behave or respond in the current load situation. The overall load may vary from time to time, depending on the situation. So we need to continually monitor and tune our systems in the same manner like we change our car parts — depending upon the road and traffic conditions. Simply put, the objective here is to understand our Linux Operating System so we can work with it better.

Performance tuning requires that you should have a sound knowledge of computer hardware—how the various computer components talk to each other, in addition to the various components of the operating system. Collecting relevant data about a problem and analysing it is very important in successful performance tuning. Let’s start with some of the data gathering tools that can be used on Linux machines. The data gathered by these tools will then be used to analyse the problem, which will eventually lead to the solution.

What is a beater box?

Start with ‘Beater Box’ Never consider performance tuning as some black art, with which you can give certain commands and your machine will start behaving just as you wanted it to. And never directly use all these labs/tools on a production server. Instead spare yourself a ‘crash test dummy’, also referred to as ‘Beater Box’. Test several or all of the changes on that system to see the effects.

Collecting hardware configuration data

There are several tools that you can use to collect important information about your machine hardware. This is very important in efficient performance tuning. Let us go over some simple tools we can use to collect information.

vmstat

vmstat provides information about processes, memory, paging, block I/O, traps, and CPU activity. It displays either average data or actual samples. You can enable the sampling mode when you additionally provide a sampling frequency and a sampling duration to vmstat. The following command will display my machine’s virtual memory report after a delay of two seconds, for four times.

vmstat 2 4
performance tuning and monitoring

vmstat displays the following statistics:

1. Process (procs)
r – number of processes waiting for runtime
b – number of processes in uninterruptible sleep

2. Memory section swpd
swpd – amount of virtual memory used (KB)
free – amount of idle memory (KB)
buff – amount of memory used as buffers (KB)
cache – amount of memory used as cache (KB)

3. Swap section
si – amount of memory swapped from the disk (KB per second)
so – Amount of memory swapped to the disk (KB per second)

4. IO section
bi – blocks sent to a block device (blocks/s)
bo – blocks received from a block device (blocks/s)

5. System section
in – number of interrupts per second, including the clock
cs – number of context switches per second

6. CPU section
us – time spent running non-kernel code (user time, including nice time)
sy – time spent running kernel code (system time) id – time spent idle

dmidecode

The dmidecode command reads the system DMI table to display hardware and BIOS information of the system. This command will give you information on the current configuration of your system, as well as the system’s maximum supported configuration. For example, dmidecode gives both the current RAM on the system and the maximum RAM supported by the system. To get information about your motherboard, I can use the following command:

dmidecode -t baseboard
peformance tuning and monitoring

In the same manner, you can get any information about your system. Check the man pages if you are not sure about the options. Running the following will show you all the options that you can use:

performance tuning and monitoring

iostat

According to its man page, “The iostat command is used for monitoring the system input/ output device loading by observing the time the devices are active in relation to their average transfer rates. The reports then can be used to change the system configuration to better balance the input/output load between physical disks.” The first line generated by iostat is the average since the last boot, so it can be ignored. The simplest way you can use this command is by simply running the name of the command without any arguments. It will show you the boot report of all devices and the CPU, since the last boot.

iostat

As you can see the report also presents me the blocks-read per second (Blk_read/ s), the blocks-written per second (Blk-wrtn/s), the total blocks-read (Blk_read) and total blockswritten (Blk_wrtn) since the last boot. Blocks are expressed in kilobytes or megabytes of data read or written per second.

This per-second report can be very handy in taking a decision for an upgrade. I personally find this command very useful in figuring out which partition of my hard disk is under heavy I/O load.

More Knowledge

I will be uploading more articles on performance tuning so look out for that.
New to Linux? Checkout: RHCSA Training
Know your basics? Learn Linux Automation with the new RHCE.
If you are an expert with Linux, grow your career with AWS, Devops, Openstack or Openshift.