Linux one-liner of the day: Normalize the load average

During a Linux class, the following question came up:

“How to normalize the load average for easier monitoring”

A student, utilizing check_mk for system monitoring, wanted to apply what he had just learned:

To properly evaluate the load average of a Linux system, one must consider the number of available CPUs.

Or simply spoken: you need to divide the reported load average by the number of available CPUs, to determine if your system is currently overloaded or not.

I have discussed this for instance here: The Linux Load Average - and what these numbers tell you

The student in question monitored a bunch of Linux servers with various CPU configurations and wanted to apply the same logic and alarm thresholds for each system.

The solution to this sounds straight forward:

• Step #1: Obtain the load average
• Step #2: Determine the number of CPUs
• Step #3: Divide the first number by the second

Looks like an ideal opportunity for a nerdy-looking one-liner. So let’s go …

Step #1: Get the load average of a Linux system

As you may know, the load average can be found in various locations within a Linux system.

It appears for instance in the first output line of top:

… or in the output of the command w:

With a little bit of datastream tweaking, we would be able to isolate the load average of the last minute (the first of the three numbers):

• Take the output of w
• Use head -n1 to obtain only the first line
• Employ tr -s ' ' to squeeze repeated spaces
• Use cut to extract the 11th field by dviding the line by spaces
• Use tr -d ',' to remove the trailing comma

Although it is perfectly doable this way, I would like to take a different source for the numbers - a source that’s easier processable: The file “/proc/loadavg”.

This file contains all three load average values as the first three fields:

[robert@demo ~]$ cat /proc/loadavg 
0.98 0.59 0.31 2/158 12479

(The remaining two values describe the number of running and total processes and the PID of the most recently created process)

So if we take “/proc/loadavg” as datasource, we could simplify the command line from above a lot:

cut -d " " -f 1 /proc/loadavg
0.98

Step #1 … done.

Step #2: Determine the number of CPUs in a Linux system

Now we need to know the number of CPUs - how can we get this information on a Linux-system?

Well - there are again different sources available.

First - let’s have a look into the “/proc” directory. The file “/proc/cpuinfo” gives us for every CPU core the system sees a huge block of information.

robert@demo:~$ cat /proc/cpuinfo 
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 23
...
... a load of more information ...
...
address sizes   : 48 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : AuthenticAMD
cpu family      : 23
...
... a load of more information ...
...
address sizes   : 48 bits physical, 48 bits virtual
power management:

If we count these information blocks, we have the number of CPUs.

Nothing easier than that …

I chose to simply search for the lines “processor : <ID>” with grep …

robert@demo:~$ grep "^processor" /proc/cpuinfo
processor       : 0
processor       : 1

and then I count these lines with wc -l

robert@demo:~$ grep "^processor" /proc/cpuinfo | wc -l
2

Perfect - this last command line gives us straight the number of CPUs.

I would do it exactly this way until the end of my life, weren’t there a student who came up with a tiny tool I haven’t heard about yet: nproc.

With nproc, you get exactly one single number: the number of available CPUs.

robert@demo:~$ nproc
2

I think this is - for our planned one-liner - much better than the combination of grep, tr and wc from above. So let’s stick with it.

Step #3: Divide the first number by the second one

For calculating, we could use a feature of every POSIX-compliant shell called “arithmetic expansion”:

If you somewhere on the command line use a dollar sign followed by an arithmetic expression in double-parentheses (as in $(( 3 + 5))), then the shell replaces this pattern with the result of the expression:

robert@demo:~$ echo "The sum of 3 and 5 is: $((3+5))"
The sum of 3 and 5 is: 8

Easy - he?

And instead of calculating a sum, we would simply divide the two numbers via “/”.

And within the “arithmetic expansion” we will use a feature called “command expansions” instead of hardcoded numbers.

The “command expansion” is again available in every POSIX-compliant shell:

If you somewhere on a command line use a dollar-sign followed by a command in single parentheses (as in $(nproc)), then the shell first executes the command and replaces the pattern with the output the command prints out via stdout.

Let’s first try this out and print the numbers along with some text:

robert@demo:~$ echo "Number of CPUs: $(nproc)"
Number of CPUs: 2
robert@demo:~$ echo "1min load average: $(cut -d " " -f 1 /proc/loadavg)"
1min load average: 0.87

Great!

Now lets combine these two expressions to get our result:

robert@demo:~$ echo "normalized load: $(( $(cut -d ' ' -f 1 /proc/loadavg) / $(nproc) )) "
-bash: 0.87 / 2 : syntax error: invalid arithmetic operator (error token is ".87 / 2 ")

Duh … that’s ugly.

What went wrong?

Well - it’s simply that the shell only supports integer numbers in the “arithmetic expansion”.

And if we try to calculate “0.87 / 2” , we simply get this error:

robert@demo:~$ echo "normalized load: $(( 0.87 / 2 )) "
-bash: 0.87 / 2 : syntax error: invalid arithmetic operator (error token is ".87 / 2 ")

Can we solve this? Of course - in two different ways.

First, if we are ok with just using the integer values of the load average without the dezimal places - we could remove the decimal places from this number with a second use of the cut command:

robert@demo:~$ cut -d " " -f 1 /proc/loadavg | cut -d "." -f 1
0

But for me, this would remove too much detail from the load information.

I would better leverage a tool for the calculation, that can handle decimal numbers too: the tool bc.

This tool reads the expression to calculate from its input-datastream “stdin”. So you need to “pipe” the expression via echo to bc:

robert@demo:~$ echo 3 + 5  |  bc
8

And it can handle decimal numbers:

robert@demo:~$ echo 0.87 / 2 | bc
0

The result here is shown as “0”, because bc calculates with decimal numbers but it doesn’t print out the decimal numbers after the point.

If we want to see them (and I want), we need to set the variable “scale” just before the expression to the number of decimal places we want to see after the decimal point:

robert@demo:~$ echo "scale=2;0.87 / 2" | bc
.43

yes!

And now let’s combine this with our command lines for the numbers from step #1 and step #2:

robert@demo:~$ echo "scale=2;$(cut -d " " -f 1 /proc/loadavg) / $(nproc)" | bc
.41

Mission accomplished!

Did I mention, that I love nerdy looking command lines? 😉

Final thoughts on hyperthreading

If you followed me through this post and thought

“Hey Robert, what about the load in combination with hyperthreading?”

…then yes, you are right:

If we want to account for hyperthreading, we need to halve the number of cores the system sees. But first we would need to detect if hyperthreading is active or not.

To include this in the one-liner or into a short script is exactly your homework for today. 🙂

If you like, share your solution!

Have fun!

Wanna take an unfair advantage?

If it comes to working on the Linux command line - at the end of the day it is always about knowing the right tool for the right task.

And it is about knowing the tools that are most certainly available on the Linux system you are currently on.

To give you all the tools for your day-to-day work at the Linux command line, I have created “The ShellToolbox”.

This book gives you everything

• from the very basic commands, through
• everything you need for working with files and filesystems,
• managing processes,
• managing users and permissions, through
• software management,
• hardware analyses and
• simple shell-scripting to the tools you need for
• doing simple “networking stuff”.

Everything in one single, easy to read book. With explanations and example calls for illustration.

If you are interested, go to shelltoolbox.com and have a look (as long as it is available).