DevOps for Server Health

Here at CKH Consulting we host many websites that we have to keep our keen eye on our server resources and health. If you are the same, I have some great linux commands to show you that will help keep your servers healthy. The first thing I do when a server is getting a big sluggish is look at the RAM resources.

free -h

When the free RAM is getting low, that’s when you have to look into adding more or dialing back your website resources.

        total   used   free   shared   buff/cache   available
Mem:    3.8G    772M   180M   120M     2.9G         2.7G
Swap:   255M    178M   77M

I’ve also used that command to show your RAM resources each second over a certain amount of time.

free -h -c 20

This command will run the same output as ‘free’ but will return each second for how long you want it to. I have used it for when my PHP process has gotten too many hits and is expanding too much!

Speaking of PHP, I’ve also used this command to see the status of the process currently on the server:

service php7.4-fpm status

This will give you a good overview of how the process is currently running.

● php7.4-fpm.service - The PHP 7.4 FastCGI Process Manager
Loaded: loaded (/lib/systemd/system/php7.4-fpm.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2022-08-23 13:52:10 CDT; 2 weeks 0 days ago
Docs: man:php-fpm7.4(8)
Process: 22027 ExecStopPost=/usr/lib/php/php-fpm-socket-helper remove /run/php/php-fpm.sock /etc/php/7.4/fpm/pool.d/www.conf 74 (code=exited, status=0/SUCCESS)
Process: 22114 ExecStartPost=/usr/lib/php/php-fpm-socket-helper install /run/php/php-fpm.sock /etc/php/7.4/fpm/pool.d/www.conf 74 (code=exited, status=0/SUCCESS)
Main PID: 22110 (php-fpm7.4)
Status: "Processes active: 0, idle: 3, Requests: 363697, slow: 0, Traffic: 0.2req/sec"
Tasks: 4 (limit: 4681)
CGroup: /system.slice/php7.4-fpm.service
├─12676 php-fpm: pool www
├─14074 php-fpm: pool www
├─15600 php-fpm: pool www
└─22110 php-fpm: master process (/etc/php/7.4/fpm/php-fpm.conf)

The CGroup shows how many processes are running, and that really has helped me in situations where processes are out of control. When our customers sites get more and more traffic, this helps me to reign in the process. Remember, this command will work for any other service the server is running. I use it a lot for nginx as well.

service nginx status

● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2022-08-23 13:54:22 CDT; 2 weeks 0 days ago
Docs: man:nginx(8)
Process: 22361 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCCESS)
Process: 8122 ExecReload=/usr/sbin/nginx -g daemon on; master_process on; -s reload (code=exited, status=0/SUCCESS)
Process: 22411 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 22410 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 22412 (nginx)
Tasks: 3 (limit: 4681)
CGroup: /system.slice/nginx.service
├─22412 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
├─22413 nginx: worker process
└─22414 nginx: worker process

Along with seeing the status of one process, you have to know how to see all the processes running! Oh, I guess I’ll tell you that now:

top -b -o +%MEM | head -n 35

The output of this is really the best way of seeing the overall processes that your server is running and the resources each is taking up. This also shows memory usage, but I showed you a more clear way to see that. So use which one you like better 😉

top - 12:41:30 up 175 days, 22:12, 1 user, load average: 0.41, 0.81, 0.73
Tasks: 189 total, 2 running, 94 sleeping, 0 stopped, 0 zombie
%Cpu(s): 7.0 us, 3.4 sy, 0.0 ni, 88.8 id, 0.1 wa, 0.2 hi, 0.5 si, 0.1 st
KiB Mem : 4002552 total, 144124 free, 733632 used, 3124796 buff/cache
KiB Swap: 262140 total, 66416 free, 195724 used. 2854992 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16666 user 20 0 701300 198384 63532 S 18.8 5.0 0:20.04 php-fpm7.4
16691 user 20 0 652480 150432 62884 S 0.0 3.8 0:17.44 php-fpm7.4
16732 user 20 0 603988 102232 62660 R 87.5 2.6 0:14.21 php-fpm7.4
29928 user 20 0 594716 79136 16220 S 0.0 2.0 0:51.24 php-fpm7.2
16417 user 20 0 592812 78024 16240 S 0.0 1.9 1:19.71 php-fpm7.2
29934 user 20 0 592516 77564 16188 S 0.0 1.9 0:51.99 php-fpm7.2
352 root 19 -1 383628 58264 48220 S 0.0 1.5 35:40.89 systemd-journal
31766 netdata 20 0 195716 35460 3008 S 0.0 0.9 501:09.60 netdata
31786 netdata 20 0 135004 32152 4536 S 0.0 0.8 215:41.89 python3
22110 root 20 0 490892 11740 10064 S 0.0 0.3 1:10.54 php-fpm7.4
17443 schaumb+ 20 0 41780 3608 3028 R 6.2 0.1 0:00.01 top
15187 netdata 20 0 18640 3132 2716 S 0.0 0.1 0:01.63 bash
14028 schaumb+ 20 0 103868 2804 2528 S 0.0 0.1 0:00.01 sshd

Those are my go-to commands for my work as a devOps engineer when we need eyes on our server health. What do you think? Will you be adding any of these to your snippet library?