[arm-allstar] systemd very high CPU usage

[David McGough]

Hi Patrick,

Looking at the processes listed below, I don't see anything burning much
actual CPU time.  The Linux load average numbers can be confusing, since
those numbers indicate processes that are queued ready to run, not
necessarily the processes actually running at the given moment. Several
factors can artificially inflate the load average numbers, one example
being a process waiting on slow disk or network I/O (or some other kernel
resource, causing uninterruptable sleep)....On top of this confusion, it
has been reported in RPi forums that reported load averages on ARM/RPi
kernels are a bit wonky.

There are a several tools that can help identify the culprit causing these
load average numbers.  First, probably the atop utility has the most
comprehensive diagnostic output, it is truly a swiss army knife for
diagnosing bottlenecks, etc.  Also, there is the iotop command, which
displays a subset of the info available with atop.  Both the atop and
iotop packages are available for HamVoIP, for example:

pacman -S atop

As I mentioned above, processes stuck in "uninterruptable sleep" or
"Zombie" states, etc., can inflate the load numbers, even though they're
buring almost no CPU cycles.  A simple way to show these stuck processes
is with the ps command and awk filter, like:

ps axu | awk '$8 ~ /A|D|X|T|t|Z/ {print $0}'

The process state info for the ps command, as found in the manual, is:

man ps ---------------------------------------------------------------------

PROCESS STATE CODES
Here are the different values that the s, stat and state output specifiers 
(header "STAT" or "S") will display to describe the state of a process:

D    uninterruptible sleep (usually IO)
R    running or runnable (on run queue) 
S    interruptible sleep (waiting for an event to complete)
T    stopped by job control signal 
t    stopped by debugger during the tracing
W    paging (not valid since the 2.6.xx kernel) 
X    dead (should never be seen) 
Z    defunct ("zombie") process, terminated but not reaped by its parent

For BSD formats and when the stat keyword is used, additional characters 
may be displayed:

<    high-priority (not nice to other users)
N    low-priority (nice to other users)
L    has pages locked into memory (for real-time and custom IO) 
s    is a session leader 
l    is multi-threaded (using CLONE_THREAD, like NPTL pthreads do)
+    is in the foreground process group

----------------------------------------------------------------------------


I'm curious are to what's going on....

73, David K4FXC




On Thu, 17 Nov 2022, "Patrick Perdue via ARM-allstar" wrote:

> Greetings:

Others have posted about this before, but what is the possible reason 
for a node having such high CPU usage after a few days uptime? It seems 
like mostly processes related to systemd and crond are at fault. I can 
of course reboot the node, but I'd love to know what's actually going on 
here, as this happens with relative consistency. I do have a couple of 
tasks that periodically restart a service (Analog_Bridge) and other 
related things in my cron. This is on a Raspberry Pi 400. 5.4.75-1-ARCH.


[root at radioless ~]# top
top - 05:53:24 up 19 days, 11:40,  1 user,  load average: 10.22, 9.82, 9.67
Tasks: 154 total,   1 running,  90 sleeping,   0 stopped,   0 zombie
%Cpu0Â  :Â Â  0.0/2.8 3[||| ]
%Cpu1Â  :Â Â  0.0/5.0 5[||||| ]
%Cpu2Â  :Â Â  0.7/2.1 3[||| ]
%Cpu3Â  :Â Â  1.4/2.7 4[|||| ]
GiB Mem : 13.4/3.748 [ ]
GiB Swap:Â  0.0/0.000 [ ]

 Â  PID USERÂ Â Â Â Â  PRÂ  NIÂ Â Â  VIRTÂ Â Â  RESÂ  %CPU %MEMÂ Â Â Â  TIME+ S COMMAND
     1 root      20   0   29.7m   4.3m   0.0  0.1  14:13.01 S systemd
   173 root      20   0   27.8m  22.9m   0.0  0.6   3:56.86 S  `- 
systemd-journal
   201 root      20   0   12.0m   3.0m   0.0  0.1   0:02.60 S  `- 
systemd-udevd
   266 systemd+  20   0   12.3m   1.8m   0.0  0.0   0:03.06 S  `- 
systemd-timesyn
   270 root      20   0    3.1m   2.0m   0.0  0.1   0:02.80 S  `- crond
   272 root      20   0    3.1m   2.3m   0.0  0.1   0:34.17 S  `- 
systemd-logind
   274 dbus      20   0    5.5m   2.8m   0.0  0.1   1:35.67 S  `- 
dbus-daemon
   282 systemd+  20   0   11.4m   1.9m   0.0  0.0   0:03.50 S  `- 
systemd-network
   283 root      20   0    7.6m   4.3m   0.0  0.1  13:49.41 S  `- haveged
   284 root      20   0   57.8m  38.9m   2.0  1.0 477:51.04 S  `- 
MMDVM_Bridge
   292 root      20   0    1.8m   0.0m   0.0  0.0   0:00.00 S  `- md380-emu
   296 root      20   0    2.0m   1.3m   0.0  0.0   0:00.01 S  `- agetty
   298 root      20   0   29.3m  13.8m   0.0  0.4   1:00.79 S  `- httpd
12226 http      20   0   29.3m   6.1m   0.0  0.2   0:00.00 S `- httpd
12227 http      20   0   29.3m   6.1m   0.0  0.2   0:00.01 S `- httpd
12228 http      20   0   29.3m   6.1m   0.0  0.2   0:00.00 S `- httpd
12229 http      20   0   29.3m   6.1m   0.0  0.2   0:00.00 S `- httpd
12230 http      20   0   29.3m   6.1m   0.0  0.2   0:00.00 S `- httpd
  2640 http      20   0   29.3m   6.1m   0.0  0.2   0:00.00 S `- httpd
   299 root      20   0    6.6m   4.1m   0.0  0.1   0:00.07 S  `- sshd
[root at radioless ~]#

_______________________________________________

ARM-allstar mailing list
ARM-allstar at hamvoip.org
http://lists.hamvoip.org/cgi-bin/mailman/listinfo/arm-allstar

Visit the BBB and RPi2/3/4 web page - http://hamvoip.org