Page 1 of 1
[SOLVED] Agent - RAM usage under Linux
Published: June 12, 2024 - 10:06 AM
by Klug
Hello everyone,
I have a client who uses WAPT to manage their workstations and servers (VMs).
They have Windows (workstations and servers), Ubuntu (18.04 to 22.04), Debian (12), and CentOS (7).
We just realized that, on some Linux VMs, the WAPT client (waptagent 2.4.0) is using several GB of RAM.
I just randomly selected two VMs: one is using 1.3 GB of RAM, and the other 2.5 GB.
Does this seem "normal," or is there a problem with the deployment (or the client version, or something else)?
Re: Agent - RAM usage under Linux
Published: June 12, 2024 - 3:57 PM
by dcardon
Hi David,
regarding the figures you provided, are they RSS (resident memory size) or VSZ (virtual memory size)?
RSS should normally stay around 150-200MB; otherwise, there might be something to look into. Calculating VSZ virtual memory is a bit more complex because it includes many things, such as .so libraries that are mapped with mmap() but don't necessarily occupy RAM.
Regards,
Denis
Re: Agent - RAM usage under Linux
Published: June 12, 2024 - 4:47 PM
by Klug
Good morning,
Here is a little more detail about what I saw.
For the VM with 2 GB and a bit more:
Code: Select all
# pmap -x 1152862
1152862: waptservice
Address Kbytes RSS Dirty Mode Mapping
.../... obligé de couper pour pouvoir poster
---------------- ------- ------- -------
total kB 2498900 1188768 1178376
And for the other one:
Code: Select all
# pmap -x 1325965
1325965: waptservice
Address Kbytes RSS Dirty Mode Mapping
.../... obligé de couper pour pouvoir poster
---------------- ------- ------- -------
total kB 1398308 269136 248088
Re: Agent - RAM usage under Linux
Published: June 12, 2024 - 4:49 PM
by Klug
A different perspective (top) for the first one (3 GB):
Code: Select all
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1152862 root 20 0 3082032 1.4g 9048 S 25.0 24.9 253:43.04 waptservice
2116295 root 20 0 17936 13720 6408 R 5.7 0.2 0:00.17 python
Re: Agent - RAM usage under Linux
Published: June 12, 2024 - 4:52 PM
by Klug
And in another way, which gives 3 GB in VmSize:
Code: Select all
# cat /proc/1152862/status
Name: waptservice
Umask: 0022
State: S (sleeping)
Tgid: 1152862
Ngid: 0
Pid: 1152862
PPid: 1
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
FDSize: 128
Groups:
NStgid: 1152862
NSpid: 1152862
NSpgid: 1152862
NSsid: 1152862
VmPeak: 3139120 kB
VmSize: 3082032 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 1578124 kB
VmRSS: 1513280 kB
RssAnon: 1504232 kB
RssFile: 9048 kB
RssShmem: 0 kB
VmData: 2086008 kB
VmStk: 132 kB
VmExe: 4 kB
VmLib: 18396 kB
VmPTE: 4072 kB
VmSwap: 430308 kB
HugetlbPages: 0 kB
CoreDumping: 0
THP_enabled: 1
Threads: 16
SigQ: 1/23523
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000001001000
SigCgt: 0000000100000002
CapInh: 0000000000000000
CapPrm: 000001ffffffffff
CapEff: 000001ffffffffff
CapBnd: 000001ffffffffff
CapAmb: 0000000000000000
NoNewPrivs: 0
Seccomp: 0
Seccomp_filters: 0
Speculation_Store_Bypass: thread vulnerable
SpeculationIndirectBranch: conditional enabled
Cpus_allowed: 3
Cpus_allowed_list: 0-1
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 182496
nonvoluntary_ctxt_switches: 276
Re: Agent - RAM usage under Linux
Published: June 12, 2024 - 6:08 PM
by dcardon
Hi again David,
what's the exact WAPT version? Edition? (see forum rules)
The 1.4GB RSS feed size is indeed very unusual. Are there any unusual entries in the logs? If you restart the agent, what memory level does it return to, and how quickly does it increase? How long does it take for the agents to reach 1.4GB?
Regards,
Denis
Re: Agent - RAM usage under Linux
Published: June 13, 2024 - 09:53
by Klug
Good morning.
On the VMs in question, I have this: 2.4.0.14143-9847ee8b-debian-12-amd64
I don't know about the server, I'll ask the question.
On one of the VMs I was talking about yesterday (the one with the 3 GB process), RAM was added yesterday (so the VM was restarted).
Code: Select all
# ps -p 765 -o etime
ELAPSED
16:00:46
We're already at 1.6 GB on VmRSS, in 16 hours...
Code: Select all
# cat /proc/765/status
Name: waptservice
Umask: 0022
State: S (sleeping)
Tgid: 765
Ngid: 0
Pid: 765
PPid: 1
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
FDSize: 128
Groups:
NStgid: 765
NSpid: 765
NSpgid: 765
NSsid: 765
VmPeak: 2668472 kB
VmSize: 2661228 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 1582920 kB
VmRSS: 1575892 kB
RssAnon: 1553640 kB
RssFile: 22252 kB
RssShmem: 0 kB
VmData: 1703408 kB
VmStk: 132 kB
VmExe: 4 kB
VmLib: 18396 kB
VmPTE: 3320 kB
VmSwap: 0 kB
HugetlbPages: 0 kB
CoreDumping: 0
THP_enabled: 1
Threads: 16
SigQ: 1/47667
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000001001000
SigCgt: 0000000100000002
CapInh: 0000000000000000
CapPrm: 000001ffffffffff
CapEff: 000001ffffffffff
CapBnd: 000001ffffffffff
CapAmb: 0000000000000000
NoNewPrivs: 0
Seccomp: 0
Seccomp_filters: 0
Speculation_Store_Bypass: thread vulnerable
SpeculationIndirectBranch: conditional enabled
Cpus_allowed: 3
Cpus_allowed_list: 0-1
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 124632
nonvoluntary_ctxt_switches: 792
And at the top
Code: Select all
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
780384 root 20 0 17936 13684 6368 R 32.2 0.1 0:00.97 python
765 root 20 0 2663276 1.5g 22252 S 12.6 12.9 167:54.41 waptservice
As for the logs, I don't know what to look for. But I see that it's looping non-stop during communication with the server.
Code: Select all
2024-06-13T09:57:18.115855+02:00 hostname waptservice[765]: 2024-06-13 09:57:18,115 [wapttasks WaptTaskManager 139878128477888] WARNING Host on the server is not known or not known under this FQDN name (known as hostname.domain.tld). Trying to register the computer...
2024-06-13T09:57:20.103487+02:00 hostname waptservice[765]: 2024-06-13 09:57:20,103 [wapttasks WaptTaskManager 139878128477888] INFO Got signed certificate from server. Issuer: wapt-server.domain.tld. CN: hostname.domain.tld
2024-06-13T09:57:20.109998+02:00 hostname waptservice[765]: 2024-06-13 09:57:20,109 [wapttasks WaptTaskManager 139878128477888] INFO Save host key to /opt/wapt/private/hostname.domain.tld.pem
2024-06-13T09:57:20.110554+02:00 hostname waptservice[765]: 2024-06-13 09:57:20,110 [wapttasks WaptTaskManager 139878128477888] INFO Save host cert to /opt/wapt/private/hostname.domain.tld.crt
2024-06-13T09:57:20.132849+02:00 hostname waptservice[765]: 2024-06-13 09:57:20,132 [wapttasks WaptTaskManager 139878128477888] INFO New registration successful. Retring sending host status.
2024-06-13T09:57:20.133393+02:00 hostname waptservice[765]: 2024-06-13 09:57:20,133 [wapttasks WaptTaskManager 139878128477888] INFO Reloading waptcore configuration for Wapt instance thread 139878128477888
2024-06-13T09:57:21.789508+02:00 hostname waptservice[765]: 2024-06-13 09:57:21,789 [wapttasks WaptTaskManager 139878128477888] INFO update_server_status successful (data size 1735)
2024-06-13T09:57:21.790316+02:00 hostname waptservice[765]: 2024-06-13 09:57:21,790 [wapttasks WaptTaskManager 139878128477888] WARNING Host on the server is not known or not known under this FQDN name (known as hostname.domain.tld). Trying to register the computer...
2024-06-13T09:57:23.712029+02:00 hostname waptservice[765]: 2024-06-13 09:57:23,711 [wapttasks WaptTaskManager 139878128477888] INFO Got signed certificate from server. Issuer: wapt-server.domain.tld. CN: hostname.domain.tld
2024-06-13T09:57:23.716954+02:00 hostname waptservice[765]: 2024-06-13 09:57:23,716 [wapttasks WaptTaskManager 139878128477888] INFO Save host key to /opt/wapt/private/hostname.domain.tld.pem
2024-06-13T09:57:23.717392+02:00 hostname waptservice[765]: 2024-06-13 09:57:23,717 [wapttasks WaptTaskManager 139878128477888] INFO Save host cert to /opt/wapt/private/hostname.domain.tld.crt
2024-06-13T09:57:23.737197+02:00 hostname waptservice[765]: 2024-06-13 09:57:23,736 [wapttasks WaptTaskManager 139878128477888] INFO New registration successful. Retring sending host status.
2024-06-13T09:57:23.738109+02:00 hostname waptservice[765]: 2024-06-13 09:57:23,737 [wapttasks WaptTaskManager 139878128477888] INFO Reloading waptcore configuration for Wapt instance thread 139878128477888
2024-06-13T09:57:25.439076+02:00 hostname waptservice[765]: 2024-06-13 09:57:25,438 [wapttasks WaptTaskManager 139878128477888] INFO update_server_status successful (data size 1735)
2024-06-13T09:57:25.439932+02:00 hostname waptservice[765]: 2024-06-13 09:57:25,439 [wapttasks WaptTaskManager 139878128477888] WARNING Host on the server is not known or not known under this FQDN name (known as hostname.domain.tld). Trying to register the computer...
On the other VM, there are no errors in the log, no communication problems with the server.
Re: Agent - RAM usage under Linux
Published: June 13, 2024 - 10:37
by dcardon
Hi David,
basically there are two problems here. A small memory leak on the WAPT side (we're working on it), and a register that's looping and triggering this leak.
We need to check if two machines with the same UUID are crashing each other on the WAPT server.
Regards,
Denis
Re: Agent - RAM usage under Linux
Published: June 13, 2024 - 12:50
by Klug
Thanks for the feedback.
We'll check the UUID and wait for news about the memory leak.
Re: Agent - RAM usage under Linux
Published: June 20, 2024 - 2:16 PM
by dcardon
Hi David,
we've just released a new version of WAPT 2.5.5 with a fix for a memory leak (hopefully there aren't any other corner cases we haven't found
).
https://www.wapt.fr/en/doc/wapt-changel ... 2024-06-18
I'm marking this topic as resolved. Thanks for bringing it to our attention.
Best regards,
Denis