[SOLVED] Agent - RAM usage under Linux

[Klug]

Hello everyone,

I have a client who uses WAPT to manage their workstations and servers (VMs).
They have Windows (workstations and servers), Ubuntu (18.04 to 22.04), Debian (12), and CentOS (7).

We just realized that, on some Linux VMs, the WAPT client (waptagent 2.4.0) is using several GB of RAM.
I just randomly selected two VMs: one is using 1.3 GB of RAM, and the other 2.5 GB.

Does this seem "normal," or is there a problem with the deployment (or the client version, or something else)?

[dcardon]

Hi David,

regarding the figures you provided, are they RSS (resident memory size) or VSZ (virtual memory size)?

RSS should normally stay around 150-200MB; otherwise, there might be something to look into. Calculating VSZ virtual memory is a bit more complex because it includes many things, such as .so libraries that are mapped with mmap() but don't necessarily occupy RAM.

Regards,

Denis

[Klug]

Good morning,

Here is a little more detail about what I saw.

For the VM with 2 GB and a bit more:

Code: Select all

# pmap -x 1152862
1152862:   waptservice
Address           Kbytes     RSS   Dirty Mode  Mapping
.../... obligé de couper pour pouvoir poster
---------------- ------- ------- -------
total kB         2498900 1188768 1178376

And for the other one:

Code: Select all

# pmap -x 1325965
1325965:   waptservice
Address           Kbytes     RSS   Dirty Mode  Mapping
.../... obligé de couper pour pouvoir poster
---------------- ------- ------- -------
total kB         1398308  269136  248088

[Klug]

A different perspective (top) for the first one (3 GB):

Code: Select all

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
1152862 root      20   0 3082032   1.4g   9048 S  25.0  24.9 253:43.04 waptservice
2116295 root      20   0   17936  13720   6408 R   5.7   0.2   0:00.17 python

[Klug]

And in another way, which gives 3 GB in VmSize:

Code: Select all

# cat /proc/1152862/status
Name:   waptservice
Umask:  0022
State:  S (sleeping)
Tgid:   1152862
Ngid:   0
Pid:    1152862
PPid:   1
TracerPid:      0
Uid:    0       0       0       0
Gid:    0       0       0       0
FDSize: 128
Groups:
NStgid: 1152862
NSpid:  1152862
NSpgid: 1152862
NSsid:  1152862
VmPeak:  3139120 kB
VmSize:  3082032 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:   1578124 kB
VmRSS:   1513280 kB
RssAnon:         1504232 kB
RssFile:            9048 kB
RssShmem:              0 kB
VmData:  2086008 kB
VmStk:       132 kB
VmExe:         4 kB
VmLib:     18396 kB
VmPTE:      4072 kB
VmSwap:   430308 kB
HugetlbPages:          0 kB
CoreDumping:    0
THP_enabled:    1
Threads:        16
SigQ:   1/23523
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000001001000
SigCgt: 0000000100000002
CapInh: 0000000000000000
CapPrm: 000001ffffffffff
CapEff: 000001ffffffffff
CapBnd: 000001ffffffffff
CapAmb: 0000000000000000
NoNewPrivs:     0
Seccomp:        0
Seccomp_filters:        0
Speculation_Store_Bypass:       thread vulnerable
SpeculationIndirectBranch:      conditional enabled
Cpus_allowed:   3
Cpus_allowed_list:      0-1
Mems_allowed:   00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list:      0
voluntary_ctxt_switches:        182496
nonvoluntary_ctxt_switches:     276

[dcardon]

Hi again David,

what's the exact WAPT version? Edition? (see forum rules)

The 1.4GB RSS feed size is indeed very unusual. Are there any unusual entries in the logs? If you restart the agent, what memory level does it return to, and how quickly does it increase? How long does it take for the agents to reach 1.4GB?

Regards,

Denis

[Klug]

Good morning.

On the VMs in question, I have this: 2.4.0.14143-9847ee8b-debian-12-amd64
I don't know about the server, I'll ask the question.

On one of the VMs I was talking about yesterday (the one with the 3 GB process), RAM was added yesterday (so the VM was restarted).

Code: Select all

# ps -p 765 -o etime
    ELAPSED
   16:00:46

We're already at 1.6 GB on VmRSS, in 16 hours...

Code: Select all

# cat /proc/765/status
Name:   waptservice
Umask:  0022
State:  S (sleeping)
Tgid:   765
Ngid:   0
Pid:    765
PPid:   1
TracerPid:      0
Uid:    0       0       0       0
Gid:    0       0       0       0
FDSize: 128
Groups:
NStgid: 765
NSpid:  765
NSpgid: 765
NSsid:  765
VmPeak:  2668472 kB
VmSize:  2661228 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:   1582920 kB
VmRSS:   1575892 kB
RssAnon:         1553640 kB
RssFile:           22252 kB
RssShmem:              0 kB
VmData:  1703408 kB
VmStk:       132 kB
VmExe:         4 kB
VmLib:     18396 kB
VmPTE:      3320 kB
VmSwap:        0 kB
HugetlbPages:          0 kB
CoreDumping:    0
THP_enabled:    1
Threads:        16
SigQ:   1/47667
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000001001000
SigCgt: 0000000100000002
CapInh: 0000000000000000
CapPrm: 000001ffffffffff
CapEff: 000001ffffffffff
CapBnd: 000001ffffffffff
CapAmb: 0000000000000000
NoNewPrivs:     0
Seccomp:        0
Seccomp_filters:        0
Speculation_Store_Bypass:       thread vulnerable
SpeculationIndirectBranch:      conditional enabled
Cpus_allowed:   3
Cpus_allowed_list:      0-1
Mems_allowed:   00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list:      0
voluntary_ctxt_switches:        124632
nonvoluntary_ctxt_switches:     792

And at the top

Code: Select all

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 780384 root      20   0   17936  13684   6368 R  32.2   0.1   0:00.97 python
    765 root      20   0 2663276   1.5g  22252 S  12.6  12.9 167:54.41 waptservice

As for the logs, I don't know what to look for. But I see that it's looping non-stop during communication with the server.

Code: Select all

2024-06-13T09:57:18.115855+02:00 hostname waptservice[765]: 2024-06-13 09:57:18,115 [wapttasks WaptTaskManager 139878128477888] WARNING Host on the server is not known or not known under this FQDN name (known as hostname.domain.tld). Trying to register the computer...
2024-06-13T09:57:20.103487+02:00 hostname waptservice[765]: 2024-06-13 09:57:20,103 [wapttasks WaptTaskManager 139878128477888] INFO Got signed certificate from server. Issuer: wapt-server.domain.tld. CN: hostname.domain.tld
2024-06-13T09:57:20.109998+02:00 hostname waptservice[765]: 2024-06-13 09:57:20,109 [wapttasks WaptTaskManager 139878128477888] INFO Save host key to /opt/wapt/private/hostname.domain.tld.pem
2024-06-13T09:57:20.110554+02:00 hostname waptservice[765]: 2024-06-13 09:57:20,110 [wapttasks WaptTaskManager 139878128477888] INFO Save host cert to /opt/wapt/private/hostname.domain.tld.crt
2024-06-13T09:57:20.132849+02:00 hostname waptservice[765]: 2024-06-13 09:57:20,132 [wapttasks WaptTaskManager 139878128477888] INFO New registration successful. Retring sending host status.
2024-06-13T09:57:20.133393+02:00 hostname waptservice[765]: 2024-06-13 09:57:20,133 [wapttasks WaptTaskManager 139878128477888] INFO Reloading waptcore configuration for Wapt instance thread 139878128477888
2024-06-13T09:57:21.789508+02:00 hostname waptservice[765]: 2024-06-13 09:57:21,789 [wapttasks WaptTaskManager 139878128477888] INFO update_server_status successful (data size 1735)
2024-06-13T09:57:21.790316+02:00 hostname waptservice[765]: 2024-06-13 09:57:21,790 [wapttasks WaptTaskManager 139878128477888] WARNING Host on the server is not known or not known under this FQDN name (known as hostname.domain.tld). Trying to register the computer...
2024-06-13T09:57:23.712029+02:00 hostname waptservice[765]: 2024-06-13 09:57:23,711 [wapttasks WaptTaskManager 139878128477888] INFO Got signed certificate from server. Issuer: wapt-server.domain.tld. CN: hostname.domain.tld
2024-06-13T09:57:23.716954+02:00 hostname waptservice[765]: 2024-06-13 09:57:23,716 [wapttasks WaptTaskManager 139878128477888] INFO Save host key to /opt/wapt/private/hostname.domain.tld.pem
2024-06-13T09:57:23.717392+02:00 hostname waptservice[765]: 2024-06-13 09:57:23,717 [wapttasks WaptTaskManager 139878128477888] INFO Save host cert to /opt/wapt/private/hostname.domain.tld.crt
2024-06-13T09:57:23.737197+02:00 hostname waptservice[765]: 2024-06-13 09:57:23,736 [wapttasks WaptTaskManager 139878128477888] INFO New registration successful. Retring sending host status.
2024-06-13T09:57:23.738109+02:00 hostname waptservice[765]: 2024-06-13 09:57:23,737 [wapttasks WaptTaskManager 139878128477888] INFO Reloading waptcore configuration for Wapt instance thread 139878128477888
2024-06-13T09:57:25.439076+02:00 hostname waptservice[765]: 2024-06-13 09:57:25,438 [wapttasks WaptTaskManager 139878128477888] INFO update_server_status successful (data size 1735)
2024-06-13T09:57:25.439932+02:00 hostname waptservice[765]: 2024-06-13 09:57:25,439 [wapttasks WaptTaskManager 139878128477888] WARNING Host on the server is not known or not known under this FQDN name (known as hostname.domain.tld). Trying to register the computer...

On the other VM, there are no errors in the log, no communication problems with the server.

[dcardon]

Hi David,

basically there are two problems here. A small memory leak on the WAPT side (we're working on it), and a register that's looping and triggering this leak.

We need to check if two machines with the same UUID are crashing each other on the WAPT server.

Regards,

Denis

[Klug]

Thanks for the feedback.

We'll check the UUID and wait for news about the memory leak.

[dcardon]

Hi David,

we've just released a new version of WAPT 2.5.5 with a fix for a memory leak (hopefully there aren't any other corner cases we haven't found ).

https://www.wapt.fr/en/doc/wapt-changel ... 2024-06-18

I'm marking this topic as resolved. Thanks for bringing it to our attention.

Best regards,

Denis