[SOLVED] Wapt server 1.8 enterprise crash

olaplanche · May 18, 2021 - 09:40

Good morning,

I have a new problem following the upgrade of the server to version 1.8 enterprise.
For your information, I also upgraded the Debian Stretch server to Buster and PostgreSQL 9.6 version 11.

After approximately 24 hours of trouble-free operation, the wapt server stopped responding (web and console = 500 Internal Server Error)

# systemctl status waptserver

● waptserver.service - WAPT Server startup script
Loaded: loaded (/usr/lib/systemd/system/waptserver.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-05-17 08:22:25 CEST; 1 day 1h ago
Main PID: 502 (waptserver)
Tasks: 1 (limit: 4649)
Memory: 181.6M
CGroup: /system.slice/waptserver.service
└─502 waptserver

May 18 08:41:20 wapt-svr1 waptserver[502]: 2021-05-18 08:41:20,651 [waptws ] WARNING Invalid session d855214ff97e4de0a70fe10ca4f7576b
May 18 08:59:19 wapt-svr1 waptserver[502]: 2021-05-18 08:59:19,738 [waptws ] WARNING Invalid session 1982b44631e3467f9ee26692f192a3c9
May 18 09:02:34 wapt-svr1 waptserver[502]: 2021-05-18 09:02:34,811 [waptws ] WARNING Invalid session a7598043c4cd45ef8c707ae135325b31
May 18 09:03:33 wapt-svr1 waptserver[502]: 2021-05-18 09:03:33,024 [waptws ] WARNING Invalid session 551799f8134c41aca1f4bb202b3d63d3
May 18 09:04:24 wapt-svr1 waptserver[502]: 2021-05-18 09:04:24,840 [waptws ] WARNING Invalid session ae9fc3e3bc654402ade40c0978e7d144
May 18 09:04:52 wapt-svr1 waptserver[502]: 2021-05-18 09:04:52,668 [waptws ] WARNING Invalid session 93877a82dbcd48fe86ff5cd87df63a17
May 18 09:18:22 wapt-svr1 waptserver[502]: 2021-05-18 09:18:22,735 [waptws ] WARNING Invalid session f6015710fce14bf8aa1ee86b24c90391
May 18 09:19:44 wapt-svr1 waptserver[502]: 2021-05-18 09:19:44,804 [waptws ] WARNING Invalid session 1674992d543c47ea9c341082748935fd
May 18 09:20:28 wapt-svr1 waptserver[502]: 2021-05-18 09:20:28,566 [waptws ] WARNING Invalid session 076a04ffa5684960bbae4563b47e512c
May 18 09:21:54 wapt-svr1 waptserver[502]: 2021-05-18 09:21:54,495 [waptws ] WARNING Invalid session ac73b071b4e94c1285509cea2e6f6f36

No problem with the admin password, after restarting I can log in to the console with it.

No issues with the status of the postgresql service.

# sudo systemctl status nginx.service

● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/nginx.service.d
└─nginx_worker_files_limit.conf
Active: active (running) since Mon 2021-05-17 08:22:20 IT IS; 1 day 1h ago
Docs: man:nginx(8)
Process: 459 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_processon; (code=exited, status=0/SUCCESS)
Process: 479 ExecStart=/usr/sbin/nginx -g daemon on; master_processon; (code=exited, status=0/SUCCESS)
Main PID: 480 (nginx)
Tasks: 3 (limit: 4649)
Memory: 771.3M
CGroup: /system.slice/nginx.service
├─480 nginx: master process /usr/sbin/nginx -g daemon on; master_processon;
├─481 nginx: worker process
└─482 nginx: worker process

May 17 08:22:18 wapt-svr1 systemd[1]: Starting A high performance web server and a reverse proxy server...
May 17 08:22:20 wapt-svr1 nginx[459]: nginx: [warn] "ssl_stapling" ignored, issuer certificate not found for certificate "/opt/wapt/waptserver/ssl/cert.pem"
May 17 08:22:20 wapt-svr1 nginx[479]: nginx: [warn] "ssl_stapling" ignored, issuer certificate not found for certificate "/opt/wapt/waptserver/ssl/cert.pem"
May 17 08:22:20 wapt-svr1 systemd[1]: Started A high performance web server and a reverse proxy server.

A more conclusive excerpt from the waptserver.log file?

May 17 08:21:33 wapt-svr1 waptserver[7666]: 2021-05-17 08:21:33,259 [waptws ] CRITICAL SocketIO pong error for uuid etude1-dell-02 and sid 6c48df0413bb48858e7736387cdb032d : Traceback (most recent call last):
May 17 08:21:33 wapt-svr1 waptserver[7666]: File "/opt/wapt/waptserver/server_socketio.py", line 284, in on_wapt_pong
May 17 08:21:33 wapt-svr1 waptserver[7666]: with WaptDB():
May 17 08:21:33 wapt-svr1 waptserver[7666]: File "/opt/wapt/waptserver/model.py", line 124, in __enter__
May 17 08:21:33 wapt-svr1 waptserver[7666]: self.must_close = wapt_db_connect()
May 17 08:21:33 wapt-svr1 waptserver[7666]: File "/opt/wapt/waptserver/model.py", line 101, in wapt_db_connect
May 17 08:21:33 wapt-svr1 waptserver[7666]: wapt_db.connect()
May 17 08:21:33 wapt-svr1 waptserver[7666]: File "/opt/wapt/lib/python2.7/site-packages/playhouse/pool.py", line 114, in connect
May 17 08:21:33 wapt-svr1 waptserver[7666]: ret = super(PooledDatabase, self).connect(reuse_if_open)
May 17 08:21:33 wapt-svr1 waptserver[7666]: File "/opt/wapt/lib/python2.7/site-packages/peewee.py", line 2947, in connect
May 17 08:21:33 wapt-svr1 waptserver[7666]: self._initialize_connection(self._state.conn)
May 17 08:21:33 wapt-svr1 waptserver[7666]: File "/opt/wapt/lib/python2.7/site-packages/peewee.py", line 2783, in __exit__
May 17 08:21:33 wapt-svr1 waptserver[7666]: reraise(new_type, new_type(*exc_args), traceback)
May 17 08:21:33 wapt-svr1 waptserver[7666]: File "/opt/wapt/lib/python2.7/site-packages/peewee.py", line 2944, in connect
May 17 08:21:33 wapt-svr1 waptserver[7666]: self._state.set_connection(self._connect())
May 17 08:21:33 wapt-svr1 waptserver[7666]: File "/opt/wapt/lib/python2.7/site-packages/playhouse/pool.py", line 156, in _connect
May 17 08:21:33 wapt-svr1 waptserver[7666]: conn = super(PooledDatabase, self)._connect()
May 17 08:21:33 wapt-svr1 waptserver[7666]: File "/opt/wapt/lib/python2.7/site-packages/playhouse/postgres_ext.py", line 448, in _connect
May 17 08:21:33 wapt-svr1 waptserver[7666]: conn = super(PostgresqlExtDatabase, self)._connect()
May 17 08:21:33 wapt-svr1 waptserver[7666]: File "/opt/wapt/lib/python2.7/site-packages/peewee.py", line 3637, in _connect
May 17 08:21:33 wapt-svr1 waptserver[7666]: conn = psycopg2.connect(database=self.database, **self.connect_params)
May 17 08:21:33 wapt-svr1 waptserver[7666]: File "/opt/wapt/lib/python2.7/site-packages/psycopg2/__init__.py", line 127, in connect
May 17 08:21:33 wapt-svr1 waptserver[7666]: conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
May 17 08:21:33 wapt-svr1 waptserver[7666]: OperationalError: could not connect to server: No such file or directory
May 17 08:21:33 wapt-svr1 waptserver[7666]: #011Is the server running locally and accepting
May 17 08:21:33 wapt-svr1 waptserver[7666]: #011connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
May 17 08:21:33 wapt-svr1 waptserver[7666]: , instance:

An excerpt from the postresql-11.log file (You can see that I restarted the server yesterday at 8:22 AM)

2021-05-17 08:21:22.943 CEST [479] LOG: received fast shutdown request
2021-05-17 08:21:22.950 CEST [479] LOG: aborting any active transactions
2021-05-17 08:21:22.950 CEST [42359] wapt@wapt FATAL: terminating connection due to administrator command
2021-05-17 08:21:22.950 CEST [42262] wapt@wapt FATAL: terminating connection due to administrator command
2021-05-17 08:21:22.954 CEST [42358] wapt@wapt FATAL: terminating connection due to administrator command
2021-05-17 08:21:22.955 CEST [42357] wapt@wapt FATAL: terminating connection due to administrator command
2021-05-17 08:21:23.004 CEST [479] LOG: background worker "logical replication launcher" (PID 490) exited with exit code 1
2021-05-17 08:21:23.012 CEST [485] LOG: shutting down
2021-05-17 08:21:23.208 CEST [479] LOG: database system is shut down
2021-05-17 08:21:33.427 CEST [43824] LOG: listening on IPv6 address "::1", port 5432
2021-05-17 08:21:33.427 CEST [43824] LOG: listening on IPv4 address "127.0.0.1", port 5432
2021-05-17 08:21:33.433 CEST [43824] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-05-17 08:21:33.557 CEST [43825] LOG: database system was shut down at 2021-05-17 08:21:23 CEST
2021-05-17 08:21:33.612 CEST [43824] LOG: database system is ready to accept connections
2021-05-17 08:21:34.125 CEST [43832] [unknown]@[unknown] LOG: incomplete startup packet
2021-05-17 08:21:59.862 CEST [43824] LOG: received fast shutdown request
2021-05-17 08:21:59.877 CEST [43824] LOG: aborting any active transactions
2021-05-17 08:21:59.884 CEST [43824] LOG: background worker "logical replication launcher" (PID 43831) exited with exit code 1
2021-05-17 08:21:59.884 CEST [43826] LOG: shutting down
2021-05-17 08:22:00.175 CEST [43824] LOG: database system is shut down
2021-05-17 08:22:22.338 CEST [483] LOG: listening on IPv6 address "::1", port 5432
2021-05-17 08:22:22.340 CEST [483] LOG: listening on IPv4 address "127.0.0.1", port 5432
2021-05-17 08:22:22.360 CEST [483] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-05-17 08:22:22.546 CEST [484] LOG: database system was shut down at 2021-05-17 08:22:00 CEST
2021-05-17 08:22:22.672 CEST [483] LOG: database system is ready to accept connections
2021-05-17 08:22:23.196 CEST [491] [unknown]@[unknown] LOG: incomplete startup packet

Thank you for reading, hoping to find some help.

olaplanche · May 18, 2021 - 10:50

Additionally, here is an excerpt from the waptserver.log file after restarting the waptserver service and once web and console access are working again:

May 18 10:48:11 wapt-svr1 waptserver[9423]: 2021-05-18 10:48:11,617 [waptws ] WARNING Invalid session None
May 18 10:48:11 wapt-svr1 waptserver[9423]: 2021-05-18 10:48:11,632 [waptws ] WARNING Invalid session None
May 18 10:48:11 wapt-svr1 waptserver[9423]: 2021-05-18 10:48:11,633 [waptws ] WARNING Invalid session None

May 18, 2021 - 3:26 PM

Does this server have a graphical interface? Are there any up/down times on the network interfaces? We've already seen this type of issue on an installation with a GUI. (
Denis)

olaplanche · May 18, 2021 - 4:23 PM

Hello,

there's no graphical interface, but your questions reminded me of a problem I encountered on a Windows VM due to the scheduled nightly backup. Although I've never had any issues before, I've temporarily disabled the VM backup on the WAPT server for tonight.

I'll check tomorrow morning if the server is still accessible.

Thank you.

olaplanche · May 19, 2021 - 11:47

Hello,

I'm having the same problem this morning, the Wapt server is inaccessible...

Yesterday I only restarted the Waptserver service to regain console and web access. This morning I got the same result after restarting only the Nginx service...

olaplanche · May 19, 2021 - 4:51 PM

Looking at the nginx side, it turns out that my config file /etc/nginx/site-enabled/wapt.conf is not identical to the default one in the documentation https://www.wapt.fr/fr/doc/wapt-configuration ... index.html

server {

listen 80;

listen 443 ssl;
server_name _;

ssl_certificate "/opt/wapt/waptserver/ssl/cert.pem";
ssl_certificate_key "/opt/wapt/waptserver/ssl/key.pem";
ssl_protocols TLSv1.2;
ssl_dhparam /etc/ssl/certs/dhparam.pem;
ssl_prefer_server_ciphers on;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
ssl_staplingon;
ssl_stapling_verify on;
ssl_session_cache none;
ssl_session_tickets off;

#ssl_client_certificate "/opt/wapt/conf/ca-wapt-svr1.mondomaine.fr.crt";
#ssl_crl "None";
#ssl_verify_client optional;

gzip_min_length 1000;
gzip_buffers 4 8k;
gzip_http_version 1.0;
gzip_disable "msie6";
gzip_types text/plain text/css application/json;
gzip_vary on;

indexindex.html;

location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# be sure these headers are not forwarded
proxy_set_header X-Ssl-Client-Dn "";
proxy_set_header X-Ssl-Authenticated "";

client_max_body_size 4096m;
client_body_timeout 1800;

location /static {
alias "/opt/wapt/waptserver/static";
}

location /ssl {
alias "/var/www/ssl";
}

location ~ ^/(wapt/waptsetup-tis.exe|wapt/waptagent.exe|wapt/waptdeploy.exe|sync.json|rules.json)$ {
add_header Cache-Control "store, no-cache, must-revalidate, post-check=0, pre-check=0";
add_header Pragma "no-cache";
root "/var/www";
}

location ~ ^/(wapt/.*|waptwua/.*|wapt-diff-repos/.*)$ {
add_header Cache-Control "store, no-cache, must-revalidate, post-check=0, pre-check=0";
add_header Pragma "no-cache";

root "/var/www";
}

location ~ ^/(wapt-host/.*)$ {
log_not_found off;
add_header Cache-Control "store, no-cache, must-revalidate, post-check=0, pre-check=0";
add_header Pragma "no-cache";

root "/var/www";
}

location / {
add_header Cache-Control "store, no-cache, must-revalidate, post-check=0, pre-check=0";
add_header Pragma "no-cache";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

client_max_body_size 4096m;
client_body_timeout 1800;

location /add_host_kerberos {
auth_gss on;
auth_gss_keytab /etc/nginx/http-krb5.keytab;
proxy_pass http://127.0.0.1:8080;
}

location /wapt-host/Packages {
return 403;
}

location / {

add_header X-Forwarded-List $http_x_forwarded_for;
add_header X-Remote-IP $remote_addr;
proxy_pass http://127.0.0.1:8080;
}

location /socket.io {
proxy_http_version 1.1;
proxy_bufferingoff;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_pass http://127.0.0.1:8080/socket.io;
}
}
}
}

Do you think it's necessary to restore the default configuration file?

THANKS

olaplanche · May 20, 2021 - 10:49

Hello and good news (I hope), the server didn't crash last night!!!

Yesterday I implemented the following recommendations because I have "a little" more than 400 workstations: https://www.wapt.fr/fr/doc/wapt-configu ... index.html

And I also restarted the postconf, agreeing to reconfigure nginx (which doesn't change my /etc/nginx/site-enabled/wapt.conf file above, but rather this one /etc/nginx/nginx.conf).

I'll wait until tomorrow before marking this post as [RESOLVED] if the server hasn't crashed again in the meantime.

olaplanche · May 21, 2021 - 08:16

Hello,

The server is still up following the previous modifications

. I am marking the topic as resolved.