Page 1 sur 1

[RESOLU] Plantage wapt server 1.8 enterprise

Publié : 18 mai 2021 - 09:40
par olaplanche
Bonjour,

J'ai un nouveau problème suite au passage du serveur en version 1.8 enterprise.
Pour info j'ai par la même occasion fait l'upgrade du serveur debian strech vers buster et postgresql 9.6 ver 11.

Après environ 24h d'exécution sans problème, le serveur wapt ne répond plus (web et console = 500 Internal Server Error)

# systemctl status waptserver
● waptserver.service - WAPT Server startup script
Loaded: loaded (/usr/lib/systemd/system/waptserver.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-05-17 08:22:25 CEST; 1 day 1h ago
Main PID: 502 (waptserver)
Tasks: 1 (limit: 4649)
Memory: 181.6M
CGroup: /system.slice/waptserver.service
└─502 waptserver

May 18 08:41:20 wapt-svr1 waptserver[502]: 2021-05-18 08:41:20,651 [waptws ] WARNING Invalid session d855214ff97e4de0a70fe10ca4f7576b
May 18 08:59:19 wapt-svr1 waptserver[502]: 2021-05-18 08:59:19,738 [waptws ] WARNING Invalid session 1982b44631e3467f9ee26692f192a3c9
May 18 09:02:34 wapt-svr1 waptserver[502]: 2021-05-18 09:02:34,811 [waptws ] WARNING Invalid session a7598043c4cd45ef8c707ae135325b31
May 18 09:03:33 wapt-svr1 waptserver[502]: 2021-05-18 09:03:33,024 [waptws ] WARNING Invalid session 551799f8134c41aca1f4bb202b3d63d3
May 18 09:04:24 wapt-svr1 waptserver[502]: 2021-05-18 09:04:24,840 [waptws ] WARNING Invalid session ae9fc3e3bc654402ade40c0978e7d144
May 18 09:04:52 wapt-svr1 waptserver[502]: 2021-05-18 09:04:52,668 [waptws ] WARNING Invalid session 93877a82dbcd48fe86ff5cd87df63a17
May 18 09:18:22 wapt-svr1 waptserver[502]: 2021-05-18 09:18:22,735 [waptws ] WARNING Invalid session f6015710fce14bf8aa1ee86b24c90391
May 18 09:19:44 wapt-svr1 waptserver[502]: 2021-05-18 09:19:44,804 [waptws ] WARNING Invalid session 1674992d543c47ea9c341082748935fd
May 18 09:20:28 wapt-svr1 waptserver[502]: 2021-05-18 09:20:28,566 [waptws ] WARNING Invalid session 076a04ffa5684960bbae4563b47e512c
May 18 09:21:54 wapt-svr1 waptserver[502]: 2021-05-18 09:21:54,495 [waptws ] WARNING Invalid session ac73b071b4e94c1285509cea2e6f6f36
Aucun problème avec le mot de passe admin, après redémarrage je peux bien me connecter avec à la console.

RAS sur le status du service postgresql.

# sudo systemctl status nginx.service
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/nginx.service.d
└─nginx_worker_files_limit.conf
Active: active (running) since Mon 2021-05-17 08:22:20 CEST; 1 day 1h ago
Docs: man:nginx(8)
Process: 459 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 479 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 480 (nginx)
Tasks: 3 (limit: 4649)
Memory: 771.3M
CGroup: /system.slice/nginx.service
├─480 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
├─481 nginx: worker process
└─482 nginx: worker process

May 17 08:22:18 wapt-svr1 systemd[1]: Starting A high performance web server and a reverse proxy server...
May 17 08:22:20 wapt-svr1 nginx[459]: nginx: [warn] "ssl_stapling" ignored, issuer certificate not found for certificate "/opt/wapt/waptserver/ssl/cert.pem"
May 17 08:22:20 wapt-svr1 nginx[479]: nginx: [warn] "ssl_stapling" ignored, issuer certificate not found for certificate "/opt/wapt/waptserver/ssl/cert.pem"
May 17 08:22:20 wapt-svr1 systemd[1]: Started A high performance web server and a reverse proxy server.
Un extrait plus probant du fichier waptserver.log ?
May 17 08:21:33 wapt-svr1 waptserver[7666]: 2021-05-17 08:21:33,259 [waptws ] CRITICAL SocketIO pong error for uuid etude1-dell-02 and sid 6c48df0413bb48858e7736387cdb032d : Traceback (most recent call last):
May 17 08:21:33 wapt-svr1 waptserver[7666]: File "/opt/wapt/waptserver/server_socketio.py", line 284, in on_wapt_pong
May 17 08:21:33 wapt-svr1 waptserver[7666]: with WaptDB():
May 17 08:21:33 wapt-svr1 waptserver[7666]: File "/opt/wapt/waptserver/model.py", line 124, in __enter__
May 17 08:21:33 wapt-svr1 waptserver[7666]: self.must_close = wapt_db_connect()
May 17 08:21:33 wapt-svr1 waptserver[7666]: File "/opt/wapt/waptserver/model.py", line 101, in wapt_db_connect
May 17 08:21:33 wapt-svr1 waptserver[7666]: wapt_db.connect()
May 17 08:21:33 wapt-svr1 waptserver[7666]: File "/opt/wapt/lib/python2.7/site-packages/playhouse/pool.py", line 114, in connect
May 17 08:21:33 wapt-svr1 waptserver[7666]: ret = super(PooledDatabase, self).connect(reuse_if_open)
May 17 08:21:33 wapt-svr1 waptserver[7666]: File "/opt/wapt/lib/python2.7/site-packages/peewee.py", line 2947, in connect
May 17 08:21:33 wapt-svr1 waptserver[7666]: self._initialize_connection(self._state.conn)
May 17 08:21:33 wapt-svr1 waptserver[7666]: File "/opt/wapt/lib/python2.7/site-packages/peewee.py", line 2783, in __exit__
May 17 08:21:33 wapt-svr1 waptserver[7666]: reraise(new_type, new_type(*exc_args), traceback)
May 17 08:21:33 wapt-svr1 waptserver[7666]: File "/opt/wapt/lib/python2.7/site-packages/peewee.py", line 2944, in connect
May 17 08:21:33 wapt-svr1 waptserver[7666]: self._state.set_connection(self._connect())
May 17 08:21:33 wapt-svr1 waptserver[7666]: File "/opt/wapt/lib/python2.7/site-packages/playhouse/pool.py", line 156, in _connect
May 17 08:21:33 wapt-svr1 waptserver[7666]: conn = super(PooledDatabase, self)._connect()
May 17 08:21:33 wapt-svr1 waptserver[7666]: File "/opt/wapt/lib/python2.7/site-packages/playhouse/postgres_ext.py", line 448, in _connect
May 17 08:21:33 wapt-svr1 waptserver[7666]: conn = super(PostgresqlExtDatabase, self)._connect()
May 17 08:21:33 wapt-svr1 waptserver[7666]: File "/opt/wapt/lib/python2.7/site-packages/peewee.py", line 3637, in _connect
May 17 08:21:33 wapt-svr1 waptserver[7666]: conn = psycopg2.connect(database=self.database, **self.connect_params)
May 17 08:21:33 wapt-svr1 waptserver[7666]: File "/opt/wapt/lib/python2.7/site-packages/psycopg2/__init__.py", line 127, in connect
May 17 08:21:33 wapt-svr1 waptserver[7666]: conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
May 17 08:21:33 wapt-svr1 waptserver[7666]: OperationalError: could not connect to server: No such file or directory
May 17 08:21:33 wapt-svr1 waptserver[7666]: #011Is the server running locally and accepting
May 17 08:21:33 wapt-svr1 waptserver[7666]: #011connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
May 17 08:21:33 wapt-svr1 waptserver[7666]: , instance:
Un extrait du fichier postresql-11.log (On peut voir que j'ai redémarré le serveur hier à 8h22)
2021-05-17 08:21:22.943 CEST [479] LOG: received fast shutdown request
2021-05-17 08:21:22.950 CEST [479] LOG: aborting any active transactions
2021-05-17 08:21:22.950 CEST [42359] wapt@wapt FATAL: terminating connection due to administrator command
2021-05-17 08:21:22.950 CEST [42262] wapt@wapt FATAL: terminating connection due to administrator command
2021-05-17 08:21:22.954 CEST [42358] wapt@wapt FATAL: terminating connection due to administrator command
2021-05-17 08:21:22.955 CEST [42357] wapt@wapt FATAL: terminating connection due to administrator command
2021-05-17 08:21:23.004 CEST [479] LOG: background worker "logical replication launcher" (PID 490) exited with exit code 1
2021-05-17 08:21:23.012 CEST [485] LOG: shutting down
2021-05-17 08:21:23.208 CEST [479] LOG: database system is shut down
2021-05-17 08:21:33.427 CEST [43824] LOG: listening on IPv6 address "::1", port 5432
2021-05-17 08:21:33.427 CEST [43824] LOG: listening on IPv4 address "127.0.0.1", port 5432
2021-05-17 08:21:33.433 CEST [43824] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-05-17 08:21:33.557 CEST [43825] LOG: database system was shut down at 2021-05-17 08:21:23 CEST
2021-05-17 08:21:33.612 CEST [43824] LOG: database system is ready to accept connections
2021-05-17 08:21:34.125 CEST [43832] [unknown]@[unknown] LOG: incomplete startup packet
2021-05-17 08:21:59.862 CEST [43824] LOG: received fast shutdown request
2021-05-17 08:21:59.877 CEST [43824] LOG: aborting any active transactions
2021-05-17 08:21:59.884 CEST [43824] LOG: background worker "logical replication launcher" (PID 43831) exited with exit code 1
2021-05-17 08:21:59.884 CEST [43826] LOG: shutting down
2021-05-17 08:22:00.175 CEST [43824] LOG: database system is shut down
2021-05-17 08:22:22.338 CEST [483] LOG: listening on IPv6 address "::1", port 5432
2021-05-17 08:22:22.340 CEST [483] LOG: listening on IPv4 address "127.0.0.1", port 5432
2021-05-17 08:22:22.360 CEST [483] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-05-17 08:22:22.546 CEST [484] LOG: database system was shut down at 2021-05-17 08:22:00 CEST
2021-05-17 08:22:22.672 CEST [483] LOG: database system is ready to accept connections
2021-05-17 08:22:23.196 CEST [491] [unknown]@[unknown] LOG: incomplete startup packet
Merci de m'avoir lu, en espérant trouver de l'aide.

Re: Plantage wapt server 1.8 enterprise

Publié : 18 mai 2021 - 10:50
par olaplanche
En complément un extrait du fichier waptserver.log après avoir redémarré le service waptserver et que l'accès web et console fonctionne à nouveau :
May 18 10:48:11 wapt-svr1 waptserver[9423]: 2021-05-18 10:48:11,617 [waptws ] WARNING Invalid session None
May 18 10:48:11 wapt-svr1 waptserver[9423]: 2021-05-18 10:48:11,632 [waptws ] WARNING Invalid session None
May 18 10:48:11 wapt-svr1 waptserver[9423]: 2021-05-18 10:48:11,633 [waptws ] WARNING Invalid session None

Re: Plantage wapt server 1.8 enterprise

Publié : 18 mai 2021 - 15:26
par dcardon
est ce qu'il y a une interface graphique sur ce serveur? Est ce qu'il y a des up/down sur les interfaces réseau? On a déjà observé ce type de soucis sur une install avec gui
Denis

Re: Plantage wapt server 1.8 enterprise

Publié : 18 mai 2021 - 16:23
par olaplanche
Bonjour,

Pas d'interface graphique, mais vos questions m'ont fait penser à un problème rencontré sur une VM Windows à cause de la sauvegarde planifiée de la VM la nuit. Bien que je n'ai jamais eu de soucis jusqu'à présent, j'ai temporairement désactivé la sauvegarde de la VM du serveur wapt pour cette nuit.

Je verrai demain matin si le serveur est toujours accessible.

Merci

Re: Plantage wapt server 1.8 enterprise

Publié : 19 mai 2021 - 11:47
par olaplanche
Bonjour,

Même problème ce matin, serveur wapt inaccessible...

Hier j'ai seulement redémarré le service waptserver pour retrouver l'accès console et web. Ce matin même résultat en redémarrant seulement le service nginx...

Re: Plantage wapt server 1.8 enterprise

Publié : 19 mai 2021 - 16:51
par olaplanche
En regardant du côté de nginx, il s'avère que mon fichier de conf /etc/nginx/site-enabled/wapt.conf n'est pas identique à celui par défaut dans la doc https://www.wapt.fr/fr/doc/wapt-configu ... index.html
server {


listen 80;

listen 443 ssl;
server_name _;

ssl_certificate "/opt/wapt/waptserver/ssl/cert.pem";
ssl_certificate_key "/opt/wapt/waptserver/ssl/key.pem";
ssl_protocols TLSv1.2;
ssl_dhparam /etc/ssl/certs/dhparam.pem;
ssl_prefer_server_ciphers on;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
ssl_stapling on;
ssl_stapling_verify on;
ssl_session_cache none;
ssl_session_tickets off;


#ssl_client_certificate "/opt/wapt/conf/ca-wapt-svr1.mondomaine.fr.crt";
#ssl_crl "None";
#ssl_verify_client optional;


gzip_min_length 1000;
gzip_buffers 4 8k;
gzip_http_version 1.0;
gzip_disable "msie6";
gzip_types text/plain text/css application/json;
gzip_vary on;

index index.html;


location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# be sure these headers are not forwarded
proxy_set_header X-Ssl-Client-Dn "";
proxy_set_header X-Ssl-Authenticated "";

client_max_body_size 4096m;
client_body_timeout 1800;

location /static {
alias "/opt/wapt/waptserver/static";
}


location /ssl {
alias "/var/www/ssl";
}


location ~ ^/(wapt/waptsetup-tis.exe|wapt/waptagent.exe|wapt/waptdeploy.exe|sync.json|rules.json)$ {
add_header Cache-Control "store, no-cache, must-revalidate, post-check=0, pre-check=0";
add_header Pragma "no-cache";
root "/var/www";
}

location ~ ^/(wapt/.*|waptwua/.*|wapt-diff-repos/.*)$ {
add_header Cache-Control "store, no-cache, must-revalidate, post-check=0, pre-check=0";
add_header Pragma "no-cache";

root "/var/www";
}

location ~ ^/(wapt-host/.*)$ {
log_not_found off;
add_header Cache-Control "store, no-cache, must-revalidate, post-check=0, pre-check=0";
add_header Pragma "no-cache";

root "/var/www";
}

location / {
add_header Cache-Control "store, no-cache, must-revalidate, post-check=0, pre-check=0";
add_header Pragma "no-cache";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

client_max_body_size 4096m;
client_body_timeout 1800;


location /add_host_kerberos {
auth_gss on;
auth_gss_keytab /etc/nginx/http-krb5.keytab;
proxy_pass http://127.0.0.1:8080;
}


location /wapt-host/Packages {
return 403;
}

location / {

add_header X-Forwarded-List $http_x_forwarded_for;
add_header X-Remote-IP $remote_addr;
proxy_pass http://127.0.0.1:8080;
}

location /socket.io {
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_pass http://127.0.0.1:8080/socket.io;
}
}
}
}
Pensez-vous qu'il est nécessaire de restaurer le fichier de conf par défaut ?

Merci

Re: Plantage wapt server 1.8 enterprise

Publié : 20 mai 2021 - 10:49
par olaplanche
Bonjour et bonne nouvelle (je l'espère), le serveur n'a pas planté cette nuit !!!

Hier j'ai mis en pratique les recommandations suivantes car j'ai "un peu" plus de 400 postes : https://www.wapt.fr/fr/doc/wapt-configu ... index.html

Et j'ai également relancé la postconf en acceptant de reconfigurer nginx (ce qui ne change rien à mon fichier /etc/nginx/site-enabled/wapt.conf ci-dessus mais plutôt celui-ci /etc/nginx/nginx.conf)

J'attends jusqu'à demain avant de passer ce post en [RESOLU] si le serveur n'a pas replanté entre temps.

Re: Plantage wapt server 1.8 enterprise

Publié : 21 mai 2021 - 08:16
par olaplanche
Bonjour,

Le serveur est toujours debout suite aux modifications précédentes :D

Je passe le sujet en résolu.