WAPT server has stopped working

Questions about WAPT Server / Requests and help related to the WAPT server
Forum Rules
Community Forum Rules
* English support on www.reddit.com/r/wapt
* French community support is available on this forum
* Please prefix the topic title with [RESOLVED] if it is resolved.
* Please do not edit a topic that is tagged [RESOLVED]. Open a new topic referencing the old one.
* Specify the installed WAPT version, full version, and build number (2.2.1.11957 / 2.2.2.12337 / etc.) as well as the Enterprise/Discovery edition.
* Versions 1.8.2 and earlier are no longer supported. The only questions accepted regarding version 1.8.2 are related to upgrading to a supported version (2.1, 2.2, etc.).
* Specify the server OS (Linux/Windows) and version (Debian Buster/Bullseye - CentOS 7 - Windows Server 2012/2016/2019).
* Specify the OS of the administration/package creation machine and the machine with the problematic agent, if applicable (Windows 7/10/11/Debian 11/etc.).
* Avoid asking multiple questions when opening a topic, otherwise it may be ignored. If there are multiple topics, open separate topics, preferably one after the other and not all at the same time (i.e., do not spam the forum).
* Include code snippets, screenshots, and other images directly in the post. Links to Pastebin, Bitly, and other third-party sites will be systematically removed.
* As with any community forum, support is provided voluntarily by members. If you require commercial support, you can contact Tranquil IT's sales department at 02.40.97.57.55
Locked
Floflobel
Messages: 135
Registration: Oct 15, 2015 - 5:32 p.m.

June 20, 2018 - 2:56 PM

Good morning,

I own several WAPT servers, but at the moment one of them is going into error quite regularly.
A restart of the service and it works again.

Here is the error:

Code: Select all

Jun 20 10:03:20 wapt1 python[20624]: 2018-06-20 10:03:20,140 ERROR Receive error
Jun 20 10:03:20 wapt1 python[20624]: Traceback (most recent call last):
Jun 20 10:03:20 wapt1 python[20624]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Jun 20 10:03:20 wapt1 python[20624]:     self.receive(pkt)
Jun 20 10:03:20 wapt1 python[20624]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Jun 20 10:03:20 wapt1 python[20624]:     self.send(packet.Packet(packet.PONG, pkt.data))
Jun 20 10:03:20 wapt1 python[20624]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Jun 20 10:03:20 wapt1 python[20624]:     raise IOError('Socket is closed')
Jun 20 10:03:20 wapt1 python[20624]: IOError: Socket is closed
Jun 20 10:03:20 wapt1 python[20624]: 2018-06-20 10:03:20,217 ERROR Receive error
Jun 20 10:03:20 wapt1 python[20624]: Traceback (most recent call last):
Jun 20 10:03:20 wapt1 python[20624]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Jun 20 10:03:20 wapt1 python[20624]:     self.receive(pkt)
Jun 20 10:03:20 wapt1 python[20624]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Jun 20 10:03:20 wapt1 python[20624]:     self.send(packet.Packet(packet.PONG, pkt.data))
Jun 20 10:03:20 wapt1 python[20624]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Jun 20 10:03:20 wapt1 python[20624]:     raise IOError('Socket is closed')
Jun 20 10:03:20 wapt1 python[20624]: IOError: Socket is closed
Jun 20 10:03:20 wapt1 python[20624]: 2018-06-20 10:03:20,283 ERROR Receive error
Jun 20 10:03:20 wapt1 python[20624]: Traceback (most recent call last):
Jun 20 10:03:20 wapt1 python[20624]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Jun 20 10:03:20 wapt1 python[20624]:     self.receive(pkt)
Jun 20 10:03:20 wapt1 python[20624]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Jun 20 10:03:20 wapt1 python[20624]:     self.send(packet.Packet(packet.PONG, pkt.data))
Jun 20 10:03:20 wapt1 python[20624]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Jun 20 10:03:20 wapt1 python[20624]:     raise IOError('Socket is closed')
Jun 20 10:03:20 wapt1 python[20624]: IOError: Socket is closed
Debian GNU/Linux 9 (stretch)
WAPT Server version: 1.5.1.23
WAPT Agent version: 1.5.1.23
WAPT Setup version: 1.5.1.23
WAPT Deploy version: 1.5.1.23
DB status: OK (1.5.1.22)
Last edited by Floflobel on 20 Nov 2018 - 15:11, edited 1 time.
High
Floflobel
Messages: 135
Registration: Oct 15, 2015 - 5:32 p.m.

August 30, 2018 - 11:54

I increased the number of connections to the postgres database and it crashes less often, but it's still the case.
Any ideas?

Code: Select all

Aug 28 09:58:04 wapt1 python[454]: 2018-08-28 09:58:04,419 ERROR Receive error
Aug 28 09:58:04 wapt1 python[454]: Traceback (most recent call last):
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Aug 28 09:58:04 wapt1 python[454]:     self.receive(pkt)
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Aug 28 09:58:04 wapt1 python[454]:     self.send(packet.Packet(packet.PONG, pkt.data))
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Aug 28 09:58:04 wapt1 python[454]:     raise IOError('Socket is closed')
Aug 28 09:58:04 wapt1 python[454]: IOError: Socket is closed
Aug 28 09:58:04 wapt1 python[454]: 2018-08-28 09:58:04,501 ERROR Receive error
Aug 28 09:58:04 wapt1 python[454]: Traceback (most recent call last):
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Aug 28 09:58:04 wapt1 python[454]:     self.receive(pkt)
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Aug 28 09:58:04 wapt1 python[454]:     self.send(packet.Packet(packet.PONG, pkt.data))
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Aug 28 09:58:04 wapt1 python[454]:     raise IOError('Socket is closed')
Aug 28 09:58:04 wapt1 python[454]: IOError: Socket is closed
Aug 28 09:58:04 wapt1 python[454]: 2018-08-28 09:58:04,587 ERROR Receive error
Aug 28 09:58:04 wapt1 python[454]: Traceback (most recent call last):
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Aug 28 09:58:04 wapt1 python[454]:     self.receive(pkt)
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Aug 28 09:58:04 wapt1 python[454]:     self.send(packet.Packet(packet.PONG, pkt.data))
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Aug 28 09:58:04 wapt1 python[454]:     raise IOError('Socket is closed')
Aug 28 09:58:04 wapt1 python[454]: IOError: Socket is closed
Aug 28 09:58:04 wapt1 python[454]: 2018-08-28 09:58:04,666 ERROR Receive error
Aug 28 09:58:04 wapt1 python[454]: Traceback (most recent call last):
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Aug 28 09:58:04 wapt1 python[454]:     self.receive(pkt)
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Aug 28 09:58:04 wapt1 python[454]:     self.send(packet.Packet(packet.PONG, pkt.data))
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Aug 28 09:58:04 wapt1 python[454]:     raise IOError('Socket is closed')
Aug 28 09:58:04 wapt1 python[454]: IOError: Socket is closed
Aug 28 09:58:04 wapt1 python[454]: 2018-08-28 09:58:04,702 ERROR Receive error
Aug 28 09:58:04 wapt1 python[454]: Traceback (most recent call last):
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Aug 28 09:58:04 wapt1 python[454]:     self.receive(pkt)
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Aug 28 09:58:04 wapt1 python[454]:     self.send(packet.Packet(packet.PONG, pkt.data))
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Aug 28 09:58:04 wapt1 python[454]:     raise IOError('Socket is closed')
Aug 28 09:58:04 wapt1 python[454]: IOError: Socket is closed
Aug 28 09:58:04 wapt1 python[454]: 2018-08-28 09:58:04,783 ERROR Receive error
Aug 28 09:58:04 wapt1 python[454]: Traceback (most recent call last):
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Aug 28 09:58:04 wapt1 python[454]:     self.receive(pkt)
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Aug 28 09:58:04 wapt1 python[454]:     self.send(packet.Packet(packet.PONG, pkt.data))
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Aug 28 09:58:04 wapt1 python[454]:     raise IOError('Socket is closed')
Aug 28 09:58:04 wapt1 python[454]: IOError: Socket is closed
Aug 28 09:58:05 wapt1 python[454]: 2018-08-28 09:58:05,285 ERROR Receive error
Aug 28 09:58:05 wapt1 python[454]: Traceback (most recent call last):
Aug 28 09:58:05 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Aug 28 09:58:05 wapt1 python[454]:     self.receive(pkt)
Aug 28 09:58:05 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Aug 28 09:58:05 wapt1 python[454]:     self.send(packet.Packet(packet.PONG, pkt.data))
Aug 28 09:58:05 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Aug 28 09:58:05 wapt1 python[454]:     raise IOError('Socket is closed')
Aug 28 09:58:05 wapt1 python[454]: IOError: Socket is closed
Aug 28 09:58:05 wapt1 peewee 2018-08-28 09:58:05,289 WARNING SocketIO connection refused for uuid 4C4C4544-0030-5310-8034-B4C04F433932, sid 4fc2093fb33f4abdbffef38ed03d2333: Data too old or in the futur age : 3487s...
Aug 28 09:58:05 wapt1 python[454]: 2018-08-28 09:58:05,289 WARNING SocketIO connection refused for uuid 4C4C4544-0030-5310-8034-B4C04F433932, sid 4fc2093fb33f4abdbffef38ed03d2333: Data too old or in the futur age : 3487s...
Aug 28 09:58:05 wapt1 python[454]: 2018-08-28 09:58:05,290 WARNING Application rejected connection
Aug 28 09:58:05 wapt1 python[454]: 2018-08-28 09:58:05,325 ERROR Receive error
Aug 28 09:58:05 wapt1 python[454]: Traceback (most recent call last):
Aug 28 09:58:05 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Aug 28 09:58:05 wapt1 python[454]:     self.receive(pkt)
Aug 28 09:58:05 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Aug 28 09:58:05 wapt1 python[454]:     self.send(packet.Packet(packet.PONG, pkt.data))
Aug 28 09:58:05 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Aug 28 09:58:05 wapt1 python[454]:     raise IOError('Socket is closed')
Aug 28 09:58:05 wapt1 python[454]: IOError: Socket is closed
Aug 28 09:58:05 wapt1 python[454]: 2018-08-28 09:58:05,431 ERROR Receive error
Aug 28 09:58:05 wapt1 python[454]: Traceback (most recent call last):
Aug 28 09:58:05 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Aug 28 09:58:05 wapt1 python[454]:     self.receive(pkt)
Aug 28 09:58:05 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Aug 28 09:58:05 wapt1 python[454]:     self.send(packet.Packet(packet.PONG, pkt.data))
Aug 28 09:58:05 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Aug 28 09:58:05 wapt1 python[454]:     raise IOError('Socket is closed')
Aug 28 09:58:05 wapt1 python[454]: IOError: Socket is closed
Last edited by Floflobel on Nov 19, 2018 - 10:36, edited 1 time.
High
Floflobel
Messages: 135
Registration: Oct 15, 2015 - 5:32 p.m.

September 18, 2018 - 2:26 PM

up?
High
User avatar
htouvet
WAPT Expert
Messages: 436
Registration: March 16, 2015 - 10:48
Contact :
Contact htouvet

September 18, 2018 - 3:20 PM

Hello,
there have been several fixes for WaptServer and updates to Python modules that may improve things in Wapt versions 1.5.1.26 and 1.6.2.

The WebSocket client and server Python modules are relatively new and still evolving.
Furthermore, the database connection pool management has undergone changes (peewee module, transaction management) to better reuse connections.

Firstly, if your server is running version 1.5.1.23, could you upgrade it to 1.5.1.26? The changes in version 1.5.1.26 were motivated by feedback from deployments at a client site with several large websites.

(Version 1.6.2.5 is available in the nightly builds : https://wapt.tranquil.it/wapt/nightly/w ... 601e0-rc2/)
Tranquil IT
High
Floflobel
Messages: 135
Registration: Oct 15, 2015 - 5:32 p.m.

September 18, 2018 - 4:24 PM

Hello htouvet,

Thank you for your feedback. I did upgrade to version 1.5.1.26 to try and resolve the bug some time ago. Unfortunately, the problem persists. The nightly version is not stable, and I wouldn't risk installing it on a production site.
I'll try cloning the server and see if the problem occurs on the clone, and then try updating.

Thank you for your feedback.
High
User avatar
dcardon
WAPT Expert
Messages: 1929
Registration: June 18, 2014 - 09:58
Location: Saint Sébastien sur Loire
Contact :
Contact dcardon

September 21, 2018 - 7:05 PM

Hello Floflobel,
Floflobel wrote: ↑Sep 18, 2018 - 4:24 PM Thank you for your feedback. I did upgrade to version 1.5.1.26 to try and resolve the bug some time ago. Unfortunately, the problem persists. The nightly version is not stable, and I wouldn't risk installing it on a production site.
I'll try cloning the server and see if the problem occurs on the clone, and then try updating.
Hubert is the person who knows the WAPT code best. If he suggests you try to reproduce your bug with version 1.6.2.5, there's a good reason. If you don't want to try to reproduce your bug, I'll close this thread for now. Version 1.6 should be released soon.

Sincerely,

Denis
Denis Cardon - Tranquil IT
Share your experiences on WAPT! Send us your blog and article URLs in the "Your Opinion of the forum, and we'll feature them on the WAPT
High
Floflobel
Messages: 135
Registration: Oct 15, 2015 - 5:32 p.m.

September 21, 2018 - 7:24 PM

Hello Denis,

I think I may have explained myself poorly. I'm going to test with this version to see if the bug recurs; unfortunately, I don't want to put it on my production site. So I'm going to clone this machine with the same data and see if the problem persists, then I'll try updating to the nightly version.

Regards,
High
Floflobel
Messages: 135
Registration: Oct 15, 2015 - 5:32 p.m.

October 22, 2018 - 5:23 PM

Hello,

The problem persists today. I've updated to version 1.6.2.7.
I'm waiting for another crash to provide you with more logs.

Regards,
High
Floflobel
Messages: 135
Registration: Oct 15, 2015 - 5:32 p.m.

November 20, 2018 - 3:10 PM

Good morning,

There are no logs that suggest an error. Can you guide me on how to resolve this problem, which is now present on several WAPT servers?

The problem is that the web interface is no longer accessible and only comes back intermittently. The service is still running.

At the time of the crash:

Code: Select all

Nov 20 13:02:54 wapt1 python[519]: 2018-11-20 13:02:54,669 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:04:56 wapt1 peewee 2018-11-20 13:04:56,970 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:04:56 wapt1 python[519]: 2018-11-20 13:04:56,970 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:06:57 wapt1 peewee 2018-11-20 13:06:57,386 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:06:57 wapt1 python[519]: 2018-11-20 13:06:57,386 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:08:57 wapt1 peewee 2018-11-20 13:08:57,690 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:08:57 wapt1 python[519]: 2018-11-20 13:08:57,690 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:10:58 wapt1 peewee 2018-11-20 13:10:58,016 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:10:58 wapt1 python[519]: 2018-11-20 13:10:58,016 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:15:00 wapt1 peewee 2018-11-20 13:15:00,665 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:15:00 wapt1 python[519]: 2018-11-20 13:15:00,665 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:15:29 wapt1 peewee 2018-11-20 13:15:29,709 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:15:29 wapt1 python[519]: 2018-11-20 13:15:29,709 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)


THANKS
High
Floflobel
Messages: 135
Registration: Oct 15, 2015 - 5:32 p.m.

December 10, 2018 - 5:06 PM

I'm bumping this thread; the problem persists and I don't have many leads...
WAPT Server version: 1.6.2.7
High
Locked

Return to "WAPT Server"


  • To go further with WAPT