Tranquil IT Forum

Published: **June 20, 2018 - 2:56 PM**

Good morning,

I own several WAPT servers, but at the moment one of them is going into error quite regularly.
A restart of the service and it works again.

Here is the error:

Code: Select all

Jun 20 10:03:20 wapt1 python[20624]: 2018-06-20 10:03:20,140 ERROR Receive error
Jun 20 10:03:20 wapt1 python[20624]: Traceback (most recent call last):
Jun 20 10:03:20 wapt1 python[20624]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Jun 20 10:03:20 wapt1 python[20624]:     self.receive(pkt)
Jun 20 10:03:20 wapt1 python[20624]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Jun 20 10:03:20 wapt1 python[20624]:     self.send(packet.Packet(packet.PONG, pkt.data))
Jun 20 10:03:20 wapt1 python[20624]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Jun 20 10:03:20 wapt1 python[20624]:     raise IOError('Socket is closed')
Jun 20 10:03:20 wapt1 python[20624]: IOError: Socket is closed
Jun 20 10:03:20 wapt1 python[20624]: 2018-06-20 10:03:20,217 ERROR Receive error
Jun 20 10:03:20 wapt1 python[20624]: Traceback (most recent call last):
Jun 20 10:03:20 wapt1 python[20624]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Jun 20 10:03:20 wapt1 python[20624]:     self.receive(pkt)
Jun 20 10:03:20 wapt1 python[20624]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Jun 20 10:03:20 wapt1 python[20624]:     self.send(packet.Packet(packet.PONG, pkt.data))
Jun 20 10:03:20 wapt1 python[20624]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Jun 20 10:03:20 wapt1 python[20624]:     raise IOError('Socket is closed')
Jun 20 10:03:20 wapt1 python[20624]: IOError: Socket is closed
Jun 20 10:03:20 wapt1 python[20624]: 2018-06-20 10:03:20,283 ERROR Receive error
Jun 20 10:03:20 wapt1 python[20624]: Traceback (most recent call last):
Jun 20 10:03:20 wapt1 python[20624]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Jun 20 10:03:20 wapt1 python[20624]:     self.receive(pkt)
Jun 20 10:03:20 wapt1 python[20624]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Jun 20 10:03:20 wapt1 python[20624]:     self.send(packet.Packet(packet.PONG, pkt.data))
Jun 20 10:03:20 wapt1 python[20624]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Jun 20 10:03:20 wapt1 python[20624]:     raise IOError('Socket is closed')
Jun 20 10:03:20 wapt1 python[20624]: IOError: Socket is closed

Debian GNU/Linux 9 (stretch)
WAPT Server version: 1.5.1.23
WAPT Agent version: 1.5.1.23
WAPT Setup version: 1.5.1.23
WAPT Deploy version: 1.5.1.23
DB status: OK (1.5.1.22)

Published: **August 30, 2018 - 11:54**

I increased the number of connections to the postgres database and it crashes less often, but it's still the case.
Any ideas?

Code: Select all

Aug 28 09:58:04 wapt1 python[454]: 2018-08-28 09:58:04,419 ERROR Receive error
Aug 28 09:58:04 wapt1 python[454]: Traceback (most recent call last):
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Aug 28 09:58:04 wapt1 python[454]:     self.receive(pkt)
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Aug 28 09:58:04 wapt1 python[454]:     self.send(packet.Packet(packet.PONG, pkt.data))
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Aug 28 09:58:04 wapt1 python[454]:     raise IOError('Socket is closed')
Aug 28 09:58:04 wapt1 python[454]: IOError: Socket is closed
Aug 28 09:58:04 wapt1 python[454]: 2018-08-28 09:58:04,501 ERROR Receive error
Aug 28 09:58:04 wapt1 python[454]: Traceback (most recent call last):
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Aug 28 09:58:04 wapt1 python[454]:     self.receive(pkt)
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Aug 28 09:58:04 wapt1 python[454]:     self.send(packet.Packet(packet.PONG, pkt.data))
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Aug 28 09:58:04 wapt1 python[454]:     raise IOError('Socket is closed')
Aug 28 09:58:04 wapt1 python[454]: IOError: Socket is closed
Aug 28 09:58:04 wapt1 python[454]: 2018-08-28 09:58:04,587 ERROR Receive error
Aug 28 09:58:04 wapt1 python[454]: Traceback (most recent call last):
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Aug 28 09:58:04 wapt1 python[454]:     self.receive(pkt)
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Aug 28 09:58:04 wapt1 python[454]:     self.send(packet.Packet(packet.PONG, pkt.data))
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Aug 28 09:58:04 wapt1 python[454]:     raise IOError('Socket is closed')
Aug 28 09:58:04 wapt1 python[454]: IOError: Socket is closed
Aug 28 09:58:04 wapt1 python[454]: 2018-08-28 09:58:04,666 ERROR Receive error
Aug 28 09:58:04 wapt1 python[454]: Traceback (most recent call last):
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Aug 28 09:58:04 wapt1 python[454]:     self.receive(pkt)
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Aug 28 09:58:04 wapt1 python[454]:     self.send(packet.Packet(packet.PONG, pkt.data))
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Aug 28 09:58:04 wapt1 python[454]:     raise IOError('Socket is closed')
Aug 28 09:58:04 wapt1 python[454]: IOError: Socket is closed
Aug 28 09:58:04 wapt1 python[454]: 2018-08-28 09:58:04,702 ERROR Receive error
Aug 28 09:58:04 wapt1 python[454]: Traceback (most recent call last):
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Aug 28 09:58:04 wapt1 python[454]:     self.receive(pkt)
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Aug 28 09:58:04 wapt1 python[454]:     self.send(packet.Packet(packet.PONG, pkt.data))
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Aug 28 09:58:04 wapt1 python[454]:     raise IOError('Socket is closed')
Aug 28 09:58:04 wapt1 python[454]: IOError: Socket is closed
Aug 28 09:58:04 wapt1 python[454]: 2018-08-28 09:58:04,783 ERROR Receive error
Aug 28 09:58:04 wapt1 python[454]: Traceback (most recent call last):
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Aug 28 09:58:04 wapt1 python[454]:     self.receive(pkt)
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Aug 28 09:58:04 wapt1 python[454]:     self.send(packet.Packet(packet.PONG, pkt.data))
Aug 28 09:58:04 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Aug 28 09:58:04 wapt1 python[454]:     raise IOError('Socket is closed')
Aug 28 09:58:04 wapt1 python[454]: IOError: Socket is closed
Aug 28 09:58:05 wapt1 python[454]: 2018-08-28 09:58:05,285 ERROR Receive error
Aug 28 09:58:05 wapt1 python[454]: Traceback (most recent call last):
Aug 28 09:58:05 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Aug 28 09:58:05 wapt1 python[454]:     self.receive(pkt)
Aug 28 09:58:05 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Aug 28 09:58:05 wapt1 python[454]:     self.send(packet.Packet(packet.PONG, pkt.data))
Aug 28 09:58:05 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Aug 28 09:58:05 wapt1 python[454]:     raise IOError('Socket is closed')
Aug 28 09:58:05 wapt1 python[454]: IOError: Socket is closed
Aug 28 09:58:05 wapt1 peewee 2018-08-28 09:58:05,289 WARNING SocketIO connection refused for uuid 4C4C4544-0030-5310-8034-B4C04F433932, sid 4fc2093fb33f4abdbffef38ed03d2333: Data too old or in the futur age : 3487s...
Aug 28 09:58:05 wapt1 python[454]: 2018-08-28 09:58:05,289 WARNING SocketIO connection refused for uuid 4C4C4544-0030-5310-8034-B4C04F433932, sid 4fc2093fb33f4abdbffef38ed03d2333: Data too old or in the futur age : 3487s...
Aug 28 09:58:05 wapt1 python[454]: 2018-08-28 09:58:05,290 WARNING Application rejected connection
Aug 28 09:58:05 wapt1 python[454]: 2018-08-28 09:58:05,325 ERROR Receive error
Aug 28 09:58:05 wapt1 python[454]: Traceback (most recent call last):
Aug 28 09:58:05 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Aug 28 09:58:05 wapt1 python[454]:     self.receive(pkt)
Aug 28 09:58:05 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Aug 28 09:58:05 wapt1 python[454]:     self.send(packet.Packet(packet.PONG, pkt.data))
Aug 28 09:58:05 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Aug 28 09:58:05 wapt1 python[454]:     raise IOError('Socket is closed')
Aug 28 09:58:05 wapt1 python[454]: IOError: Socket is closed
Aug 28 09:58:05 wapt1 python[454]: 2018-08-28 09:58:05,431 ERROR Receive error
Aug 28 09:58:05 wapt1 python[454]: Traceback (most recent call last):
Aug 28 09:58:05 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 210, in _websocket_handler
Aug 28 09:58:05 wapt1 python[454]:     self.receive(pkt)
Aug 28 09:58:05 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 54, in receive
Aug 28 09:58:05 wapt1 python[454]:     self.send(packet.Packet(packet.PONG, pkt.data))
Aug 28 09:58:05 wapt1 python[454]:   File "/opt/wapt/lib/python2.7/site-packages/engineio/socket.py", line 68, in send
Aug 28 09:58:05 wapt1 python[454]:     raise IOError('Socket is closed')
Aug 28 09:58:05 wapt1 python[454]: IOError: Socket is closed

Published: **September 18, 2018 - 2:26 PM**

Published: **September 18, 2018 - 3:20 PM**

Hello,
there have been several fixes for WaptServer and updates to Python modules that may improve things in Wapt versions 1.5.1.26 and 1.6.2.

The WebSocket client and server Python modules are relatively new and still evolving.
Furthermore, the database connection pool management has undergone changes (peewee module, transaction management) to better reuse connections.

Firstly, if your server is running version 1.5.1.23, could you upgrade it to 1.5.1.26? The changes in version 1.5.1.26 were motivated by feedback from deployments at a client site with several large websites.

(Version 1.6.2.5 is available in the nightly builds : https://wapt.tranquil.it/wapt/nightly/w ... 601e0-rc2/)

Published: **September 18, 2018 - 4:24 PM**

Hello htouvet,

Thank you for your feedback. I did upgrade to version 1.5.1.26 to try and resolve the bug some time ago. Unfortunately, the problem persists. The nightly version is not stable, and I wouldn't risk installing it on a production site.
I'll try cloning the server and see if the problem occurs on the clone, and then try updating.

Thank you for your feedback.

Published: **September 21, 2018 - 7:05 PM**

Hello Floflobel,

Floflobel wrote: ↑Sep 18, 2018 - 4:24 PM Thank you for your feedback. I did upgrade to version 1.5.1.26 to try and resolve the bug some time ago. Unfortunately, the problem persists. The nightly version is not stable, and I wouldn't risk installing it on a production site.
I'll try cloning the server and see if the problem occurs on the clone, and then try updating.

Hubert is the person who knows the WAPT code best. If he suggests you try to reproduce your bug with version 1.6.2.5, there's a good reason. If you don't want to try to reproduce your bug, I'll close this thread for now. Version 1.6 should be released soon.

Sincerely,

Denis

Published: **September 21, 2018 - 7:24 PM**

Hello Denis,

I think I may have explained myself poorly. I'm going to test with this version to see if the bug recurs; unfortunately, I don't want to put it on my production site. So I'm going to clone this machine with the same data and see if the problem persists, then I'll try updating to the nightly version.

Regards,

Published: **October 22, 2018 - 5:23 PM**

Hello,

The problem persists today. I've updated to version 1.6.2.7.
I'm waiting for another crash to provide you with more logs.

Regards,

Published: **November 20, 2018 - 3:10 PM**

Good morning,

There are no logs that suggest an error. Can you guide me on how to resolve this problem, which is now present on several WAPT servers?

The problem is that the web interface is no longer accessible and only comes back intermittently. The service is still running.

At the time of the crash:

Code: Select all

Nov 20 13:02:54 wapt1 python[519]: 2018-11-20 13:02:54,669 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:04:56 wapt1 peewee 2018-11-20 13:04:56,970 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:04:56 wapt1 python[519]: 2018-11-20 13:04:56,970 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:06:57 wapt1 peewee 2018-11-20 13:06:57,386 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:06:57 wapt1 python[519]: 2018-11-20 13:06:57,386 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:08:57 wapt1 peewee 2018-11-20 13:08:57,690 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:08:57 wapt1 python[519]: 2018-11-20 13:08:57,690 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:10:58 wapt1 peewee 2018-11-20 13:10:58,016 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:10:58 wapt1 python[519]: 2018-11-20 13:10:58,016 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:15:00 wapt1 peewee 2018-11-20 13:15:00,665 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:15:00 wapt1 python[519]: 2018-11-20 13:15:00,665 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:15:29 wapt1 peewee 2018-11-20 13:15:29,709 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)
Nov 20 13:15:29 wapt1 python[519]: 2018-11-20 13:15:29,709 CRITICAL Get_websocket_auth_token failed EWaptAuthenticationFailure(u'Unknown host UUID ec7654f1-5a25-xxxx-xxxx-xxxxxxxxxx. Please register first.',)

THANKS

Published: **Dec 10, 2018 - 5:06 PM**

I'm bumping this thread; the problem persists and I don't have many leads...
WAPT Server version: 1.6.2.7

Tranquil IT Forum

WAPT server has stopped working

WAPT server has stopped working

Re: WAPT server stops working

Re: WAPT server stops working

Re: WAPT server stops working

Re: WAPT server stops working

Re: WAPT server stops working

Re: [RESOLVED] WAPT server stops working

Re: [RESOLVED] WAPT server stops working

Re: [RESOLVED] WAPT server stops working

Re: WAPT server stops working