[WinError 10054] An existing connection was forcibly closed by the remote host

1

I am getting this error on Windows Server class VM with nameko version 2.12.0. Kombu Version 4.6.5.

[2019-10-15 19:27:48,608] [INFO] [nameko.containers] [containers.py] [kill():282] [PID:10996 TID:1401349262832] - killing <ServiceContainer [wind_api_data_service] at 0x14601b699e8> due to [WinError 10054] An existing connection was forcibly closed by the remote host
[2019-10-15 19:27:48,611] [WARNING] [kombu.mixins] [mixins.py] [run():180] [PID:10996 TID:1400188214008] - Connection to broker lost, trying to re-establish connection…
Traceback (most recent call last):
File “E:\QF\inv_qf5\App\wind_services\venv\lib\site-packages\kombu\mixins.py”, line 175, in run
for _ in self.consume(limit=None, **kwargs):
File “E:\QF\inv_qf5\App\wind_services\venv\lib\site-packages\kombu\mixins.py”, line 197, in consume
conn.drain_events(timeout=safety_interval)
File “E:\QF\inv_qf5\App\wind_services\venv\lib\site-packages\kombu\connection.py”, line 323, in drain_events
return self.transport.drain_events(self.connection, **kwargs)
File “E:\QF\inv_qf5\App\wind_services\venv\lib\site-packages\kombu\transport\pyamqp.py”, line 103, in drain_events
return connection.drain_events(**kwargs)
File “E:\QF\inv_qf5\App\wind_services\venv\lib\site-packages\amqp\connection.py”, line 505, in drain_events
while not self.blocking_read(timeout):
File “E:\QF\inv_qf5\App\wind_services\venv\lib\site-packages\amqp\connection.py”, line 510, in blocking_read
frame = self.transport.read_frame()
File “E:\QF\inv_qf5\App\wind_services\venv\lib\site-packages\amqp\transport.py”, line 252, in read_frame
frame_header = read(7, True)
File “E:\QF\inv_qf5\App\wind_services\venv\lib\site-packages\amqp\transport.py”, line 446, in _read
raise IOError(‘Server unexpectedly closed connection’)
OSError: Server unexpectedly closed connection

2

Hi @manish.leonine,

Your service is losing connection to the RabbitMQ broker. It should attempt to reconnect, as stated in the log. Does it not succeed?

How are you running and connecting to the broker?

3

Hi Mat,

It connects in some attempts but fails for others. Any thing specific on the heartbeat and timeout that can be tweaked to make the reconnect more effective?

4

Attempting to reconnect is the intended behaviour. There’s nothing Nameko can do if the server can’t be reached though. It sounds like you’ve got something going on with the network connection to RabbitMQ. Is this on the same machine, LAN or WAN?

You can reduce the heartbeat to discover a broken connection sooner – simply set HEARTBEAT: <value-in-seconds> in your config file. But this will not make reconnection more effective I’m afraid.