# Meaning of "Disconnected while waiting for reply"

**URL:** <https://discourse.nameko.io/t/meaning-of-disconnected-while-waiting-for-reply/181>\
**Category:** googlegroup\
**Created:** [March 27, 2017, 9:17am UTC](https://discourse.nameko.io/t/meaning-of-disconnected-while-waiting-for-reply/181 "2017-03-27T09:17:15Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Richard](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@Richard](https://discourse.nameko.io/u/Richard)\
**Post date:** [March 27, 2017, 9:17am UTC](https://discourse.nameko.io/t/meaning-of-disconnected-while-waiting-for-reply/181/1 "2017-03-27T09:17:15Z")

</div>

I've just deployed a nameko service (http) which calls another nameko  
service over rpc and am getting this error randomly but quite often.

RpcConnectionError("Disconnected while waiting for reply")

While this error is happening I can see from the rabbitmq management  
interface that messages are being processed.

Here's the stack-trace

RpcConnectionError: Disconnected while waiting for reply  
&nbsp;&nbsp;File "nameko/containers.py", line 405, in \_run\_worker  
&nbsp;&nbsp;&nbsp;&nbsp;result = method(\*worker\_ctx.args, \*\*worker\_ctx.kwargs)  
&nbsp;&nbsp;File "newrelic/api/background\_task.py", line 102, in wrapper  
&nbsp;&nbsp;&nbsp;&nbsp;return wrapped(\*args, \*\*kwargs)  
&nbsp;&nbsp;File "./app/helpers/monitoring.py", line 18, in wrapper  
&nbsp;&nbsp;&nbsp;&nbsp;return func(\*args, \*\*kwargs)  
&nbsp;&nbsp;File "./app/helpers/decorators.py", line 16, in wrapper  
&nbsp;&nbsp;&nbsp;&nbsp;response = func(\*args, \*\*kwargs)  
&nbsp;&nbsp;File "./app/service.py", line 65, in process\_event  
&nbsp;&nbsp;&nbsp;&nbsp;self.handle\_process\_elements(event\_data=event\_data)  
&nbsp;&nbsp;File "./app/service.py", line 77, in handle\_process\_elements  
&nbsp;&nbsp;&nbsp;&nbsp;user\_id=event\_data['user\_id']  
&nbsp;&nbsp;File "./app/components/handlers/thing\_handler.py", line 22, in  
handle\_selected\_elements  
&nbsp;&nbsp;&nbsp;&nbsp;user\_id=user\_id)  
&nbsp;&nbsp;File "nameko/rpc.py", line 395, in \_\_call\_\_  
&nbsp;&nbsp;&nbsp;&nbsp;return reply.result()  
&nbsp;&nbsp;File "nameko/rpc.py", line 376, in result  
&nbsp;&nbsp;&nbsp;&nbsp;self.resp\_body = self.reply\_event.wait()  
&nbsp;&nbsp;File "eventlet/event.py", line 121, in wait  
&nbsp;&nbsp;&nbsp;&nbsp;return hubs.get\_hub().switch()  
&nbsp;&nbsp;File "eventlet/hubs/hub.py", line 294, in switch  
&nbsp;&nbsp;&nbsp;&nbsp;return self.greenlet.switch()

What could cause this to occur? And why does the that code in nameko raise  
an error on re-connection?

> <https://github.com/nameko/nameko/blob/master/nameko/rpc.py#L328>

I'm fairly new to working with these services so could do with an  
explanation of that part of the code.

The call it's failing on is fairly simple, however if I ssh into the  
machine and run it manually I'm unable to reproduce the connection error.

stuff = self.stuff\_rpc.get\_stuff(user=123)

Thanks in advance

---

<div class="post-metadata">

**Author:** ![Richard](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@Richard](https://discourse.nameko.io/u/Richard)\
**Post date:** [March 27, 2017, 1:29pm UTC](https://discourse.nameko.io/t/meaning-of-disconnected-while-waiting-for-reply/181/2 "2017-03-27T13:29:37Z")

</div>

Looks like the issue was haproxy being in front of rabbitmq. Found this  
fix [https://deviantony.wordpress.com/2014/10/30/rabbitmq-and-haproxy-a-timeout-issue/](https://deviantony.wordpress.com/2014/10/30/rabbitmq-and-haproxy-a-timeout-issue/)

---

<div class="post-metadata">

**Author:** ![David\_Szotten](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/david_szotten/32/6_2.png) [@David\_Szotten](https://discourse.nameko.io/u/David_Szotten)\
**Post date:** [March 27, 2017, 1:38pm UTC](https://discourse.nameko.io/t/meaning-of-disconnected-while-waiting-for-reply/181/3 "2017-03-27T13:38:14Z")

</div>

Hi Richard,

Glad you found the cause of the disconnection. Regarding the exception  
raised in the code you linked to, the explanation is in the comment just  
above: Due to the disconnection, the reply queue may have been deleted, so  
it's \_possible\_ that unread replies have been lost. If that's the case, the  
client would wait forever for a reply to arrive. To avoid this, we  
"invalidate" all pending replies.

Best,  
David

> **···**
>
> On Monday, 27 March 2017 14:29:38 UTC+1, richard...@babylonhealth.com wrote:
> 
> > Looks like the issue was haproxy being in front of rabbitmq. Found this  
> > fix  
> > [RabbitMQ and HAProxy: a timeout issue – Tony's press](https://deviantony.wordpress.com/2014/10/30/rabbitmq-and-haproxy-a-timeout-issue/)

---

<div class="post-metadata">

**Author:** ![mattbennett](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/mattbennett/32/13_2.png) [@mattbennett](https://discourse.nameko.io/u/mattbennett)\
**Post date:** [March 27, 2017, 1:39pm UTC](https://discourse.nameko.io/t/meaning-of-disconnected-while-waiting-for-reply/181/4 "2017-03-27T13:39:17Z")

</div>

This happens when your service was disconnected from the rabbit broker  
while there is a pending RPC reply. It's raised in on\_consume\_ready (when  
the connection is reestablished) because that's the first available chance  
we have to detect the disconnection.

RPC reply queues are exclusive and auto-delete — meaning that the queue and  
only be consumed from one connection (the one which declared it), and when  
that consumer disconnects the queue is removed.

Disconnection and reconnection means that by definition, the original reply  
queue (the one to which the result will be sent) won’t exist anymore. The  
client has no choice but to retry the request.

So yes, having HAProxy yank connections will definitely cause this. If  
you're using a load-balancer I highly recommend you use nameko 2.4.4 or  
greater, since this has support for consumer heartbeats. The heartbeat will  
keep the connections active so your load-balancer doesn't kill them. It'll  
also mean that you detect genuine disconnections immediately, rather than  
waiting 2 hours (!) for a TCP timeout.

I saw you also replied to  
[ResourceLocked: Queue.declare: (405) RESOURCE\_LOCKED - cannot obtain exclusive access to locked queue 'rpc.reply-standalone\_rpc\_proxy' ... · Issue #359 · nameko/nameko · GitHub](https://github.com/nameko/nameko/issues/359#issuecomment-289437640), which  
is related. On reflection, I think what’s happening there is just the  
RpcProxy always tries to reconnect with the same queue. If RabbitMQ hasn’t  
auto-deleted the old one yet, you get a clash. In the standalone case we  
don’t catch it and so it bubbles out.

I’m not 100% sure there’s not a better possible implementation for the  
RpcProxy. It would be nice if RabbitMQ let you reconnect to an exclusive  
queue if you were the original consumer. If that were the case then we  
could survive small outages (just as long as the queue wasn’t auto-deleted  
in the mean time).

> **···**
>
> On Monday, March 27, 2017 at 2:29:38 PM UTC+1, richard...@babylonhealth.com wrote:
> 
> > Looks like the issue was haproxy being in front of rabbitmq. Found this  
> > fix  
> > [RabbitMQ and HAProxy: a timeout issue – Tony's press](https://deviantony.wordpress.com/2014/10/30/rabbitmq-and-haproxy-a-timeout-issue/)

---

<div class="post-metadata">

**Author:** ![David\_Szotten](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/david_szotten/32/6_2.png) [@David\_Szotten](https://discourse.nameko.io/u/David_Szotten)\
**Post date:** [March 27, 2017, 1:46pm UTC](https://discourse.nameko.io/t/meaning-of-disconnected-while-waiting-for-reply/181/5 "2017-03-27T13:46:48Z")

</div>

my understanding is that exclusive queues are owned by the \_connection\_, so  
by definition if you get disconnected and come back, you are a new  
connection and can't get back at it.

the alternative would we to use a different mechanism, though i'd worry  
that would require some manual mechanism for garbage collection of reply  
queues

d

> **···**
>
> On Monday, 27 March 2017 14:39:17 UTC+1, Matt Yule-Bennett wrote:
> 
> > This happens when your service was disconnected from the rabbit broker  
> > while there is a pending RPC reply. It's raised in on\_consume\_ready (when  
> > the connection is reestablished) because that's the first available chance  
> > we have to detect the disconnection.
> > 
> > RPC reply queues are exclusive and auto-delete — meaning that the queue  
> > and only be consumed from one connection (the one which declared it), and  
> > when that consumer disconnects the queue is removed.
> > 
> > Disconnection and reconnection means that by definition, the original  
> > reply queue (the one to which the result will be sent) won’t exist anymore.  
> > The client has no choice but to retry the request.
> > 
> > So yes, having HAProxy yank connections will definitely cause this. If  
> > you're using a load-balancer I highly recommend you use nameko 2.4.4 or  
> > greater, since this has support for consumer heartbeats. The heartbeat will  
> > keep the connections active so your load-balancer doesn't kill them. It'll  
> > also mean that you detect genuine disconnections immediately, rather than  
> > waiting 2 hours (!) for a TCP timeout.
> > 
> > I saw you also replied to  
> > [ResourceLocked: Queue.declare: (405) RESOURCE\_LOCKED - cannot obtain exclusive access to locked queue 'rpc.reply-standalone\_rpc\_proxy' ... · Issue #359 · nameko/nameko · GitHub](https://github.com/nameko/nameko/issues/359#issuecomment-289437640), which  
> > is related. On reflection, I think what’s happening there is just the  
> > RpcProxy always tries to reconnect with the same queue. If RabbitMQ hasn’t  
> > auto-deleted the old one yet, you get a clash. In the standalone case we  
> > don’t catch it and so it bubbles out.
> > 
> > I’m not 100% sure there’s not a better possible implementation for the  
> > RpcProxy. It would be nice if RabbitMQ let you reconnect to an exclusive  
> > queue if you were the original consumer. If that were the case then we  
> > could survive small outages (just as long as the queue wasn’t auto-deleted  
> > in the mean time).
> > 
> > On Monday, March 27, 2017 at 2:29:38 PM UTC+1, \> richard...@babylonhealth.com wrote:
> > 
> > > Looks like the issue was haproxy being in front of rabbitmq. Found this  
> > > fix  
> > > [RabbitMQ and HAProxy: a timeout issue – Tony's press](https://deviantony.wordpress.com/2014/10/30/rabbitmq-and-haproxy-a-timeout-issue/)