Client Time Outs

We've been experiencing issues with Nameko 2.1.2 when clients time out.
When this happens, and messages subsequently arrive for them, these
messages sit in the reply queue "unackable" as correlation IDs now will
never be matched. Trying to get a look at these messages is not easy
because they are now in a locked state, e.g.:

*405 RESOURCE_LOCKED - cannot obtain exclusive access to locked queue
'rpc.reply-standalone_rpc_proxy-e3703829-499e-484d-9ca7-cdc7fb3c8355' in
vhost 'myvhost' *

We solved this by restarting the client applications which threw away the
stacked up reply queues and created new ones.

Can we handle this better in the framework?

We've been experiencing issues with Nameko 2.1.2 when clients time out.
When this happens, and messages subsequently arrive for them, these
messages sit in the reply queue "unackable" as correlation IDs now will
never be matched.

I think this is a small omission in the handling of RPC reply messages by
the standalone proxy -- we should ``reject`` the message instead of leaving
it unacked when the correlation ID doesn't match what we expected.

Trying to get a look at these messages is not easy because they are now in
a locked state, e.g.:

*405 RESOURCE_LOCKED - cannot obtain exclusive access to locked queue
'rpc.reply-standalone_rpc_proxy-e3703829-499e-484d-9ca7-cdc7fb3c8355' in
vhost 'myvhost' *

You're actually seeing this message because reply queues are configured as
exclusive, so you can't read them from any connection other than the one
that created it. But you can't read an unacked message from a queue via the
rabbit interface anyway, even non-exclusive ones.

We solved this by restarting the client applications which threw away the
stacked up reply queues and created new ones.

Yep, restarting the client will delete the queue and its contents because
reply queues are configured as exclusive and auto-delete (i.e. it gets
removed when the last and in this case only consumer goes away)

Can we handle this better in the framework?

We should fix the bug :wink: I can only reproduce a single message being
unacked per consumer, did you ever see more than that? Can you share your
client code? Via email if you'd rather not post it here.

ยทยทยท

On Thursday, June 25, 2015 at 12:11:41 PM UTC+1, simon harrison wrote: