# RabbitMQ Under Load

**URL:** <https://discourse.nameko.io/t/rabbitmq-under-load/235>\
**Category:** googlegroup\
**Created:** [February 1, 2018, 10:43am UTC](https://discourse.nameko.io/t/rabbitmq-under-load/235 "2018-02-01T10:43:37Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![simon\_harrison](https://avatars.discourse-cdn.com/v4/letter/s/b5ac83/32.png) [@simon\_harrison](https://discourse.nameko.io/u/simon_harrison)\
**Post date:** [February 1, 2018, 10:43am UTC](https://discourse.nameko.io/t/rabbitmq-under-load/235/1 "2018-02-01T10:43:37Z")

</div>

Hi there

We're continuing to have trouble with our RabbitMQ setup. Under load things  
break down quickly. And i use the term "load" very generously here!

Sentry is flooded with alerts such as: OS Error Socket Closed, Connection  
Forced, Killing n managed threads, RecoverableConnectionError, killing  
managed thread "run", killing 1 active worker, etc. etc.

We are using Amazons load balancer with 3 rabbit instances behind it.

All we are doing is pub-sub between 2 services, and this is very light  
message publishing.

Our heartbeat is set to 5 seconds.

Can anyone advise or suggest somewhere we can get hands-on help with our  
rabbit setup?

Thanks

---

<div class="post-metadata">

**Author:** ![mattbennett](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/mattbennett/32/13_2.png) [@mattbennett](https://discourse.nameko.io/u/mattbennett)\
**Post date:** [February 1, 2018, 11:08am UTC](https://discourse.nameko.io/t/rabbitmq-under-load/235/2 "2018-02-01T11:08:46Z")

</div>

With a 5 second heartbeat this is almost certainly the same problem as  
reported  
here: [Redirecting to Google Groups](https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/nameko-dev/Hmz659s4dBY/bqfmZtS9AwAJ)

From that thread:

> If the workload inside the event handler isn't doing any I/O it is likely

to be stealing the CPU long enough to starve the thread that's sending your  
heartbeats. Once two heartbeats are missed the broker will close the  
connection from the other end. You'll be able to see this happening if you  
check your rabbit logs.

Unfortunately Nameko handles the error very ungracefully. It would be much  
better to catch it and try to reconnect rather than letting it bubble and  
kill the container, but fixing that won't solve what I suspect is your  
underlying issue: tight CPU loops will starve other greenthreads. This is a  
inherent limitation of implicitly yielding co-routines. There is a simple  
fix though -- offload non-yielding code to a (native) threadpool, or insert  
some explicit yields. Another quick fix would be to decrease the heartbeat  
frequency.

Please check your rabbit broker logs for "heartbeat missed" messages to  
confirm this is the source of your problem. You could also take a look at  
the amount of time you take to service each request -- you'll see some that  
are longer than 2x the heartbeat if this is your issue.

> **···**
>
> On Thursday, 1 February 2018 10:43:37 UTC, simon harrison wrote:
> 
> > Hi there
> > 
> > We're continuing to have trouble with our RabbitMQ setup. Under load  
> > things break down quickly. And i use the term "load" very generously here!
> > 
> > Sentry is flooded with alerts such as: OS Error Socket Closed, Connection  
> > Forced, Killing n managed threads, RecoverableConnectionError, killing  
> > managed thread "run", killing 1 active worker, etc. etc.
> > 
> > We are using Amazons load balancer with 3 rabbit instances behind it.
> > 
> > All we are doing is pub-sub between 2 services, and this is very light  
> > message publishing.
> > 
> > Our heartbeat is set to 5 seconds.
> > 
> > Can anyone advise or suggest somewhere we can get hands-on help with our  
> > rabbit setup?
> > 
> > Thanks

---

<div class="post-metadata">

**Author:** ![simon\_harrison](https://avatars.discourse-cdn.com/v4/letter/s/b5ac83/32.png) [@simon\_harrison](https://discourse.nameko.io/u/simon_harrison)\
**Post date:** [February 1, 2018, 11:29am UTC](https://discourse.nameko.io/t/rabbitmq-under-load/235/3 "2018-02-01T11:29:20Z")

</div>

Thanks Matt, I appreciate your very quick response. and I've passed that  
thread on.

All our handlers typically either write to postgres or redis.

What heartbeat do you use/recommend?

> **···**
>
> On Thursday, 1 February 2018 11:08:47 UTC, Matt Yule-Bennett wrote:
> 
> > With a 5 second heartbeat this is almost certainly the same problem as  
> > reported here:  
> > [Redirecting to Google Groups](https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/nameko-dev/Hmz659s4dBY/bqfmZtS9AwAJ)
> > 
> > From that thread:
> > 
> > \> If the workload inside the event handler isn't doing any I/O it is  
> > likely to be stealing the CPU long enough to starve the thread that's  
> > sending your heartbeats. Once two heartbeats are missed the broker will  
> > close the connection from the other end. You'll be able to see this  
> > happening if you check your rabbit logs.
> > 
> > Unfortunately Nameko handles the error very ungracefully. It would be much  
> > better to catch it and try to reconnect rather than letting it bubble and  
> > kill the container, but fixing that won't solve what I suspect is your  
> > underlying issue: tight CPU loops will starve other greenthreads. This is a  
> > inherent limitation of implicitly yielding co-routines. There is a simple  
> > fix though -- offload non-yielding code to a (native) threadpool, or insert  
> > some explicit yields. Another quick fix would be to decrease the heartbeat  
> > frequency.
> > 
> > Please check your rabbit broker logs for "heartbeat missed" messages to  
> > confirm this is the source of your problem. You could also take a look at  
> > the amount of time you take to service each request -- you'll see some that  
> > are longer than 2x the heartbeat if this is your issue.
> > 
> > On Thursday, 1 February 2018 10:43:37 UTC, simon harrison wrote:
> > 
> > > Hi there
> > > 
> > > We're continuing to have trouble with our RabbitMQ setup. Under load  
> > > things break down quickly. And i use the term "load" very generously here!
> > > 
> > > Sentry is flooded with alerts such as: OS Error Socket Closed, Connection  
> > > Forced, Killing n managed threads, RecoverableConnectionError, killing  
> > > managed thread "run", killing 1 active worker, etc. etc.
> > > 
> > > We are using Amazons load balancer with 3 rabbit instances behind it.
> > > 
> > > All we are doing is pub-sub between 2 services, and this is very light  
> > > message publishing.
> > > 
> > > Our heartbeat is set to 5 seconds.
> > > 
> > > Can anyone advise or suggest somewhere we can get hands-on help with our  
> > > rabbit setup?
> > > 
> > > Thanks

---

<div class="post-metadata">

**Author:** ![mattbennett](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/mattbennett/32/13_2.png) [@mattbennett](https://discourse.nameko.io/u/mattbennett)\
**Post date:** [February 2, 2018, 10:22am UTC](https://discourse.nameko.io/t/rabbitmq-under-load/235/4 "2018-02-02T10:22:14Z")

</div>

> Thanks Matt, I appreciate your very quick response. and I've passed that thread on.
> 
> All our handlers typically either write to postgres or redis.

It might be just one handler causing the problem. You only need one thread to hold the cpu for longer than 2x the heartbeat

> What heartbeat do you use/recommend?

We use the nameko default of 60 seconds

> **···**
>
> On Thursday, February 1, 2018 at 11:29:21 AM UTC, simon harrison wrote:
> 
> > On Thursday, 1 February 2018 11:08:47 UTC, Matt Yule-Bennett wrote:
> > 
> > With a 5 second heartbeat this is almost certainly the same problem as reported here: [Redirecting to Google Groups](https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/nameko-dev/Hmz659s4dBY/bqfmZtS9AwAJ)
> > 
> > From that thread:
> > 
> > \> If the workload inside the event handler isn't doing any I/O it is likely to be stealing the CPU long enough to starve the thread that's sending your heartbeats. Once two heartbeats are missed the broker will close the connection from the other end. You'll be able to see this happening if you check your rabbit logs.
> > 
> > Unfortunately Nameko handles the error very ungracefully. It would be much better to catch it and try to reconnect rather than letting it bubble and kill the container, but fixing that won't solve what I suspect is your underlying issue: tight CPU loops will starve other greenthreads. This is a inherent limitation of implicitly yielding co-routines. There is a simple fix though -- offload non-yielding code to a (native) threadpool, or insert some explicit yields. Another quick fix would be to decrease the heartbeat frequency.
> > 
> > Please check your rabbit broker logs for "heartbeat missed" messages to confirm this is the source of your problem. You could also take a look at the amount of time you take to service each request -- you'll see some that are longer than 2x the heartbeat if this is your issue.
> > 
> > On Thursday, 1 February 2018 10:43:37 UTC, simon harrison wrote:
> > 
> > Hi there
> > 
> > We're continuing to have trouble with our RabbitMQ setup. Under load things break down quickly. And i use the term "load" very generously here!
> > 
> > Sentry is flooded with alerts such as: OS Error Socket Closed, Connection Forced, Killing n managed threads, RecoverableConnectionError, killing managed thread "run", killing 1 active worker, etc. etc.
> > 
> > We are using Amazons load balancer with 3 rabbit instances behind it.
> > 
> > All we are doing is pub-sub between 2 services, and this is very light message publishing.
> > 
> > Our heartbeat is set to 5 seconds.
> > 
> > Can anyone advise or suggest somewhere we can get hands-on help with our rabbit setup?
> > 
> > Thanks

---

<div class="post-metadata">

**Author:** ![simon\_harrison](https://avatars.discourse-cdn.com/v4/letter/s/b5ac83/32.png) [@simon\_harrison](https://discourse.nameko.io/u/simon_harrison)\
**Post date:** [February 7, 2018, 8:27pm UTC](https://discourse.nameko.io/t/rabbitmq-under-load/235/5 "2018-02-07T20:27:07Z")

</div>

Thanks Matt.

I've set ours to the default. I'll update here what happens next.

> **···**
>
> On Friday, 2 February 2018 10:22:14 UTC, Matt Yule-Bennett wrote:
> 
> > On Thursday, February 1, 2018 at 11:29:21 AM UTC, simon harrison wrote:  
> > \> Thanks Matt, I appreciate your very quick response. and I've passed that  
> > thread on.  
> > \>  
> > \>  
> > \> All our handlers typically either write to postgres or redis.
> > 
> > It might be just one handler causing the problem. You only need one thread  
> > to hold the cpu for longer than 2x the heartbeat  
> > \>  
> > \>  
> > \> What heartbeat do you use/recommend?
> > 
> > We use the nameko default of 60 seconds
> > 
> > \>  
> > \>  
> > \>  
> > \>  
> > \>  
> > \> On Thursday, 1 February 2018 11:08:47 UTC, Matt Yule-Bennett wrote:  
> > \>  
> > \> With a 5 second heartbeat this is almost certainly the same problem as  
> > reported here:  
> > [Redirecting to Google Groups](https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/nameko-dev/Hmz659s4dBY/bqfmZtS9AwAJ)  
> > \>  
> > \>  
> > \> From that thread:  
> > \>  
> > \>  
> > \> \> If the workload inside the event handler isn't doing any I/O it is  
> > likely to be stealing the CPU long enough to starve the thread that's  
> > sending your heartbeats. Once two heartbeats are missed the broker will  
> > close the connection from the other end. You'll be able to see this  
> > happening if you check your rabbit logs.  
> > \>  
> > \>  
> > \> Unfortunately Nameko handles the error very ungracefully. It would be  
> > much better to catch it and try to reconnect rather than letting it bubble  
> > and kill the container, but fixing that won't solve what I suspect is your  
> > underlying issue: tight CPU loops will starve other greenthreads. This is a  
> > inherent limitation of implicitly yielding co-routines. There is a simple  
> > fix though -- offload non-yielding code to a (native) threadpool, or insert  
> > some explicit yields. Another quick fix would be to decrease the heartbeat  
> > frequency.  
> > \>  
> > \>  
> > \> Please check your rabbit broker logs for "heartbeat missed" messages to  
> > confirm this is the source of your problem. You could also take a look at  
> > the amount of time you take to service each request -- you'll see some that  
> > are longer than 2x the heartbeat if this is your issue.  
> > \>  
> > \>  
> > \>  
> > \> On Thursday, 1 February 2018 10:43:37 UTC, simon harrison wrote:  
> > \>  
> > \> Hi there  
> > \>  
> > \>  
> > \> We're continuing to have trouble with our RabbitMQ setup. Under load  
> > things break down quickly. And i use the term "load" very generously here!  
> > \>  
> > \>  
> > \> Sentry is flooded with alerts such as: OS Error Socket Closed,  
> > Connection Forced, Killing n managed threads, RecoverableConnectionError,  
> > killing managed thread "run", killing 1 active worker, etc. etc.  
> > \>  
> > \>  
> > \> We are using Amazons load balancer with 3 rabbit instances behind it.  
> > \>  
> > \>  
> > \> All we are doing is pub-sub between 2 services, and this is very light  
> > message publishing.  
> > \>  
> > \>  
> > \> Our heartbeat is set to 5 seconds.  
> > \>  
> > \>  
> > \> Can anyone advise or suggest somewhere we can get hands-on help with our  
> > rabbit setup?  
> > \>  
> > \>  
> > \> Thanks

---

<div class="post-metadata">

**Author:** ![mattbennett](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/mattbennett/32/13_2.png) [@mattbennett](https://discourse.nameko.io/u/mattbennett)\
**Post date:** [February 13, 2018, 11:43am UTC](https://discourse.nameko.io/t/rabbitmq-under-load/235/6 "2018-02-13T11:43:27Z")

</div>

There is a bug here. The same thing exhibits itself in  
[https://groups.google.com/forum/?utm\_medium=email&utm\_source=footer#!msg/nameko-dev/Hmz659s4dBY/bqfmZtS9AwAJ](https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/nameko-dev/Hmz659s4dBY/bqfmZtS9AwAJ)

I'm building a testcase to reproduce, but it happens when we try to  
acknowledge the message through a connection that has already closed. This  
situation arises when the connection is lost while a worker is still  
running, so it could be triggered by missed heartbeats or flaky  
connections. We actually retry [1] these errors, but even if a connection  
is re-established, the channel for the particular message will never  
recover and so the retry simply postpones the error bubbling out.

We actually have tests for this scenario but they're not precise enough and  
don't notice that the container dies.

It should be an easy fix once I'm confident I know exactly what's going on.

[1]

> <https://github.com/nameko/nameko/blob/b33a36197cc3d819360317f213f10afd0901b589/nameko/messaging.py#L313-L319>

> **···**
>
> On Wednesday, February 7, 2018 at 8:27:07 PM UTC, simon harrison wrote:
> 
> > Thanks Matt.
> > 
> > I've set ours to the default. I'll update here what happens next.
> > 
> > On Friday, 2 February 2018 10:22:14 UTC, Matt Yule-Bennett wrote:
> > 
> > > On Thursday, February 1, 2018 at 11:29:21 AM UTC, simon harrison wrote:  
> > > \> Thanks Matt, I appreciate your very quick response. and I've passed  
> > > that thread on.  
> > > \>  
> > > \>  
> > > \> All our handlers typically either write to postgres or redis.
> > > 
> > > It might be just one handler causing the problem. You only need one  
> > > thread to hold the cpu for longer than 2x the heartbeat  
> > > \>  
> > > \>  
> > > \> What heartbeat do you use/recommend?
> > > 
> > > We use the nameko default of 60 seconds
> > > 
> > > \>  
> > > \>  
> > > \>  
> > > \>  
> > > \>  
> > > \> On Thursday, 1 February 2018 11:08:47 UTC, Matt Yule-Bennett wrote:  
> > > \>  
> > > \> With a 5 second heartbeat this is almost certainly the same problem as  
> > > reported here:  
> > > [Redirecting to Google Groups](https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/nameko-dev/Hmz659s4dBY/bqfmZtS9AwAJ)  
> > > \>  
> > > \>  
> > > \> From that thread:  
> > > \>  
> > > \>  
> > > \> \> If the workload inside the event handler isn't doing any I/O it is  
> > > likely to be stealing the CPU long enough to starve the thread that's  
> > > sending your heartbeats. Once two heartbeats are missed the broker will  
> > > close the connection from the other end. You'll be able to see this  
> > > happening if you check your rabbit logs.  
> > > \>  
> > > \>  
> > > \> Unfortunately Nameko handles the error very ungracefully. It would be  
> > > much better to catch it and try to reconnect rather than letting it bubble  
> > > and kill the container, but fixing that won't solve what I suspect is your  
> > > underlying issue: tight CPU loops will starve other greenthreads. This is a  
> > > inherent limitation of implicitly yielding co-routines. There is a simple  
> > > fix though -- offload non-yielding code to a (native) threadpool, or insert  
> > > some explicit yields. Another quick fix would be to decrease the heartbeat  
> > > frequency.  
> > > \>  
> > > \>  
> > > \> Please check your rabbit broker logs for "heartbeat missed" messages to  
> > > confirm this is the source of your problem. You could also take a look at  
> > > the amount of time you take to service each request -- you'll see some that  
> > > are longer than 2x the heartbeat if this is your issue.  
> > > \>  
> > > \>  
> > > \>  
> > > \> On Thursday, 1 February 2018 10:43:37 UTC, simon harrison wrote:  
> > > \>  
> > > \> Hi there  
> > > \>  
> > > \>  
> > > \> We're continuing to have trouble with our RabbitMQ setup. Under load  
> > > things break down quickly. And i use the term "load" very generously here!  
> > > \>  
> > > \>  
> > > \> Sentry is flooded with alerts such as: OS Error Socket Closed,  
> > > Connection Forced, Killing n managed threads, RecoverableConnectionError,  
> > > killing managed thread "run", killing 1 active worker, etc. etc.  
> > > \>  
> > > \>  
> > > \> We are using Amazons load balancer with 3 rabbit instances behind it.  
> > > \>  
> > > \>  
> > > \> All we are doing is pub-sub between 2 services, and this is very light  
> > > message publishing.  
> > > \>  
> > > \>  
> > > \> Our heartbeat is set to 5 seconds.  
> > > \>  
> > > \>  
> > > \> Can anyone advise or suggest somewhere we can get hands-on help with  
> > > our rabbit setup?  
> > > \>  
> > > \>  
> > > \> Thanks

---

<div class="post-metadata">

**Author:** ![simon\_harrison](https://avatars.discourse-cdn.com/v4/letter/s/b5ac83/32.png) [@simon\_harrison](https://discourse.nameko.io/u/simon_harrison)\
**Post date:** [February 15, 2018, 3:10pm UTC](https://discourse.nameko.io/t/rabbitmq-under-load/235/7 "2018-02-15T15:10:09Z")

</div>

Thanks Matt.

Interesting to hear about this and I've seen the PR - great stuff.

Simon

> **···**
>
> On Tuesday, 13 February 2018 11:43:27 UTC, Matt Yule-Bennett wrote:
> 
> > There is a bug here. The same thing exhibits itself in  
> > [Redirecting to Google Groups](https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/nameko-dev/Hmz659s4dBY/bqfmZtS9AwAJ)
> > 
> > I'm building a testcase to reproduce, but it happens when we try to  
> > acknowledge the message through a connection that has already closed. This  
> > situation arises when the connection is lost while a worker is still  
> > running, so it could be triggered by missed heartbeats or flaky  
> > connections. We actually retry [1] these errors, but even if a connection  
> > is re-established, the channel for the particular message will never  
> > recover and so the retry simply postpones the error bubbling out.
> > 
> > We actually have tests for this scenario but they're not precise enough  
> > and don't notice that the container dies.
> > 
> > It should be an easy fix once I'm confident I know exactly what's going on.
> > 
> > [1]  
> > [https://github.com/nameko/nameko/blob/b33a36197cc3d819360317f213f10afd0901b589/nameko/messaging.py#L313-L319](https://github.com/nameko/nameko/blob/b33a36197cc3d819360317f213f10afd0901b589/nameko/messaging.py#L313-L319)
> > 
> > On Wednesday, February 7, 2018 at 8:27:07 PM UTC, simon harrison wrote:
> > 
> > > Thanks Matt.
> > > 
> > > I've set ours to the default. I'll update here what happens next.
> > > 
> > > On Friday, 2 February 2018 10:22:14 UTC, Matt Yule-Bennett wrote:
> > > 
> > > > On Thursday, February 1, 2018 at 11:29:21 AM UTC, simon harrison wrote:  
> > > > \> Thanks Matt, I appreciate your very quick response. and I've passed  
> > > > that thread on.  
> > > > \>  
> > > > \>  
> > > > \> All our handlers typically either write to postgres or redis.
> > > > 
> > > > It might be just one handler causing the problem. You only need one  
> > > > thread to hold the cpu for longer than 2x the heartbeat  
> > > > \>  
> > > > \>  
> > > > \> What heartbeat do you use/recommend?
> > > > 
> > > > We use the nameko default of 60 seconds
> > > > 
> > > > \>  
> > > > \>  
> > > > \>  
> > > > \>  
> > > > \>  
> > > > \> On Thursday, 1 February 2018 11:08:47 UTC, Matt Yule-Bennett wrote:  
> > > > \>  
> > > > \> With a 5 second heartbeat this is almost certainly the same problem as  
> > > > reported here:  
> > > > [Redirecting to Google Groups](https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/nameko-dev/Hmz659s4dBY/bqfmZtS9AwAJ)  
> > > > \>  
> > > > \>  
> > > > \> From that thread:  
> > > > \>  
> > > > \>  
> > > > \> \> If the workload inside the event handler isn't doing any I/O it is  
> > > > likely to be stealing the CPU long enough to starve the thread that's  
> > > > sending your heartbeats. Once two heartbeats are missed the broker will  
> > > > close the connection from the other end. You'll be able to see this  
> > > > happening if you check your rabbit logs.  
> > > > \>  
> > > > \>  
> > > > \> Unfortunately Nameko handles the error very ungracefully. It would be  
> > > > much better to catch it and try to reconnect rather than letting it bubble  
> > > > and kill the container, but fixing that won't solve what I suspect is your  
> > > > underlying issue: tight CPU loops will starve other greenthreads. This is a  
> > > > inherent limitation of implicitly yielding co-routines. There is a simple  
> > > > fix though -- offload non-yielding code to a (native) threadpool, or insert  
> > > > some explicit yields. Another quick fix would be to decrease the heartbeat  
> > > > frequency.  
> > > > \>  
> > > > \>  
> > > > \> Please check your rabbit broker logs for "heartbeat missed" messages  
> > > > to confirm this is the source of your problem. You could also take a look  
> > > > at the amount of time you take to service each request -- you'll see some  
> > > > that are longer than 2x the heartbeat if this is your issue.  
> > > > \>  
> > > > \>  
> > > > \>  
> > > > \> On Thursday, 1 February 2018 10:43:37 UTC, simon harrison wrote:  
> > > > \>  
> > > > \> Hi there  
> > > > \>  
> > > > \>  
> > > > \> We're continuing to have trouble with our RabbitMQ setup. Under load  
> > > > things break down quickly. And i use the term "load" very generously here!  
> > > > \>  
> > > > \>  
> > > > \> Sentry is flooded with alerts such as: OS Error Socket Closed,  
> > > > Connection Forced, Killing n managed threads, RecoverableConnectionError,  
> > > > killing managed thread "run", killing 1 active worker, etc. etc.  
> > > > \>  
> > > > \>  
> > > > \> We are using Amazons load balancer with 3 rabbit instances behind it.  
> > > > \>  
> > > > \>  
> > > > \> All we are doing is pub-sub between 2 services, and this is very light  
> > > > message publishing.  
> > > > \>  
> > > > \>  
> > > > \> Our heartbeat is set to 5 seconds.  
> > > > \>  
> > > > \>  
> > > > \> Can anyone advise or suggest somewhere we can get hands-on help with  
> > > > our rabbit setup?  
> > > > \>  
> > > > \>  
> > > > \> Thanks