# Nameko is VERY slow since version 2.3.0

**URL:** <https://discourse.nameko.io/t/nameko-is-very-slow-since-version-2-3-0/204>\
**Category:** googlegroup\
**Created:** [August 22, 2017, 10:56am UTC](https://discourse.nameko.io/t/nameko-is-very-slow-since-version-2-3-0/204 "2017-08-22T10:56:04Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tsachi\_Shuval](https://avatars.discourse-cdn.com/v4/letter/t/5e9695/32.png) [@Tsachi\_Shuval](https://discourse.nameko.io/u/Tsachi_Shuval)\
**Post date:** [August 22, 2017, 10:56am UTC](https://discourse.nameko.io/t/nameko-is-very-slow-since-version-2-3-0/204/1 "2017-08-22T10:56:04Z")

</div>

Hi  
We've been using nameko 2.2.0 in production for over a year now.  
Recently, we were requested to use the "heartbeat" feature of rabbit  
(introduced in 2.4.4).

I started by upgrading to the most recent version.  
The moment I started the services, I noticed it took them a long to start.

Using the CLI, I called a service to fetch a record from the database, and  
the response took more than 1 sec to return.  
This is VERY slow.

I downgraded to 2.4.4. The slowness remained.  
I started downgrading, and noticed that this "slowness" was introduced in  
version 2.3.0.

Do you have any insight regarding this issue?

Thanks,  
Tsachi

---

<div class="post-metadata">

**Author:** ![Tsachi\_Shuval](https://avatars.discourse-cdn.com/v4/letter/t/5e9695/32.png) [@Tsachi\_Shuval](https://discourse.nameko.io/u/Tsachi_Shuval)\
**Post date:** [August 22, 2017, 11:19am UTC](https://discourse.nameko.io/t/nameko-is-very-slow-since-version-2-3-0/204/2 "2017-08-22T11:19:59Z")

</div>

In addition, specifically to heartbeats, the kombu docs state that:

if you have Connection(heartbeat=10),  
then you must call Connection.heartbeat\_check() every 5 seconds.

I didn't see anything in version 2.4.4 calling the heartbeat\_check() method  
on the connection object.

How, then, would heartbeat work?

T.

> **···**
>
> On Tuesday, August 22, 2017 at 1:56:04 PM UTC+3, tsachi...@gmail.com wrote:
> 
> > Hi  
> > We've been using nameko 2.2.0 in production for over a year now.  
> > Recently, we were requested to use the "heartbeat" feature of rabbit  
> > (introduced in 2.4.4).
> > 
> > I started by upgrading to the most recent version.  
> > The moment I started the services, I noticed it took them a long to start.
> > 
> > Using the CLI, I called a service to fetch a record from the database, and  
> > the response took more than 1 sec to return.  
> > This is VERY slow.
> > 
> > I downgraded to 2.4.4. The slowness remained.  
> > I started downgrading, and noticed that this "slowness" was introduced in  
> > version 2.3.0.
> > 
> > Do you have any insight regarding this issue?
> > 
> > Thanks,  
> > Tsachi

---

<div class="post-metadata">

**Author:** ![mattbennett](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/mattbennett/32/13_2.png) [@mattbennett](https://discourse.nameko.io/u/mattbennett)\
**Post date:** [August 22, 2017, 1:40pm UTC](https://discourse.nameko.io/t/nameko-is-very-slow-since-version-2-3-0/204/3 "2017-08-22T13:40:45Z")

</div>

That doc refers to the bare Connection object. The QueueConsumer uses the  
ConsumerMixin, which performs that check. It's here  
\<[https://github.com/celery/kombu/blob/cf2ca7c018af9a9e2f4f32386a34fbcc8dc7af0d/kombu/mixins.py#L201&gt](https://github.com/celery/kombu/blob/cf2ca7c018af9a9e2f4f32386a34fbcc8dc7af0d/kombu/mixins.py#L201&gt);  
.

Regarding performance, please post an example. At a minimum, which  
entrypoint and dependency providers are you using?

> **···**
>
> On Tuesday, 22 August 2017 12:19:59 UTC+1, tsachi...@gmail.com wrote:
> 
> > In addition, specifically to heartbeats, the kombu docs state that:
> > 
> > if you have Connection(heartbeat=10),  
> > then you must call Connection.heartbeat\_check() every 5 seconds.
> > 
> > I didn't see anything in version 2.4.4 calling the heartbeat\_check()  
> > method on the connection object.
> > 
> > How, then, would heartbeat work?
> > 
> > T.
> > 
> > On Tuesday, August 22, 2017 at 1:56:04 PM UTC+3, tsachi...@gmail.com \> wrote:
> > 
> > > Hi  
> > > We've been using nameko 2.2.0 in production for over a year now.  
> > > Recently, we were requested to use the "heartbeat" feature of rabbit  
> > > (introduced in 2.4.4).
> > > 
> > > I started by upgrading to the most recent version.  
> > > The moment I started the services, I noticed it took them a long to start.
> > > 
> > > Using the CLI, I called a service to fetch a record from the database,  
> > > and the response took more than 1 sec to return.  
> > > This is VERY slow.
> > > 
> > > I downgraded to 2.4.4. The slowness remained.  
> > > I started downgrading, and noticed that this "slowness" was introduced in  
> > > version 2.3.0.
> > > 
> > > Do you have any insight regarding this issue?
> > > 
> > > Thanks,  
> > > Tsachi

---

<div class="post-metadata">

**Author:** ![Tsachi\_Shuval](https://avatars.discourse-cdn.com/v4/letter/t/5e9695/32.png) [@Tsachi\_Shuval](https://discourse.nameko.io/u/Tsachi_Shuval)\
**Post date:** [August 23, 2017, 7:11am UTC](https://discourse.nameko.io/t/nameko-is-very-slow-since-version-2-3-0/204/4 "2017-08-23T07:11:05Z")

</div>

The entrypoint is rpc, and no dependency providers are used.

You can see the slowness the moment you start the services. We have a dozen  
of them.  
With the old version (2.2.0), once you issued the "nameko run" command, you  
could see a dozen "Connected to amqp..." messages immediately.  
With newer versions, these messages appear one by one. So you can see it  
takes longer to start.

This is also verified when calling an rpc endpoint that returns a record  
from the db.

> **···**
>
> On Tuesday, August 22, 2017 at 4:40:45 PM UTC+3, Matt Yule-Bennett wrote:
> 
> > That doc refers to the bare Connection object. The QueueConsumer uses the  
> > ConsumerMixin, which performs that check. It's here  
> > \<[https://github.com/celery/kombu/blob/cf2ca7c018af9a9e2f4f32386a34fbcc8dc7af0d/kombu/mixins.py#L201&gt](https://github.com/celery/kombu/blob/cf2ca7c018af9a9e2f4f32386a34fbcc8dc7af0d/kombu/mixins.py#L201&gt);  
> > .
> > 
> > Regarding performance, please post an example. At a minimum, which  
> > entrypoint and dependency providers are you using?
> > 
> > On Tuesday, 22 August 2017 12:19:59 UTC+1, tsachi...@gmail.com wrote:
> > 
> > > In addition, specifically to heartbeats, the kombu docs state that:
> > > 
> > > if you have Connection(heartbeat=10),  
> > > then you must call Connection.heartbeat\_check() every 5 seconds.
> > > 
> > > I didn't see anything in version 2.4.4 calling the heartbeat\_check()  
> > > method on the connection object.
> > > 
> > > How, then, would heartbeat work?
> > > 
> > > T.
> > > 
> > > On Tuesday, August 22, 2017 at 1:56:04 PM UTC+3, tsachi...@gmail.com \>\> wrote:
> > > 
> > > > Hi  
> > > > We've been using nameko 2.2.0 in production for over a year now.  
> > > > Recently, we were requested to use the "heartbeat" feature of rabbit  
> > > > (introduced in 2.4.4).
> > > > 
> > > > I started by upgrading to the most recent version.  
> > > > The moment I started the services, I noticed it took them a long to  
> > > > start.
> > > > 
> > > > Using the CLI, I called a service to fetch a record from the database,  
> > > > and the response took more than 1 sec to return.  
> > > > This is VERY slow.
> > > > 
> > > > I downgraded to 2.4.4. The slowness remained.  
> > > > I started downgrading, and noticed that this "slowness" was introduced  
> > > > in version 2.3.0.
> > > > 
> > > > Do you have any insight regarding this issue?
> > > > 
> > > > Thanks,  
> > > > Tsachi

---

<div class="post-metadata">

**Author:** ![David\_Szotten](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/david_szotten/32/6_2.png) [@David\_Szotten](https://discourse.nameko.io/u/David_Szotten)\
**Post date:** [August 23, 2017, 8:18am UTC](https://discourse.nameko.io/t/nameko-is-very-slow-since-version-2-3-0/204/5 "2017-08-23T08:18:38Z")

</div>

Hi,

To help us understand what's going on, please post a complete (but as  
minimal as possible) example that we can run to reproduce the behaviour you  
are seeing.

Best,  
D

> **···**
>
> On Wednesday, 23 August 2017 08:11:05 UTC+1, tsachi...@gmail.com wrote:
> 
> > The entrypoint is rpc, and no dependency providers are used.
> > 
> > You can see the slowness the moment you start the services. We have a  
> > dozen of them.  
> > With the old version (2.2.0), once you issued the "nameko run" command,  
> > you could see a dozen "Connected to amqp..." messages immediately.  
> > With newer versions, these messages appear one by one. So you can see it  
> > takes longer to start.
> > 
> > This is also verified when calling an rpc endpoint that returns a record  
> > from the db.
> > 
> > On Tuesday, August 22, 2017 at 4:40:45 PM UTC+3, Matt Yule-Bennett wrote:
> > 
> > > That doc refers to the bare Connection object. The QueueConsumer uses the  
> > > ConsumerMixin, which performs that check. It's here  
> > > \<[https://github.com/celery/kombu/blob/cf2ca7c018af9a9e2f4f32386a34fbcc8dc7af0d/kombu/mixins.py#L201&gt](https://github.com/celery/kombu/blob/cf2ca7c018af9a9e2f4f32386a34fbcc8dc7af0d/kombu/mixins.py#L201&gt);  
> > > .
> > > 
> > > Regarding performance, please post an example. At a minimum, which  
> > > entrypoint and dependency providers are you using?
> > > 
> > > On Tuesday, 22 August 2017 12:19:59 UTC+1, tsachi...@gmail.com wrote:
> > > 
> > > > In addition, specifically to heartbeats, the kombu docs state that:
> > > > 
> > > > if you have Connection(heartbeat=10),  
> > > > then you must call Connection.heartbeat\_check() every 5 seconds.
> > > > 
> > > > I didn't see anything in version 2.4.4 calling the heartbeat\_check()  
> > > > method on the connection object.
> > > > 
> > > > How, then, would heartbeat work?
> > > > 
> > > > T.
> > > > 
> > > > On Tuesday, August 22, 2017 at 1:56:04 PM UTC+3, tsachi...@gmail.com \>\>\> wrote:
> > > > 
> > > > > Hi  
> > > > > We've been using nameko 2.2.0 in production for over a year now.  
> > > > > Recently, we were requested to use the "heartbeat" feature of rabbit  
> > > > > (introduced in 2.4.4).
> > > > > 
> > > > > I started by upgrading to the most recent version.  
> > > > > The moment I started the services, I noticed it took them a long to  
> > > > > start.
> > > > > 
> > > > > Using the CLI, I called a service to fetch a record from the database,  
> > > > > and the response took more than 1 sec to return.  
> > > > > This is VERY slow.
> > > > > 
> > > > > I downgraded to 2.4.4. The slowness remained.  
> > > > > I started downgrading, and noticed that this "slowness" was introduced  
> > > > > in version 2.3.0.
> > > > > 
> > > > > Do you have any insight regarding this issue?
> > > > > 
> > > > > Thanks,  
> > > > > Tsachi

---

<div class="post-metadata">

**Author:** ![Tsachi\_Shuval](https://avatars.discourse-cdn.com/v4/letter/t/5e9695/32.png) [@Tsachi\_Shuval](https://discourse.nameko.io/u/Tsachi_Shuval)\
**Post date:** [August 23, 2017, 8:40am UTC](https://discourse.nameko.io/t/nameko-is-very-slow-since-version-2-3-0/204/6 "2017-08-23T08:40:26Z")

</div>

Hi David,

I managed to trace the origin of the slow queries.  
Background: we're using Django models for accessing MySQL db.  
Django uses MySQLdb (written in C), which means we lose concurrency in a  
greenlet-based environment.

To remedy this, we install the pymysql package like so:

import pymysql  
pymysql.install\_as\_MySQLdb()

This should preserve greenlet concurrency.

I just noticed that when I comment the "install\_as\_MySQLdb()" line, nameko  
(2.3.0 and up) returns to it's normal speed/behavior (v 2.2.0) when  
querying the db.

So maybe there's something related to greenlets that has changed in this  
version?

T.

> **···**
>
> On Wednesday, August 23, 2017 at 11:18:38 AM UTC+3, David Szotten wrote:
> 
> > Hi,
> > 
> > To help us understand what's going on, please post a complete (but as  
> > minimal as possible) example that we can run to reproduce the behaviour you  
> > are seeing.
> > 
> > Best,  
> > D
> > 
> > On Wednesday, 23 August 2017 08:11:05 UTC+1, tsachi...@gmail.com wrote:
> > 
> > > The entrypoint is rpc, and no dependency providers are used.
> > > 
> > > You can see the slowness the moment you start the services. We have a  
> > > dozen of them.  
> > > With the old version (2.2.0), once you issued the "nameko run" command,  
> > > you could see a dozen "Connected to amqp..." messages immediately.  
> > > With newer versions, these messages appear one by one. So you can see it  
> > > takes longer to start.
> > > 
> > > This is also verified when calling an rpc endpoint that returns a record  
> > > from the db.
> > > 
> > > On Tuesday, August 22, 2017 at 4:40:45 PM UTC+3, Matt Yule-Bennett wrote:
> > > 
> > > > That doc refers to the bare Connection object. The QueueConsumer uses  
> > > > the ConsumerMixin, which performs that check. It's here  
> > > > \<[https://github.com/celery/kombu/blob/cf2ca7c018af9a9e2f4f32386a34fbcc8dc7af0d/kombu/mixins.py#L201&gt](https://github.com/celery/kombu/blob/cf2ca7c018af9a9e2f4f32386a34fbcc8dc7af0d/kombu/mixins.py#L201&gt);  
> > > > .
> > > > 
> > > > Regarding performance, please post an example. At a minimum, which  
> > > > entrypoint and dependency providers are you using?
> > > > 
> > > > On Tuesday, 22 August 2017 12:19:59 UTC+1, tsachi...@gmail.com wrote:
> > > > 
> > > > > In addition, specifically to heartbeats, the kombu docs state that:
> > > > > 
> > > > > if you have Connection(heartbeat=10),  
> > > > > then you must call Connection.heartbeat\_check() every 5 seconds.
> > > > > 
> > > > > I didn't see anything in version 2.4.4 calling the heartbeat\_check()  
> > > > > method on the connection object.
> > > > > 
> > > > > How, then, would heartbeat work?
> > > > > 
> > > > > T.
> > > > > 
> > > > > On Tuesday, August 22, 2017 at 1:56:04 PM UTC+3, tsachi...@gmail.com \>\>\>\> wrote:
> > > > > 
> > > > > > Hi  
> > > > > > We've been using nameko 2.2.0 in production for over a year now.  
> > > > > > Recently, we were requested to use the "heartbeat" feature of rabbit  
> > > > > > (introduced in 2.4.4).
> > > > > > 
> > > > > > I started by upgrading to the most recent version.  
> > > > > > The moment I started the services, I noticed it took them a long to  
> > > > > > start.
> > > > > > 
> > > > > > Using the CLI, I called a service to fetch a record from the database,  
> > > > > > and the response took more than 1 sec to return.  
> > > > > > This is VERY slow.
> > > > > > 
> > > > > > I downgraded to 2.4.4. The slowness remained.  
> > > > > > I started downgrading, and noticed that this "slowness" was introduced  
> > > > > > in version 2.3.0.
> > > > > > 
> > > > > > Do you have any insight regarding this issue?
> > > > > > 
> > > > > > Thanks,  
> > > > > > Tsachi

---

<div class="post-metadata">

**Author:** ![David\_Szotten](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/david_szotten/32/6_2.png) [@David\_Szotten](https://discourse.nameko.io/u/David_Szotten)\
**Post date:** [August 23, 2017, 9:32am UTC](https://discourse.nameko.io/t/nameko-is-very-slow-since-version-2-3-0/204/7 "2017-08-23T09:32:36Z")

</div>

hm, i'm actually more surprised you haven't had issues before now. not sure  
what might have changed, maybe an eventlet version bump?

if pymysql works for you, great. at onefinestay we initially used pymysql,  
but later swiched back to using mysqldb wrapped in a threadpool for  
performance reasons (we had some pathological cases that were noticeably  
faster with the c library). i'm not sure we ever open-sourced that  
dependency (and i'm no longer at onefinestay) but if i recall it was quite  
tricky to get right so i'd recommend staying with pymysqldb if that works  
for you

glad you found your issue

best,  
david

> **···**
>
> On Wednesday, 23 August 2017 09:40:27 UTC+1, tsachi...@gmail.com wrote:
> 
> > Hi David,
> > 
> > I managed to trace the origin of the slow queries.  
> > Background: we're using Django models for accessing MySQL db.  
> > Django uses MySQLdb (written in C), which means we lose concurrency in a  
> > greenlet-based environment.
> > 
> > To remedy this, we install the pymysql package like so:
> > 
> > import pymysql  
> > pymysql.install\_as\_MySQLdb()
> > 
> > This should preserve greenlet concurrency.
> > 
> > I just noticed that when I comment the "install\_as\_MySQLdb()" line, nameko  
> > (2.3.0 and up) returns to it's normal speed/behavior (v 2.2.0) when  
> > querying the db.
> > 
> > So maybe there's something related to greenlets that has changed in this  
> > version?
> > 
> > T.
> > 
> > On Wednesday, August 23, 2017 at 11:18:38 AM UTC+3, David Szotten wrote:
> > 
> > > Hi,
> > > 
> > > To help us understand what's going on, please post a complete (but as  
> > > minimal as possible) example that we can run to reproduce the behaviour you  
> > > are seeing.
> > > 
> > > Best,  
> > > D
> > > 
> > > On Wednesday, 23 August 2017 08:11:05 UTC+1, tsachi...@gmail.com wrote:
> > > 
> > > > The entrypoint is rpc, and no dependency providers are used.
> > > > 
> > > > You can see the slowness the moment you start the services. We have a  
> > > > dozen of them.  
> > > > With the old version (2.2.0), once you issued the "nameko run" command,  
> > > > you could see a dozen "Connected to amqp..." messages immediately.  
> > > > With newer versions, these messages appear one by one. So you can see it  
> > > > takes longer to start.
> > > > 
> > > > This is also verified when calling an rpc endpoint that returns a record  
> > > > from the db.
> > > > 
> > > > On Tuesday, August 22, 2017 at 4:40:45 PM UTC+3, Matt Yule-Bennett wrote:
> > > > 
> > > > > That doc refers to the bare Connection object. The QueueConsumer uses  
> > > > > the ConsumerMixin, which performs that check. It's here  
> > > > > \<[https://github.com/celery/kombu/blob/cf2ca7c018af9a9e2f4f32386a34fbcc8dc7af0d/kombu/mixins.py#L201&gt](https://github.com/celery/kombu/blob/cf2ca7c018af9a9e2f4f32386a34fbcc8dc7af0d/kombu/mixins.py#L201&gt);  
> > > > > .
> > > > > 
> > > > > Regarding performance, please post an example. At a minimum, which  
> > > > > entrypoint and dependency providers are you using?
> > > > > 
> > > > > On Tuesday, 22 August 2017 12:19:59 UTC+1, tsachi...@gmail.com wrote:
> > > > > 
> > > > > > In addition, specifically to heartbeats, the kombu docs state that:
> > > > > > 
> > > > > > if you have Connection(heartbeat=10),  
> > > > > > then you must call Connection.heartbeat\_check() every 5 seconds.
> > > > > > 
> > > > > > I didn't see anything in version 2.4.4 calling the heartbeat\_check()  
> > > > > > method on the connection object.
> > > > > > 
> > > > > > How, then, would heartbeat work?
> > > > > > 
> > > > > > T.
> > > > > > 
> > > > > > On Tuesday, August 22, 2017 at 1:56:04 PM UTC+3, tsachi...@gmail.com \>\>\>\>\> wrote:
> > > > > > 
> > > > > > > Hi  
> > > > > > > We've been using nameko 2.2.0 in production for over a year now.  
> > > > > > > Recently, we were requested to use the "heartbeat" feature of rabbit  
> > > > > > > (introduced in 2.4.4).
> > > > > > > 
> > > > > > > I started by upgrading to the most recent version.  
> > > > > > > The moment I started the services, I noticed it took them a long to  
> > > > > > > start.
> > > > > > > 
> > > > > > > Using the CLI, I called a service to fetch a record from the  
> > > > > > > database, and the response took more than 1 sec to return.  
> > > > > > > This is VERY slow.
> > > > > > > 
> > > > > > > I downgraded to 2.4.4. The slowness remained.  
> > > > > > > I started downgrading, and noticed that this "slowness" was  
> > > > > > > introduced in version 2.3.0.
> > > > > > > 
> > > > > > > Do you have any insight regarding this issue?
> > > > > > > 
> > > > > > > Thanks,  
> > > > > > > Tsachi