Hi
We've been using nameko 2.2.0 in production for over a year now.
Recently, we were requested to use the "heartbeat" feature of rabbit
(introduced in 2.4.4).
I started by upgrading to the most recent version.
The moment I started the services, I noticed it took them a long to start.
Using the CLI, I called a service to fetch a record from the database, and
the response took more than 1 sec to return.
This is VERY slow.
I downgraded to 2.4.4. The slowness remained.
I started downgrading, and noticed that this "slowness" was introduced in
version 2.3.0.
Do you have any insight regarding this issue?
Thanks,
Tsachi
In addition, specifically to heartbeats, the kombu docs state that:
if you have Connection(heartbeat=10),
then you must call Connection.heartbeat_check() every 5 seconds.
I didn't see anything in version 2.4.4 calling the heartbeat_check() method
on the connection object.
How, then, would heartbeat work?
T.
···
On Tuesday, August 22, 2017 at 1:56:04 PM UTC+3, tsachi...@gmail.com wrote:
Hi
We've been using nameko 2.2.0 in production for over a year now.
Recently, we were requested to use the "heartbeat" feature of rabbit
(introduced in 2.4.4).
I started by upgrading to the most recent version.
The moment I started the services, I noticed it took them a long to start.
Using the CLI, I called a service to fetch a record from the database, and
the response took more than 1 sec to return.
This is VERY slow.
I downgraded to 2.4.4. The slowness remained.
I started downgrading, and noticed that this "slowness" was introduced in
version 2.3.0.
Do you have any insight regarding this issue?
Thanks,
Tsachi
That doc refers to the bare Connection object. The QueueConsumer uses the
ConsumerMixin, which performs that check. It's here
<https://github.com/celery/kombu/blob/cf2ca7c018af9a9e2f4f32386a34fbcc8dc7af0d/kombu/mixins.py#L201>
.
Regarding performance, please post an example. At a minimum, which
entrypoint and dependency providers are you using?
···
On Tuesday, 22 August 2017 12:19:59 UTC+1, tsachi...@gmail.com wrote:
In addition, specifically to heartbeats, the kombu docs state that:
if you have Connection(heartbeat=10),
then you must call Connection.heartbeat_check() every 5 seconds.
I didn't see anything in version 2.4.4 calling the heartbeat_check()
method on the connection object.
How, then, would heartbeat work?
T.
On Tuesday, August 22, 2017 at 1:56:04 PM UTC+3, tsachi...@gmail.com > wrote:
Hi
We've been using nameko 2.2.0 in production for over a year now.
Recently, we were requested to use the "heartbeat" feature of rabbit
(introduced in 2.4.4).
I started by upgrading to the most recent version.
The moment I started the services, I noticed it took them a long to start.
Using the CLI, I called a service to fetch a record from the database,
and the response took more than 1 sec to return.
This is VERY slow.
I downgraded to 2.4.4. The slowness remained.
I started downgrading, and noticed that this "slowness" was introduced in
version 2.3.0.
Do you have any insight regarding this issue?
Thanks,
Tsachi
The entrypoint is rpc, and no dependency providers are used.
You can see the slowness the moment you start the services. We have a dozen
of them.
With the old version (2.2.0), once you issued the "nameko run" command, you
could see a dozen "Connected to amqp..." messages immediately.
With newer versions, these messages appear one by one. So you can see it
takes longer to start.
This is also verified when calling an rpc endpoint that returns a record
from the db.
···
On Tuesday, August 22, 2017 at 4:40:45 PM UTC+3, Matt Yule-Bennett wrote:
That doc refers to the bare Connection object. The QueueConsumer uses the
ConsumerMixin, which performs that check. It's here
<https://github.com/celery/kombu/blob/cf2ca7c018af9a9e2f4f32386a34fbcc8dc7af0d/kombu/mixins.py#L201>
.
Regarding performance, please post an example. At a minimum, which
entrypoint and dependency providers are you using?
On Tuesday, 22 August 2017 12:19:59 UTC+1, tsachi...@gmail.com wrote:
In addition, specifically to heartbeats, the kombu docs state that:
if you have Connection(heartbeat=10),
then you must call Connection.heartbeat_check() every 5 seconds.
I didn't see anything in version 2.4.4 calling the heartbeat_check()
method on the connection object.
How, then, would heartbeat work?
T.
On Tuesday, August 22, 2017 at 1:56:04 PM UTC+3, tsachi...@gmail.com >> wrote:
Hi
We've been using nameko 2.2.0 in production for over a year now.
Recently, we were requested to use the "heartbeat" feature of rabbit
(introduced in 2.4.4).
I started by upgrading to the most recent version.
The moment I started the services, I noticed it took them a long to
start.
Using the CLI, I called a service to fetch a record from the database,
and the response took more than 1 sec to return.
This is VERY slow.
I downgraded to 2.4.4. The slowness remained.
I started downgrading, and noticed that this "slowness" was introduced
in version 2.3.0.
Do you have any insight regarding this issue?
Thanks,
Tsachi
Hi,
To help us understand what's going on, please post a complete (but as
minimal as possible) example that we can run to reproduce the behaviour you
are seeing.
Best,
D
···
On Wednesday, 23 August 2017 08:11:05 UTC+1, tsachi...@gmail.com wrote:
The entrypoint is rpc, and no dependency providers are used.
You can see the slowness the moment you start the services. We have a
dozen of them.
With the old version (2.2.0), once you issued the "nameko run" command,
you could see a dozen "Connected to amqp..." messages immediately.
With newer versions, these messages appear one by one. So you can see it
takes longer to start.
This is also verified when calling an rpc endpoint that returns a record
from the db.
On Tuesday, August 22, 2017 at 4:40:45 PM UTC+3, Matt Yule-Bennett wrote:
That doc refers to the bare Connection object. The QueueConsumer uses the
ConsumerMixin, which performs that check. It's here
<https://github.com/celery/kombu/blob/cf2ca7c018af9a9e2f4f32386a34fbcc8dc7af0d/kombu/mixins.py#L201>
.
Regarding performance, please post an example. At a minimum, which
entrypoint and dependency providers are you using?
On Tuesday, 22 August 2017 12:19:59 UTC+1, tsachi...@gmail.com wrote:
In addition, specifically to heartbeats, the kombu docs state that:
if you have Connection(heartbeat=10),
then you must call Connection.heartbeat_check() every 5 seconds.
I didn't see anything in version 2.4.4 calling the heartbeat_check()
method on the connection object.
How, then, would heartbeat work?
T.
On Tuesday, August 22, 2017 at 1:56:04 PM UTC+3, tsachi...@gmail.com >>> wrote:
Hi
We've been using nameko 2.2.0 in production for over a year now.
Recently, we were requested to use the "heartbeat" feature of rabbit
(introduced in 2.4.4).
I started by upgrading to the most recent version.
The moment I started the services, I noticed it took them a long to
start.
Using the CLI, I called a service to fetch a record from the database,
and the response took more than 1 sec to return.
This is VERY slow.
I downgraded to 2.4.4. The slowness remained.
I started downgrading, and noticed that this "slowness" was introduced
in version 2.3.0.
Do you have any insight regarding this issue?
Thanks,
Tsachi
Hi David,
I managed to trace the origin of the slow queries.
Background: we're using Django models for accessing MySQL db.
Django uses MySQLdb (written in C), which means we lose concurrency in a
greenlet-based environment.
To remedy this, we install the pymysql package like so:
import pymysql
pymysql.install_as_MySQLdb()
This should preserve greenlet concurrency.
I just noticed that when I comment the "install_as_MySQLdb()" line, nameko
(2.3.0 and up) returns to it's normal speed/behavior (v 2.2.0) when
querying the db.
So maybe there's something related to greenlets that has changed in this
version?
T.
···
On Wednesday, August 23, 2017 at 11:18:38 AM UTC+3, David Szotten wrote:
Hi,
To help us understand what's going on, please post a complete (but as
minimal as possible) example that we can run to reproduce the behaviour you
are seeing.
Best,
D
On Wednesday, 23 August 2017 08:11:05 UTC+1, tsachi...@gmail.com wrote:
The entrypoint is rpc, and no dependency providers are used.
You can see the slowness the moment you start the services. We have a
dozen of them.
With the old version (2.2.0), once you issued the "nameko run" command,
you could see a dozen "Connected to amqp..." messages immediately.
With newer versions, these messages appear one by one. So you can see it
takes longer to start.
This is also verified when calling an rpc endpoint that returns a record
from the db.
On Tuesday, August 22, 2017 at 4:40:45 PM UTC+3, Matt Yule-Bennett wrote:
That doc refers to the bare Connection object. The QueueConsumer uses
the ConsumerMixin, which performs that check. It's here
<https://github.com/celery/kombu/blob/cf2ca7c018af9a9e2f4f32386a34fbcc8dc7af0d/kombu/mixins.py#L201>
.
Regarding performance, please post an example. At a minimum, which
entrypoint and dependency providers are you using?
On Tuesday, 22 August 2017 12:19:59 UTC+1, tsachi...@gmail.com wrote:
In addition, specifically to heartbeats, the kombu docs state that:
if you have Connection(heartbeat=10),
then you must call Connection.heartbeat_check() every 5 seconds.
I didn't see anything in version 2.4.4 calling the heartbeat_check()
method on the connection object.
How, then, would heartbeat work?
T.
On Tuesday, August 22, 2017 at 1:56:04 PM UTC+3, tsachi...@gmail.com >>>> wrote:
Hi
We've been using nameko 2.2.0 in production for over a year now.
Recently, we were requested to use the "heartbeat" feature of rabbit
(introduced in 2.4.4).
I started by upgrading to the most recent version.
The moment I started the services, I noticed it took them a long to
start.
Using the CLI, I called a service to fetch a record from the database,
and the response took more than 1 sec to return.
This is VERY slow.
I downgraded to 2.4.4. The slowness remained.
I started downgrading, and noticed that this "slowness" was introduced
in version 2.3.0.
Do you have any insight regarding this issue?
Thanks,
Tsachi
hm, i'm actually more surprised you haven't had issues before now. not sure
what might have changed, maybe an eventlet version bump?
if pymysql works for you, great. at onefinestay we initially used pymysql,
but later swiched back to using mysqldb wrapped in a threadpool for
performance reasons (we had some pathological cases that were noticeably
faster with the c library). i'm not sure we ever open-sourced that
dependency (and i'm no longer at onefinestay) but if i recall it was quite
tricky to get right so i'd recommend staying with pymysqldb if that works
for you
glad you found your issue
best,
david
···
On Wednesday, 23 August 2017 09:40:27 UTC+1, tsachi...@gmail.com wrote:
Hi David,
I managed to trace the origin of the slow queries.
Background: we're using Django models for accessing MySQL db.
Django uses MySQLdb (written in C), which means we lose concurrency in a
greenlet-based environment.
To remedy this, we install the pymysql package like so:
import pymysql
pymysql.install_as_MySQLdb()
This should preserve greenlet concurrency.
I just noticed that when I comment the "install_as_MySQLdb()" line, nameko
(2.3.0 and up) returns to it's normal speed/behavior (v 2.2.0) when
querying the db.
So maybe there's something related to greenlets that has changed in this
version?
T.
On Wednesday, August 23, 2017 at 11:18:38 AM UTC+3, David Szotten wrote:
Hi,
To help us understand what's going on, please post a complete (but as
minimal as possible) example that we can run to reproduce the behaviour you
are seeing.
Best,
D
On Wednesday, 23 August 2017 08:11:05 UTC+1, tsachi...@gmail.com wrote:
The entrypoint is rpc, and no dependency providers are used.
You can see the slowness the moment you start the services. We have a
dozen of them.
With the old version (2.2.0), once you issued the "nameko run" command,
you could see a dozen "Connected to amqp..." messages immediately.
With newer versions, these messages appear one by one. So you can see it
takes longer to start.
This is also verified when calling an rpc endpoint that returns a record
from the db.
On Tuesday, August 22, 2017 at 4:40:45 PM UTC+3, Matt Yule-Bennett wrote:
That doc refers to the bare Connection object. The QueueConsumer uses
the ConsumerMixin, which performs that check. It's here
<https://github.com/celery/kombu/blob/cf2ca7c018af9a9e2f4f32386a34fbcc8dc7af0d/kombu/mixins.py#L201>
.
Regarding performance, please post an example. At a minimum, which
entrypoint and dependency providers are you using?
On Tuesday, 22 August 2017 12:19:59 UTC+1, tsachi...@gmail.com wrote:
In addition, specifically to heartbeats, the kombu docs state that:
if you have Connection(heartbeat=10),
then you must call Connection.heartbeat_check() every 5 seconds.
I didn't see anything in version 2.4.4 calling the heartbeat_check()
method on the connection object.
How, then, would heartbeat work?
T.
On Tuesday, August 22, 2017 at 1:56:04 PM UTC+3, tsachi...@gmail.com >>>>> wrote:
Hi
We've been using nameko 2.2.0 in production for over a year now.
Recently, we were requested to use the "heartbeat" feature of rabbit
(introduced in 2.4.4).
I started by upgrading to the most recent version.
The moment I started the services, I noticed it took them a long to
start.
Using the CLI, I called a service to fetch a record from the
database, and the response took more than 1 sec to return.
This is VERY slow.
I downgraded to 2.4.4. The slowness remained.
I started downgrading, and noticed that this "slowness" was
introduced in version 2.3.0.
Do you have any insight regarding this issue?
Thanks,
Tsachi