Handling many database connections

googlegroup
1

Hi,

Thanks for a nice library, it have helped us extend our application in a
nice way!

We're breaking out parts of our application (built in Django) and are using
nameko to do communication between the old application and newer services.

A problem we have started to experience is that nameko will happily handle
100s of messages in parallell (on multiple servers) and start new
connections to our SQL database. The database max connection limit will
kick in and block any new connection attempts (which in itself isn't a
problem).

What's the best approach to limit the number of parallell messages handled
by nameko?
Is there a way to do connectionpooling against the database? (With or
without using django)

2

Thanks :slight_smile: Glad you like it.

Nameko uses a pool of worker threads to limit the amount of concurrent
work. Its size is controlled by the `max_workers` config key (which must be
lowercase). The default value is 10, so you should only be seeing at most
that many workers per service instance.

Are you using GitHub - nameko/nameko-sqlalchemy: SQLAlchemy dependency for nameko services? If so,
sqlalchemy will take care of pooling connections for you.

···

On Thursday, February 9, 2017 at 9:31:48 AM UTC, mat...@kundo.se wrote:

Hi,

Thanks for a nice library, it have helped us extend our application in a
nice way!

We're breaking out parts of our application (built in Django) and are
using nameko to do communication between the old application and newer
services.

A problem we have started to experience is that nameko will happily handle
100s of messages in parallell (on multiple servers) and start new
connections to our SQL database. The database max connection limit will
kick in and block any new connection attempts (which in itself isn't a
problem).

What's the best approach to limit the number of parallell messages handled
by nameko?
Is there a way to do connectionpooling against the database? (With or
without using django)

3

Thanks for your reply!

It looks like it should limit when I have checked more of the code related to this. We'll make an attempt to tweak the number.

nameko-sqlalchemy looks promising, we'll look into it and see if we can use the same approach with djangos database wrapping or if we can use it directly.

···

On 09/02/17 12:08, Matt Yule-Bennett wrote:

Thanks :slight_smile: Glad you like it.

Nameko uses a pool of worker threads to limit the amount of concurrent work. Its size is controlled by the `max_workers` config key (which must be lowercase). The default value is 10, so you should only be seeing at most that many workers per service instance.

Are you using GitHub - nameko/nameko-sqlalchemy: SQLAlchemy dependency for nameko services? If so, sqlalchemy will take care of pooling connections for you.

On Thursday, February 9, 2017 at 9:31:48 AM UTC, mat...@kundo.se wrote:

    Hi,

    Thanks for a nice library, it have helped us extend our
    application in a nice way!

    We're breaking out parts of our application (built in Django) and
    are using nameko to do communication between the old application
    and newer services.

    A problem we have started to experience is that nameko will
    happily handle 100s of messages in parallell (on multiple servers)
    and start new connections to our SQL database. The database max
    connection limit will kick in and block any new connection
    attempts (which in itself isn't a problem).

    What's the best approach to limit the number of parallell messages
    handled by nameko?
    Is there a way to do connectionpooling against the database? (With
    or without using django)