# Calling an RPC service within a dependency

**URL:** <https://discourse.nameko.io/t/calling-an-rpc-service-within-a-dependency/231>\
**Category:** googlegroup\
**Created:** [January 13, 2018, 7:24pm UTC](https://discourse.nameko.io/t/calling-an-rpc-service-within-a-dependency/231 "2018-01-13T19:24:35Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Chris\_Platts](https://avatars.discourse-cdn.com/v4/letter/c/ec9cab/32.png) [@Chris\_Platts](https://discourse.nameko.io/u/Chris_Platts)\
**Post date:** [January 13, 2018, 7:24pm UTC](https://discourse.nameko.io/t/calling-an-rpc-service-within-a-dependency/231/1 "2018-01-13T19:24:35Z")

</div>

Hi,

I have a LogService which receives log and exception info via RPC in the  
Graylog GELF format and then posts it to Graylog via AMQP.

Now, I'm trying to write a dependency for my other services. The idea is  
that this dependency provides methods to log at various levels (e.g.  
log\_info(msg), log\_warn(msg) etc., and also collects exception info (via  
worker\_result). The dependency would then build an appropriate GELF object  
and send it to the log service via RPC.

I'm struggling making the call to the @rpc entrypoint from inside the  
dependency. Any tips appreciated!

Thanks,  
Chris

---

<div class="post-metadata">

**Author:** ![mattbennett](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/mattbennett/32/13_2.png) [@mattbennett](https://discourse.nameko.io/u/mattbennett)\
**Post date:** [January 13, 2018, 8:13pm UTC](https://discourse.nameko.io/t/calling-an-rpc-service-within-a-dependency/231/2 "2018-01-13T20:13:23Z")

</div>

Nice. I've been waiting for someone to write a GELF DependencyProvider 🙂

Is there a reason the DP has to make RPC calls to your logging service  
rather than just publishing AMQP messages directly?

You may also want to take a look  
at [GitHub - Overseas-Student-Living/nameko-tracer](https://github.com/Overseas-Student-Living/nameko-tracer), which you  
could probably use by simply plugging in a GELF log handler.

> **···**
>
> On Saturday, January 13, 2018 at 7:24:35 PM UTC, Chris Platts wrote:
> 
> > Hi,
> > 
> > I have a LogService which receives log and exception info via RPC in the  
> > Graylog GELF format and then posts it to Graylog via AMQP.
> > 
> > Now, I'm trying to write a dependency for my other services. The idea is  
> > that this dependency provides methods to log at various levels (e.g.  
> > log\_info(msg), log\_warn(msg) etc., and also collects exception info (via  
> > worker\_result). The dependency would then build an appropriate GELF object  
> > and send it to the log service via RPC.
> > 
> > I'm struggling making the call to the @rpc entrypoint from inside the  
> > dependency. Any tips appreciated!
> > 
> > Thanks,  
> > Chris

---

<div class="post-metadata">

**Author:** ![Chris\_Platts](https://avatars.discourse-cdn.com/v4/letter/c/ec9cab/32.png) [@Chris\_Platts](https://discourse.nameko.io/u/Chris_Platts)\
**Post date:** [January 13, 2018, 8:48pm UTC](https://discourse.nameko.io/t/calling-an-rpc-service-within-a-dependency/231/3 "2018-01-13T20:48:48Z")

</div>

Thanks, Matt -- I'll take a look through the tracer to see how it works.

I suppose there's no harm putting the GELF-over-AMQP stuff into the DP  
rather than a separate service. Would I still be able to use  
nameko.messaging.Publisher inside a DP to handle the AMQP stuff, or would  
an AMQP client have to be handled in the DP's own code?

Thanks,  
Chris

---

<div class="post-metadata">

**Author:** ![Brooke\_Lynne\_Weaver](https://avatars.discourse-cdn.com/v4/letter/b/e68b1a/32.png) [@Brooke\_Lynne\_Weaver](https://discourse.nameko.io/u/Brooke_Lynne_Weaver)\
**Post date:** [January 15, 2018, 11:55pm UTC](https://discourse.nameko.io/t/calling-an-rpc-service-within-a-dependency/231/4 "2018-01-15T23:55:48Z")

</div>

Hi, Matt

I'm in a similar situation where I'm trying to capture all logging output  
(including from nameko-tracer) and send it to a specific service, although  
this service writes it to a PostgreSQL db rather than sending it to Graylog.

Each of our databases is 'owned' by a single service, so in theory all logs  
would be inserted into the logging (service information) database through  
the logging (service information) service, because the service owns that  
data domain and handles all interactions with that database.

Is there a good way to configure the logging to send all logs to a queue  
that can be processed by a service? Is there a better approach?

(I'm currently working off the example you've provided for sending to  
Graylog, but I wondered if you had any opinions on sending traffic to a  
nameko service.)

Thank you for your help

> **···**
>
> On Saturday, January 13, 2018 at 1:13:23 PM UTC-7, Matt Yule-Bennett wrote:
> 
> > Nice. I've been waiting for someone to write a GELF DependencyProvider 🙂
> > 
> > Is there a reason the DP has to make RPC calls to your logging service  
> > rather than just publishing AMQP messages directly?
> > 
> > You may also want to take a look at  
> > [GitHub - Overseas-Student-Living/nameko-tracer](https://github.com/Overseas-Student-Living/nameko-tracer), which you could  
> > probably use by simply plugging in a GELF log handler.
> > 
> > On Saturday, January 13, 2018 at 7:24:35 PM UTC, Chris Platts wrote:
> > 
> > > Hi,
> > > 
> > > I have a LogService which receives log and exception info via RPC in the  
> > > Graylog GELF format and then posts it to Graylog via AMQP.
> > > 
> > > Now, I'm trying to write a dependency for my other services. The idea is  
> > > that this dependency provides methods to log at various levels (e.g.  
> > > log\_info(msg), log\_warn(msg) etc., and also collects exception info (via  
> > > worker\_result). The dependency would then build an appropriate GELF object  
> > > and send it to the log service via RPC.
> > > 
> > > I'm struggling making the call to the @rpc entrypoint from inside the  
> > > dependency. Any tips appreciated!
> > > 
> > > Thanks,  
> > > Chris

---

<div class="post-metadata">

**Author:** ![Chris\_Platts](https://avatars.discourse-cdn.com/v4/letter/c/ec9cab/32.png) [@Chris\_Platts](https://discourse.nameko.io/u/Chris_Platts)\
**Post date:** [January 13, 2018, 10:27pm UTC](https://discourse.nameko.io/t/calling-an-rpc-service-within-a-dependency/231/5 "2018-01-13T22:27:39Z")

</div>

...answered my own question - looks like dependencies are injected into  
services, but not each other. Naively trying to use Publisher() in my  
Logger DP leads to:

`TypeError: bind() missing 1 required positional argument: 'attr_name'`

So would I be better off importing ampqlib or somesuch and using it  
directly in the DP?

---

<div class="post-metadata">

**Author:** ![mattbennett](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/mattbennett/32/13_2.png) [@mattbennett](https://discourse.nameko.io/u/mattbennett)\
**Post date:** [January 16, 2018, 7:20am UTC](https://discourse.nameko.io/t/calling-an-rpc-service-within-a-dependency/231/6 "2018-01-16T07:20:59Z")

</div>

Hi Brooke,

I would recommend almost the same solution as suggested above. In your  
setup though, the AMQP publisher would publish messages to be consumed by  
your logging service, rather than a Graylog consumer. You can use the  
Consumer entrypoint in nameko.messaging to consume messages from an  
arbitrary queue, and then simply write them to your logging database. This  
would be asynchronous messaging rather than RPC, but I think in this  
logging scenario a reply is not desired.

Incidentally, the above pattern is exactly what we did at Student.com for a  
while -- using a separate Nameko service to consume AMQP messages and write  
to a database (Elasticsearch in our case). We have since ditched the  
intermediate service in favour of writing to ES directly from the tracer.  
While I strongly advocate the one-database-per-service rule for \*domain\*  
data, I think logging is a separate concern and it's fine for the  
repository of those logs to be shared. This is just my opinion with no  
knowledge of what your logging / service-information service is actually  
doing though, so take it with a pinch of salt.

Hope that helps,  
Matt.

> **···**
>
> On Monday, January 15, 2018 at 11:55:48 PM UTC, Brooke Lynne Weaver Skousen wrote:
> 
> > Hi, Matt
> > 
> > I'm in a similar situation where I'm trying to capture all logging output  
> > (including from nameko-tracer) and send it to a specific service, although  
> > this service writes it to a PostgreSQL db rather than sending it to Graylog.
> > 
> > Each of our databases is 'owned' by a single service, so in theory all  
> > logs would be inserted into the logging (service information) database  
> > through the logging (service information) service, because the service owns  
> > that data domain and handles all interactions with that database.
> > 
> > Is there a good way to configure the logging to send all logs to a queue  
> > that can be processed by a service? Is there a better approach?
> > 
> > (I'm currently working off the example you've provided for sending to  
> > Graylog, but I wondered if you had any opinions on sending traffic to a  
> > nameko service.)
> > 
> > Thank you for your help
> > 
> > On Saturday, January 13, 2018 at 1:13:23 PM UTC-7, Matt Yule-Bennett wrote:
> > 
> > > Nice. I've been waiting for someone to write a GELF DependencyProvider 🙂
> > > 
> > > Is there a reason the DP has to make RPC calls to your logging service  
> > > rather than just publishing AMQP messages directly?
> > > 
> > > You may also want to take a look at  
> > > [GitHub - Overseas-Student-Living/nameko-tracer](https://github.com/Overseas-Student-Living/nameko-tracer), which you  
> > > could probably use by simply plugging in a GELF log handler.
> > > 
> > > On Saturday, January 13, 2018 at 7:24:35 PM UTC, Chris Platts wrote:
> > > 
> > > > Hi,
> > > > 
> > > > I have a LogService which receives log and exception info via RPC in the  
> > > > Graylog GELF format and then posts it to Graylog via AMQP.
> > > > 
> > > > Now, I'm trying to write a dependency for my other services. The idea  
> > > > is that this dependency provides methods to log at various levels (e.g.  
> > > > log\_info(msg), log\_warn(msg) etc., and also collects exception info (via  
> > > > worker\_result). The dependency would then build an appropriate GELF object  
> > > > and send it to the log service via RPC.
> > > > 
> > > > I'm struggling making the call to the @rpc entrypoint from inside the  
> > > > dependency. Any tips appreciated!
> > > > 
> > > > Thanks,  
> > > > Chris

---

<div class="post-metadata">

**Author:** ![mattbennett](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/mattbennett/32/13_2.png) [@mattbennett](https://discourse.nameko.io/u/mattbennett)\
**Post date:** [January 14, 2018, 11:22am UTC](https://discourse.nameko.io/t/calling-an-rpc-service-within-a-dependency/231/7 "2018-01-14T11:22:48Z")

</div>

> ...answered my own question - looks like dependencies are injected into  
> services, but not each other. Naively trying to use Publisher() in my  
> Logger DP leads to:
> 
> `TypeError: bind() missing 1 required positional argument: 'attr_name'`

Right, DependencyProviders are specifically about injecting dependencies  
into Nameko services -- you can't put a DP inside another DP.

> So would I be better off importing ampqlib or somesuch and using it  
> directly in the DP?

Actually if you look at the nameko.messaging.Publisher DP you'll see that  
(apart from inheriting from DependencyProvider and implementing  
get\_dependency()) it doesn't do very much except create an instance of  
nameko.amqp.publish.Publisher -- this is a utility class that makes it  
easier to publish messages to RabbitMQ. I would use this, rather than  
amqplib or kombu directly.

When I directed you to nameko-tracer I forgot that the critical part of  
actually publishing messages to AMQP was spun into another (currently  
closed-source) library. If you choose to use the tracer, you'll want to  
plug the AMQP publishing part in as a Python logging handler.

Here's a skeleton implementation:

import logging

from nameko.amqp.publish import Publisher

logger = logging.getLogger(\_\_name\_\_)

class PublisherHandler(logging.Handler):  
&nbsp;&nbsp;&nbsp;&nbsp;""" Handler for publishing trace messages to RabbitMQ  
&nbsp;&nbsp;&nbsp;&nbsp;"""  
&nbsp;&nbsp;&nbsp;&nbsp;def \_\_init\_\_(  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self, amqp\_uri, exchange\_name, routing\_key,  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;serializer=None, content\_type=None  
&nbsp;&nbsp;&nbsp;&nbsp;):  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.publisher = Publisher(  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;amqp\_uri,  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;exchange=exchange\_name,  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;routing\_key=routing\_key,  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;serializer=serializer,  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;content\_type=content\_type  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;super(PublisherHandler, self).\_\_init\_\_()

&nbsp;&nbsp;&nbsp;&nbsp;def emit(self, log\_record):  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;try:  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.publisher.publish(self.format(log\_record))  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;except Exception:  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.handleError(log\_record)

Use by registering it as a handler using the normal logging configuration:

LOGGING:  
&nbsp;&nbsp;formatters:  
&nbsp;&nbsp;&nbsp;&nbsp;tracer:  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(): yourib.logging.formatters.GELFFormatter  
&nbsp;&nbsp;handlers:  
&nbsp;&nbsp;&nbsp;&nbsp;tracer:  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;formatter: tracer  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(): yourlib.logging.handlers.PublisherHandler  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;amqp\_uri: \<GELF\_AMQP\_URI\>  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;exchange\_name: \<GELF\_EXCHANGE\_NAME\>  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;routing\_key: \<GELF\_ROUTING\_KEY\>  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;serializer: json  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;content\_type: application/json  
&nbsp;&nbsp;loggers:  
&nbsp;&nbsp;&nbsp;&nbsp;nameko\_tracer:  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;level: INFO  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;handlers:  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- tracer

I would personally do it this way, implementing the GELF format as a  
standard logging formatter too. If there's anything required by Graylog  
which is not already collected by the tracer, I'm sure a PR to  
nameko-tracer would be gratefully received.

> **···**
>
> On Saturday, 13 January 2018 22:27:39 UTC, Chris Platts wrote:

---

<div class="post-metadata">

**Author:** ![Brooke\_Lynne\_Weaver](https://avatars.discourse-cdn.com/v4/letter/b/e68b1a/32.png) [@Brooke\_Lynne\_Weaver](https://discourse.nameko.io/u/Brooke_Lynne_Weaver)\
**Post date:** [January 19, 2018, 7:31pm UTC](https://discourse.nameko.io/t/calling-an-rpc-service-within-a-dependency/231/8 "2018-01-19T19:31:54Z")

</div>

I forgot to respond a couple days ago; this helped a lot, thank you. We've  
got it working exactly how we want it now. I appreciate your input.

> **···**
>
> On Tuesday, January 16, 2018 at 12:20:59 AM UTC-7, Matt Yule-Bennett wrote:
> 
> > Hi Brooke,
> > 
> > I would recommend almost the same solution as suggested above. In your  
> > setup though, the AMQP publisher would publish messages to be consumed by  
> > your logging service, rather than a Graylog consumer. You can use the  
> > Consumer entrypoint in nameko.messaging to consume messages from an  
> > arbitrary queue, and then simply write them to your logging database. This  
> > would be asynchronous messaging rather than RPC, but I think in this  
> > logging scenario a reply is not desired.
> > 
> > Incidentally, the above pattern is exactly what we did at Student.com for  
> > a while -- using a separate Nameko service to consume AMQP messages and  
> > write to a database (Elasticsearch in our case). We have since ditched the  
> > intermediate service in favour of writing to ES directly from the tracer.  
> > While I strongly advocate the one-database-per-service rule for \*domain\*  
> > data, I think logging is a separate concern and it's fine for the  
> > repository of those logs to be shared. This is just my opinion with no  
> > knowledge of what your logging / service-information service is actually  
> > doing though, so take it with a pinch of salt.
> > 
> > Hope that helps,  
> > Matt.
> > 
> > On Monday, January 15, 2018 at 11:55:48 PM UTC, Brooke Lynne Weaver \> Skousen wrote:
> > 
> > > Hi, Matt
> > > 
> > > I'm in a similar situation where I'm trying to capture all logging output  
> > > (including from nameko-tracer) and send it to a specific service, although  
> > > this service writes it to a PostgreSQL db rather than sending it to Graylog.
> > > 
> > > Each of our databases is 'owned' by a single service, so in theory all  
> > > logs would be inserted into the logging (service information) database  
> > > through the logging (service information) service, because the service owns  
> > > that data domain and handles all interactions with that database.
> > > 
> > > Is there a good way to configure the logging to send all logs to a queue  
> > > that can be processed by a service? Is there a better approach?
> > > 
> > > (I'm currently working off the example you've provided for sending to  
> > > Graylog, but I wondered if you had any opinions on sending traffic to a  
> > > nameko service.)
> > > 
> > > Thank you for your help
> > > 
> > > On Saturday, January 13, 2018 at 1:13:23 PM UTC-7, Matt Yule-Bennett \>\> wrote:
> > > 
> > > > Nice. I've been waiting for someone to write a GELF DependencyProvider 🙂
> > > > 
> > > > Is there a reason the DP has to make RPC calls to your logging service  
> > > > rather than just publishing AMQP messages directly?
> > > > 
> > > > You may also want to take a look at  
> > > > [GitHub - Overseas-Student-Living/nameko-tracer](https://github.com/Overseas-Student-Living/nameko-tracer), which you  
> > > > could probably use by simply plugging in a GELF log handler.
> > > > 
> > > > On Saturday, January 13, 2018 at 7:24:35 PM UTC, Chris Platts wrote:
> > > > 
> > > > > Hi,
> > > > > 
> > > > > I have a LogService which receives log and exception info via RPC in  
> > > > > the Graylog GELF format and then posts it to Graylog via AMQP.
> > > > > 
> > > > > Now, I'm trying to write a dependency for my other services. The idea  
> > > > > is that this dependency provides methods to log at various levels (e.g.  
> > > > > log\_info(msg), log\_warn(msg) etc., and also collects exception info (via  
> > > > > worker\_result). The dependency would then build an appropriate GELF object  
> > > > > and send it to the log service via RPC.
> > > > > 
> > > > > I'm struggling making the call to the @rpc entrypoint from inside the  
> > > > > dependency. Any tips appreciated!
> > > > > 
> > > > > Thanks,  
> > > > > Chris

---

<div class="post-metadata">

**Author:** ![Chris\_Platts](https://avatars.discourse-cdn.com/v4/letter/c/ec9cab/32.png) [@Chris\_Platts](https://discourse.nameko.io/u/Chris_Platts)\
**Post date:** [January 29, 2018, 11:10am UTC](https://discourse.nameko.io/t/calling-an-rpc-service-within-a-dependency/231/9 "2018-01-29T11:10:16Z")

</div>

...and I forgot too 🙂

Ended up using Publisher within a DP. The DP does a handful of things:

\* Builds GELF messages with a few standard extra properties -- e.g.  
\_serviceName, \_callID, \_hostName, etc.  
\* Logs service lifecycle events via GELF-over-AMQP  
\* Provides log.info(), log.debug(), log.error(), log.warn(), log.fatal()  
functions to the service code.

However, I'm wondering if I should copy trace's approach and use Python's  
logging library. That would give config-file based setup of targets and  
levels. However, using Graylog's making me think that perhaps I shouldn't  
worry about such things -- I'm quite happy for all log data to always be  
sent.

(we're using Nameko due to the architectural advantages, not because we  
handle hundreds of calls per second, so log data is a stream rather than a  
flood!)

> **···**
>
> On Friday, 19 January 2018 19:31:54 UTC, Brooke Lynne Weaver Skousen wrote:
> 
> > I forgot to respond a couple days ago; this helped a lot, thank you. We've  
> > got it working exactly how we want it now. I appreciate your input.

---

<div class="post-metadata">

**Author:** ![Chris\_Platts](https://avatars.discourse-cdn.com/v4/letter/c/ec9cab/32.png) [@Chris\_Platts](https://discourse.nameko.io/u/Chris_Platts)\
**Post date:** [February 22, 2018, 8:40pm UTC](https://discourse.nameko.io/t/calling-an-rpc-service-within-a-dependency/231/10 "2018-02-22T20:40:46Z")

</div>

Well, this was all going well until it wasn't!

I think I'm being naive when creating the kombu AMQP connection/queue in  
the logging dependency. Once we have several workers active, we start  
getting exceptions regarding multiple writes to a file descriptor  
(apologies, I'm paraphrasing).

The dependecy's creating a single queue in its setup() method, which I now  
realise is being shared by all workers. I think the right thing to do is  
refactor this so that a queue connection is created for each worker.  
Perhaps set up the connection in \*worker\_setup() \*and store it in a shared  
dictionary for inclusion in the object returned by \*get\_dependency()\* and  
then cleaned-up in \*worker\_result()\*.

I'll see how that goes!

---

<div class="post-metadata">

**Author:** ![mattbennett](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/mattbennett/32/13_2.png) [@mattbennett](https://discourse.nameko.io/u/mattbennett)\
**Post date:** [February 25, 2018, 7:41am UTC](https://discourse.nameko.io/t/calling-an-rpc-service-within-a-dependency/231/11 "2018-02-25T07:41:32Z")

</div>

> Well, this was all going well until it wasn't!
> 
> I think I'm being naive when creating the kombu AMQP connection/queue in  
> the logging dependency. Once we have several workers active, we start  
> getting exceptions regarding multiple writes to a file descriptor  
> (apologies, I'm paraphrasing).

"Second simultaneous write detected"?

This is Eventlet protecting you from multiple threads writing to the same  
file descriptor (probably a socket in this case) at the same time. As  
per [Writing Extensions — nameko 2.12.0 documentation](http://nameko.readthedocs.io/en/stable/writing_extensions.html#concurrency-and-thread-safety),  
whatever is returned from get\_dependency() needs to be safe for concurrent  
access by multiple worker threads.

> The dependecy's creating a single queue in its setup() method, which I now  
> realise is being shared by all workers.

In AMQP you publish to \*exchanges\* rather than queues. RabbitMQ has a  
shortcut that allows you to publish a message into a named queue, but  
really what it's doing is publishing to the default exchange, which is  
implicitly bound to every queue by their name.

So the shared object is not the queue, but perhaps the connection being  
used by your publisher.

What's weird is that the built-in Nameko Publisher and EventDispatcher use  
a publisher shared between worker threads and don't have any protection  
against multiple access. This is fine if the write to the socket is atomic  
and I've never seen the lack of protection cause a problem.

Are you sending particularly large payloads?

> I think the right thing to do is refactor this so that a queue connection  
> is created for each worker. Perhaps set up the connection in \*worker\_setup()  
> \*and store it in a shared dictionary for inclusion in the object returned  
> by \*get\_dependency()\* and then cleaned-up in \*worker\_result()\*.

This would work, but it would be slow because you're creating a new  
connection every time you publish a message. If you're going to do that you  
may as well ignore worker\_setup() and worker\_teardown() and just create  
the connection in get\_dependency() or lazily when you publish. A more  
efficient way would be to project the publisher with a lock, to serialise  
the concurrent access.

> I'll see how that goes!

If you can post some code here and/or a stack trace we might be able to  
figure out what's going wrong. It may be a platform specific problem.

> **···**
>
> On Thursday, February 22, 2018 at 8:40:46 PM UTC, Chris Platts wrote: