Dependency on a Nameko Service / Registration with another Service on Startup

googlegroup
1

Hello,

We have a service whose purpose is to sort of oversee and monitor other
services. It keeps track of updates and information from other services. We
want each service in our system to register with this monitor service on
startup.

Specifically, every service should have a uuid to identify it so we can
group updates and information together and associate them with a particular
run of a service. The uuid should be specific to a run of a service, so if
we restart it, the uuid should change, and if we start a second instance of
that service on a different (or even the same) server, it would also have a
different uuid.

We can generate a uuid and store it as an attribute on the service class,
and report it when requested, but we're looking for a way to report it on
startup. For example:
MonitorService is running
ServiceA is not running
ServiceA is invoked (using `nameko run servicea` or a more manual run
process)
ServiceA establishes its dependencies and entrypoints and is ready to
listen on queues, and has generated a uuid for itself
ServiceA makes an RPC (or publishes to a queue, etc.) to MonitorService to
report its uuid and indicate that it's online
ServiceA processes requests as normal
At some point in the future we'd also like:
ServiceA is told to gracefully shut down
ServiceA makes an RPC (or publishes to a queue, etc.) to MonitorService to
indicate that it's going offline (and supplies its uuid so MonitorService
knows who's going down)

We've also considered having MonitorService generate the uuid and return it
to ServiceA in a response to the startup request, but we weren't sure how
to get ServiceA to store it as an attribute that all future workers would
inherit, and that seemed like the wrong approach anyway.

We tried using a generic publisher to publish to a queue that the monitor
service is listening to in the service's __init__() or in a dependency
provider, but it didn't like that and we don't want to register with every
worker, just on the service setup.

We have a method listening on an event broadcast that all services will
respond to, and we can set that event on a timer, but again, we'd like a
service to announce itself rather than wait to be pinged. (It might come
online and go offline between firings of the timer and we wouldn't know.)

We're considering writing our own run script and wrapping the
ServiceContainer in order to make a REST call during start() and stop().

We've looked at nameko.testing.services @once, nameko.extensions.Extension,
and nameko.extensions.DependencyProvider among other things. It looks like
Extension should allow us to do what we want, namely make a call when the
service is ready to go and also when it's shutting down, but I haven't
figured out how to get this to work. @once is part of a testing module (so
probably not meant for production) and DependencyProvider seems like its
meant for non-nameko dependencies, so I'm not really sure which direction
we should be pursuing.

TL;DR:
How would you recommend we make a nameko service depend on another nameko
service?

2

Followup: I was using the wrong amqp uri in my publisher and that was
making me think certain things weren't working when they otherwise would've.

So we have services publishing to a queue on startup using @once, but I
wonder if we should still be trying to use Extension so we can publish
messages both on start and on stop. Is this a valid use case for extending
Extension?

···

On Tuesday, January 23, 2018 at 11:34:19 AM UTC-7, Brooke Lynne Weaver Skousen wrote:

Hello,

We have a service whose purpose is to sort of oversee and monitor other
services. It keeps track of updates and information from other services. We
want each service in our system to register with this monitor service on
startup.

Specifically, every service should have a uuid to identify it so we can
group updates and information together and associate them with a particular
run of a service. The uuid should be specific to a run of a service, so if
we restart it, the uuid should change, and if we start a second instance of
that service on a different (or even the same) server, it would also have a
different uuid.

We can generate a uuid and store it as an attribute on the service class,
and report it when requested, but we're looking for a way to report it on
startup. For example:
MonitorService is running
ServiceA is not running
ServiceA is invoked (using `nameko run servicea` or a more manual run
process)
ServiceA establishes its dependencies and entrypoints and is ready to
listen on queues, and has generated a uuid for itself
ServiceA makes an RPC (or publishes to a queue, etc.) to MonitorService to
report its uuid and indicate that it's online
ServiceA processes requests as normal
At some point in the future we'd also like:
ServiceA is told to gracefully shut down
ServiceA makes an RPC (or publishes to a queue, etc.) to MonitorService to
indicate that it's going offline (and supplies its uuid so MonitorService
knows who's going down)

We've also considered having MonitorService generate the uuid and return
it to ServiceA in a response to the startup request, but we weren't sure
how to get ServiceA to store it as an attribute that all future workers
would inherit, and that seemed like the wrong approach anyway.

We tried using a generic publisher to publish to a queue that the monitor
service is listening to in the service's __init__() or in a dependency
provider, but it didn't like that and we don't want to register with every
worker, just on the service setup.

We have a method listening on an event broadcast that all services will
respond to, and we can set that event on a timer, but again, we'd like a
service to announce itself rather than wait to be pinged. (It might come
online and go offline between firings of the timer and we wouldn't know.)

We're considering writing our own run script and wrapping the
ServiceContainer in order to make a REST call during start() and stop().

We've looked at nameko.testing.services @once,
nameko.extensions.Extension, and nameko.extensions.DependencyProvider among
other things. It looks like Extension should allow us to do what we want,
namely make a call when the service is ready to go and also when it's
shutting down, but I haven't figured out how to get this to work. @once is
part of a testing module (so probably not meant for production) and
DependencyProvider seems like its meant for non-nameko dependencies, so I'm
not really sure which direction we should be pursuing.

TL;DR:
How would you recommend we make a nameko service depend on another nameko
service?

3

A custom ServiceContainer is probably the easiest solve. You can actually
specify a custom ServiceContainer class via config
<https://github.com/nameko/nameko/blob/0d8239a1580b1879c872c77dfcdba601040e064f/nameko/containers.py#L50-L52&gt;\.
Just overload start() and stop() to register and de-register from your
MonitorService.

Using an Extension is another valid approach, again hooking in to the
start() and stop() methods. One thing to note though is that Extensions can
only be attached to DependencyProviders or Entrypoints and aren't
discovered when attached directly to service classes (which is confusing
and probably poor design). Best illustrated with an example:

from nameko.extensions import Extension, DependencyProvider
from nameko.rpc import rpc

class MyExt(Extension):

    def start(self):
        print("start ext")

    def stop(self):
        print("stop ext")

class MyDP(DependencyProvider):

    def start(self):
        print("start dp")

    def stop(self):
        print("stop dp")

class Service:
    name = "service"

    ext = MyExt() # effectively ignored
    dp = MyDP()

    @rpc
    def method(self):
        pass

You only see "start dp" and "stop dp" when you start and stop this service.

Even though you don't need to inject any dependency into your service
class, you could still use a DependencyProvider to hook in to the service
start/stop and register with the MonitorService.

I prefer the ServiceContainer approach though, because this registration
process is part of the *container* lifecycle rather than the *worker*
lifecycle.

···

On Tuesday, January 23, 2018 at 6:34:19 PM UTC, Brooke Lynne Weaver Skousen wrote:

Hello,

We have a service whose purpose is to sort of oversee and monitor other
services. It keeps track of updates and information from other services. We
want each service in our system to register with this monitor service on
startup.

Specifically, every service should have a uuid to identify it so we can
group updates and information together and associate them with a particular
run of a service. The uuid should be specific to a run of a service, so if
we restart it, the uuid should change, and if we start a second instance of
that service on a different (or even the same) server, it would also have a
different uuid.

We can generate a uuid and store it as an attribute on the service class,
and report it when requested, but we're looking for a way to report it on
startup. For example:
MonitorService is running
ServiceA is not running
ServiceA is invoked (using `nameko run servicea` or a more manual run
process)
ServiceA establishes its dependencies and entrypoints and is ready to
listen on queues, and has generated a uuid for itself
ServiceA makes an RPC (or publishes to a queue, etc.) to MonitorService to
report its uuid and indicate that it's online
ServiceA processes requests as normal
At some point in the future we'd also like:
ServiceA is told to gracefully shut down
ServiceA makes an RPC (or publishes to a queue, etc.) to MonitorService to
indicate that it's going offline (and supplies its uuid so MonitorService
knows who's going down)

We've also considered having MonitorService generate the uuid and return
it to ServiceA in a response to the startup request, but we weren't sure
how to get ServiceA to store it as an attribute that all future workers
would inherit, and that seemed like the wrong approach anyway.

We tried using a generic publisher to publish to a queue that the monitor
service is listening to in the service's __init__() or in a dependency
provider, but it didn't like that and we don't want to register with every
worker, just on the service setup.

We have a method listening on an event broadcast that all services will
respond to, and we can set that event on a timer, but again, we'd like a
service to announce itself rather than wait to be pinged. (It might come
online and go offline between firings of the timer and we wouldn't know.)

We're considering writing our own run script and wrapping the
ServiceContainer in order to make a REST call during start() and stop().

We've looked at nameko.testing.services @once,
nameko.extensions.Extension, and nameko.extensions.DependencyProvider among
other things. It looks like Extension should allow us to do what we want,
namely make a call when the service is ready to go and also when it's
shutting down, but I haven't figured out how to get this to work. @once is
part of a testing module (so probably not meant for production) and
DependencyProvider seems like its meant for non-nameko dependencies, so I'm
not really sure which direction we should be pursuing.

TL;DR:
How would you recommend we make a nameko service depend on another nameko
service?

4

This is unbelievably helpful, thank you so much. A custom ServiceContainer
is exactly what we were looking for. It's the perfect place to apply our
service wrapper too. Thank you for your quick response.

···

On Wednesday, January 24, 2018 at 5:01:06 AM UTC-7, Matt Yule-Bennett wrote:

A custom ServiceContainer is probably the easiest solve. You can actually
specify a custom ServiceContainer class via config
<https://github.com/nameko/nameko/blob/0d8239a1580b1879c872c77dfcdba601040e064f/nameko/containers.py#L50-L52&gt;\.
Just overload start() and stop() to register and de-register from your
MonitorService.

Using an Extension is another valid approach, again hooking in to the
start() and stop() methods. One thing to note though is that Extensions can
only be attached to DependencyProviders or Entrypoints and aren't
discovered when attached directly to service classes (which is confusing
and probably poor design). Best illustrated with an example:

from nameko.extensions import Extension, DependencyProvider
from nameko.rpc import rpc

class MyExt(Extension):

    def start(self):
        print("start ext")

    def stop(self):
        print("stop ext")

class MyDP(DependencyProvider):

    def start(self):
        print("start dp")

    def stop(self):
        print("stop dp")

class Service:
    name = "service"

    ext = MyExt() # effectively ignored
    dp = MyDP()

    @rpc
    def method(self):
        pass

You only see "start dp" and "stop dp" when you start and stop this
service.

Even though you don't need to inject any dependency into your service
class, you could still use a DependencyProvider to hook in to the service
start/stop and register with the MonitorService.

I prefer the ServiceContainer approach though, because this registration
process is part of the *container* lifecycle rather than the *worker*
lifecycle.

On Tuesday, January 23, 2018 at 6:34:19 PM UTC, Brooke Lynne Weaver > Skousen wrote:

Hello,

We have a service whose purpose is to sort of oversee and monitor other
services. It keeps track of updates and information from other services. We
want each service in our system to register with this monitor service on
startup.

Specifically, every service should have a uuid to identify it so we can
group updates and information together and associate them with a particular
run of a service. The uuid should be specific to a run of a service, so if
we restart it, the uuid should change, and if we start a second instance of
that service on a different (or even the same) server, it would also have a
different uuid.

We can generate a uuid and store it as an attribute on the service class,
and report it when requested, but we're looking for a way to report it on
startup. For example:
MonitorService is running
ServiceA is not running
ServiceA is invoked (using `nameko run servicea` or a more manual run
process)
ServiceA establishes its dependencies and entrypoints and is ready to
listen on queues, and has generated a uuid for itself
ServiceA makes an RPC (or publishes to a queue, etc.) to MonitorService
to report its uuid and indicate that it's online
ServiceA processes requests as normal
At some point in the future we'd also like:
ServiceA is told to gracefully shut down
ServiceA makes an RPC (or publishes to a queue, etc.) to MonitorService
to indicate that it's going offline (and supplies its uuid so
MonitorService knows who's going down)

We've also considered having MonitorService generate the uuid and return
it to ServiceA in a response to the startup request, but we weren't sure
how to get ServiceA to store it as an attribute that all future workers
would inherit, and that seemed like the wrong approach anyway.

We tried using a generic publisher to publish to a queue that the monitor
service is listening to in the service's __init__() or in a dependency
provider, but it didn't like that and we don't want to register with every
worker, just on the service setup.

We have a method listening on an event broadcast that all services will
respond to, and we can set that event on a timer, but again, we'd like a
service to announce itself rather than wait to be pinged. (It might come
online and go offline between firings of the timer and we wouldn't know.)

We're considering writing our own run script and wrapping the
ServiceContainer in order to make a REST call during start() and stop().

We've looked at nameko.testing.services @once,
nameko.extensions.Extension, and nameko.extensions.DependencyProvider among
other things. It looks like Extension should allow us to do what we want,
namely make a call when the service is ready to go and also when it's
shutting down, but I haven't figured out how to get this to work. @once is
part of a testing module (so probably not meant for production) and
DependencyProvider seems like its meant for non-nameko dependencies, so I'm
not really sure which direction we should be pursuing.

TL;DR:
How would you recommend we make a nameko service depend on another nameko
service?