Orchestrating a set of services

Hi all,

Having attained the status of 'reasonably competent, unlikely to blow
things up' with nameko, I'm thinking about how to reimplement some of my
company's systems in a nice microservice-y way.

We have a bunch of actions, file transfers, zip/unzip operations, data
imports, data processing, etc. These actions are chained together in
specific ways for each of our customers.

Now, if these operations were implemented as nameko services, I'd need some
way of storing the 'plan' for each of our customers and executing that plan
as needed.

What I'm thinking of is this:

A *Control* service which owns a database containing the client-specific
plan. The service is told to execute the plan for client 'A'. It consults
the plan and triggers the first service action in the plan. The service
(and all others) will end with a call back to the Control service to
indicate success/failure/etc. The Control service then kicks off the next
service in the plan. Lather, rinse, repeat until the plan is complete.

This appears to be a decent enough solution. It would give the ability to
resume a plan from a failed step, plus there could be methods in the
Control service to do things like finding out how far a plan has progressed.

So, a) does this sound sane?, and b) has anyone done something similar with
success?

Cheers,
Chris

Hi Chris,

Sorry for not replying to this sooner.

Quick answers:

a) Yes
b) Kind of. We distribute work but don't have a workflow that waits for
results before moving on to the next step.

My recommendation would be to use asynchronous messaging for the
communication between the Controller and the downstream services, i.e. have
the downstream services call back when they're done, rather than using RPC
and waiting for a reply in the controller.

Our task distribution is built using the nameko.messaging.Consumer and
Publisher, and we used custom Nameko extensions to wrap them and add
syntactic sugar. The API looks like:

class Worker:
    name = "worker"

    @task
    def some_task(self, *args):
        return "result"

class Controller:
    name = "controller"
    
    schedule_task = TaskScheduler()

    @entrypoint
    def initiate_work(self):
        self.schedule_task("some_task", "arg1", "arg2")

You could expand this concept and have the task results routed back to a
different entrypoint on the controller:

class Worker:
    name = "worker"

    @task
    def some_task(self, *args):
        return "result"

class Controller:
    name = "controller"
    
    schedule_task = TaskScheduler()

    @entrypoint
    def initiate_work(self):
        task_id = self.schedule_task("some_task", "arg1", "arg2")
        ...

    @task_result
    def handle_result(self, task_id, result):
        ...

···

On Tuesday, February 6, 2018 at 9:59:31 AM UTC, Chris Platts wrote:

Hi all,

Having attained the status of 'reasonably competent, unlikely to blow
things up' with nameko, I'm thinking about how to reimplement some of my
company's systems in a nice microservice-y way.

We have a bunch of actions, file transfers, zip/unzip operations, data
imports, data processing, etc. These actions are chained together in
specific ways for each of our customers.

Now, if these operations were implemented as nameko services, I'd need
some way of storing the 'plan' for each of our customers and executing that
plan as needed.

What I'm thinking of is this:

A *Control* service which owns a database containing the client-specific
plan. The service is told to execute the plan for client 'A'. It consults
the plan and triggers the first service action in the plan. The service
(and all others) will end with a call back to the Control service to
indicate success/failure/etc. The Control service then kicks off the next
service in the plan. Lather, rinse, repeat until the plan is complete.

This appears to be a decent enough solution. It would give the ability to
resume a plan from a failed step, plus there could be methods in the
Control service to do things like finding out how far a plan has progressed.

So, a) does this sound sane?, and b) has anyone done something similar
with success?

Cheers,
Chris

Thanks Matt -- that all makes sense, particularly using events rather than
RPC!