I’ve been thinking about how we’d like to manage our nameko services.
We’re using nameko more to orchestrate a bunch of actions across many machines, rather than for sheer throughput. So, our workers tend to be long-lived.
We’re almost certain to want to answer questions like ‘Which server is running the export for Customer X and how long has it been running for?’, and ‘The queue for service Y is rather long. What are its workers currently doing, and how far along are they?’
If a service has all of its workers active, any calls to a ‘what are you up to?’ service method will just sit in the queue along with everything else. Some of the questions can be answered with decent logging, but the ability to interrogate a service’s container would be interesting.
Is this sort of ‘out-of-band’ communication feasible? Perhaps some sort of ‘sidecar’ service that can query the main service’s container?
Perhaps this might also open up the possibility of forcing termination of workers. For example, we’ve got a ‘control’ service which follows sets of job-plans and orchestrates many service calls. We have the ability for a user to flag a running job as ‘TerminateEarly’, but it’d be great to forcibly terminate the workers that are doing things on behalf of that job.
Any thoughts appreciated. I’m going to explore the ‘sidecar service as a dependency’ approach, since it feels like it might be cleanest - although ‘nameko service inside a nameko dependency, inside a nameko service’ is a bit Inception-y.