Service talking to blocking process via stdin/out

Here's an interesting one...

One of my services will need to talk to a process which it launches. It
sends commands via stdin and reads data back via stdout.

In the current implementation, it simply blocks reading lines from stdout.
However, the commands can sometimes take many minutes to execute.

This service would be called via RPC by other services -- and the RPC
should not return until the process returns complete data from stdout.

What would be the best way of doing this, whilst not interrupting nameko's
heartbeats?

Thanks,
Chris

...thinking about it, the calls to the process which take a long time are known in advance, so...

Would this be fixed by also having a pub/sub arrangement for the long-running calls? The calling service would trigger the process command via a an RPC call_async, and subscribe to an “I’m done!” event from the process service, along with some metadata to describe which call finished?

...or would the blocking nature of the code still cause timeouts even when
being called via rpc's call_async? (sorry for the spam!)

You can wait on the subprocess inside a greenthread. That won't block the
heartbeats.

If you're happy to block a worker until your process exits, this solution
would work fine. A nicer but more complex approach would be to detach the
new process and provide a callback somehow. How about using the @timer
entrypoint and subprocess.poll() to check the status of the process, and
replying with an "I'm done" type message when it exits?

···

On Wednesday, February 21, 2018 at 4:03:18 PM UTC, Chris Platts wrote:

...or would the blocking nature of the code still cause timeouts even when
being called via rpc's call_async? (sorry for the spam!)

Thanks, Matt -- calling the stdout-read loop via eventlet.tpool.execute()
did the job.

Blocking the worker is actually the desired behaviour. This service is
purely an interface to the process, and we only want $max_workers of these
processes running at once.

(as an aside, this is turning into one of those 'proof-of-concept' projects
that's rapidly become 'just-build-the-thing'... but it is all coming
together! Thanks again for your frequent help!)

Chris