Call_async result

From @geoffjukes on Wed Jul 26 2017 05:53:16 GMT+0000 (UTC)

Hi,

I’m looking at bridging/exposing my nameko services to a PHP application. The 2 options I have come up with are a rest service (either a dedicated flask service, or directly using the http extension) or a Celery based worker (using php-celery). – Celery requires Kombu 4, which Nameko is not compatible with. So a REST service seems like my only option.

I’m open to other solutions, so please feel free to recommend an alternative approach. I kinda feel like adding additional layers to the stack detracts from the elegance of the Nameko solution.

I am using Nameko to handle processing of multiple audio files. I have a master task that uses concurrent.futures.map to create a worker pool, each firing an RPCProxy task. As they come back, the master task updates some data, until they all return. The audio processing is an all-or-nothing situation, so a single failed subtask means the whole job failed. The master task marshals all the subtasks and reacts accordingly. The processing can take a few seconds, or many minutes (20+) depending on the length of the source audio, the number of files, and the type of processing.

Originally I had this written in Celery, but the ‘noise’ from all the subtasks was insane, and made reporting/monitoring a nightmare. Nameko really fit the purpose, as I only really care about the master task, not the individual file tasks - failures etc are logged to Graylog2 for diagnostic and troubleshooting.

So is there an easy way to submit that master task using call_async, and then check its status/result at a later date using a reference ID or something?

With Celery I can call a task, get the task ID, and use that at any time to check the task status (and in RPC mode, get the result exactly once).

I guess an alternative idea might be to use call_async, and rely on event notifications to get results/status feedback.

Anyway, not really a bug, but I’ve read the documentation and the code, and can’t really figure it out.

Many thanks for any help or advice.

Geoff



Copied from original issue: https://github.com/nameko/nameko/issues/459