Newbie python development. We are putting together a microservice in
python and have been using nameko as the basis. So far so good! We are
seeing some performance issues and before we start to look into these in
detail I wanted to get some validation around how we have approached
developing this microservice. The service is calculating deferred revenue
over a period. ie company issues policy of 100,000 for a year, we need to
calculate which periods that revenue should be distributed across. The
calculations are a mix of linear and non-linear calculation patterns.
We need to support 3 different methods of interaction:
1. HTTP and rpc service that will be called periodically for individual
requests - either to retrieve previously calculated results (stored in
MongoDB) or calculate new results and store them if required.
2. Batch calculation and response of 100,000's of requests in a short
period of time from typically Mainframe COBOL applications.
3. HTTP requests to perform aggregation functions on MongoDB and return the
We have configured the nameko HTTP and rpc methods and for the 1st use
case, performance is acceptable (around 12 records a second generally per
single instance running). (we tested this using locust for HTTP - this
does appear to provide better performance)
The 2nd use case however is causing some anxiety due to performance. We
have used rabbitMQ to queue the batch requests with a inbound queue and an
outbound for the responses.
We have python code that loads the requests into MQ using struct and
stringIO - achieves around 1200 records a second from a flat file.
We have a python 2.7 queue worker which reads the queue and then calls the
nameko rpc service and puts the result back on another queue. Again each
instance of nameko seems to achieve 2.5 records (based on the
acknowledgement queue in rabbitMQ) a second when using the RPC route (which
I believe is the preferred route here.) It is implemented as a straight
service - we have not looked at any pub-sub type patterns in nameko. The
volume of open requests doesn't seem to affect the performance in any way -
tested up to 500K.
As an alternative on the same infrastructure we tested bypassing nameko
completely and just calling the python code directly from the worker and we
achieve throughput of around 11 records per second - we think this more
acceptable - with multiple instances of the Queue Worker we can reach at
least 50 records per second in a fairly linear manner (ie 5 instances
What I want to understand is whether rpc as we've used it is the right
approach for the batch wor`kloads ?
Our current thoughts on next steps is to:
1. Look at implementing multiprocessing in the python service code. (CPU
usage on the nameko service is around 10%)
2. Consider alternative methods in nameko to achieve this.
3. Abandon nameko for the batch processing and call direct to the service
code (certainly not our preference - sort of breaks the microservice
concept in our eyes).
4. Understand whether the pre-fetch setting of 10 on the rpc
implementation is causing the issue or perhaps some sort of configuration
issue in rabbitMQ.
Are we missing anything else here (I feel we might be) ?
Thanks in advance