# Process not releasing memory to container

**URL:** <https://discourse.nameko.io/t/process-not-releasing-memory-to-container/206>\
**Category:** googlegroup\
**Created:** [August 26, 2017, 2:11pm UTC](https://discourse.nameko.io/t/process-not-releasing-memory-to-container/206 "2017-08-26T14:11:36Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Johan\_Frisell](https://avatars.discourse-cdn.com/v4/letter/j/838e76/32.png) [@Johan\_Frisell](https://discourse.nameko.io/u/Johan_Frisell)\
**Post date:** [August 26, 2017, 2:11pm UTC](https://discourse.nameko.io/t/process-not-releasing-memory-to-container/206/1 "2017-08-26T14:11:36Z")

</div>

\*First of a spec of the setup I'm running:\*

&nbsp;&nbsp;&nbsp;- AWS EC2 instances in the bottom layer.  
&nbsp;&nbsp;&nbsp;- Kubernetes hosting Docker containers  
&nbsp;&nbsp;&nbsp;- Clustered RabbitMQ (two servers) with an ELB in front.  
&nbsp;&nbsp;&nbsp;- Redis  
&nbsp;&nbsp;&nbsp;- Influx  
&nbsp;&nbsp;&nbsp;- Nameko 2.6.0  
&nbsp;&nbsp;&nbsp;  
\*Problem:\*

In short I'm experiencing memory allocation problems. Where the python  
process running Nameko wont release the memory back to the container,  
causing the container to reboot when it hits its memory limit (in my case  
512MB). An easy way to reproduce this in my environment is to log some  
random messages. The more I log the bigger the leak. However this is not in  
any way a proof that the logging is the core problem, but maybe an  
indication?

Below is a sample code I use to reproduce the problem. As you might notice  
I have a circular event dispatch/handling, this is just for testing  
purposes. My "real" code does not have this implementation but still  
experience the same problem. Even with the most simple implementation, a  
service with a simple logging.info("message") will experience a memory leak.

\*service.py\*  
import logging

from nameko.timer import timer  
from nameko.events import event\_handler  
from nameko.events import EventDispatcher

from service.redis\_cli import RedisDependency

class Service:  
&nbsp;&nbsp;&nbsp;&nbsp;name = "service"  
&nbsp;&nbsp;&nbsp;&nbsp;dispatcher = EventDispatcher()  
&nbsp;&nbsp;&nbsp;&nbsp;r\_db = RedisDependency()

&nbsp;&nbsp;&nbsp;&nbsp;@timer(interval=5)  
&nbsp;&nbsp;&nbsp;&nbsp;def log(self):  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info("This is a rather long log message?")  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.dispatcher("publish", {"message": "hello world"})

&nbsp;&nbsp;&nbsp;&nbsp;@event\_handler("uc\_sensor\_pipeline", "new\_reading")  
&nbsp;&nbsp;&nbsp;&nbsp;def do\_log(self, payload):  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info(payload)

&nbsp;&nbsp;&nbsp;&nbsp;@event\_handler("service", "publish")  
&nbsp;&nbsp;&nbsp;&nbsp;def handle\_publish(self, payload):  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.r\_db.lpush('logging', '123', payload)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info(payload['message'])

\*redis\_cli.py\*  
from nameko.extensions import DependencyProvider

from redis import StrictRedis

class RedisWrapper:  
&nbsp;&nbsp;&nbsp;&nbsp;def \_\_init\_\_(self, client):  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client = client

&nbsp;&nbsp;&nbsp;&nbsp;def lpush(self, prefix, key, value):  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;full\_key = self.\_format\_key(prefix, key)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client.lpush(full\_key, value)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client.ltrim(full\_key, 0, 9)

&nbsp;&nbsp;&nbsp;&nbsp;def get(self, prefix, key):  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;full\_key = self.\_format\_key(prefix, key)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return self.client.lrange(full\_key, 0, -1)

&nbsp;&nbsp;&nbsp;&nbsp;def \_decode(self, item):  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return { k.decode(): v.decode() for k, v in item.items() }

&nbsp;&nbsp;&nbsp;&nbsp;def \_format\_key(self, prefix, key):  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return '{0}:{1}'.format(prefix, key)

class RedisDependency(DependencyProvider):  
&nbsp;&nbsp;&nbsp;&nbsp;def setup(self):  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client = StrictRedis.from\_url(  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.container.config.get('REDIS\_URI'))

&nbsp;&nbsp;&nbsp;&nbsp;def get\_dependency(self, worker\_ctx):  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return RedisWrapper(self.client)

\*config.yml\*  
AMQP\_URI:  
amqp://${RMQ\_USER:guest}:${RMQ\_PASSWORD:guest}@${RMQ\_HOST:localhost}  
REDIS\_URI: redis://myredishost:6379

LOGGING:  
&nbsp;&nbsp;&nbsp;&nbsp;version: 1  
&nbsp;&nbsp;&nbsp;&nbsp;handlers:  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;console:  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;class: logging.StreamHandler  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file:  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;class: logging.handlers.RotatingFileHandler  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;filename: service.log  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;maxBytes: 1048576  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;backupCount: 3  
&nbsp;&nbsp;&nbsp;&nbsp;root:  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;level: INFO  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;handlers: [console]

\*Overview of the mem allocation:\*

\<[https://lh3.googleusercontent.com/-UO65BwWFeBE/WaB5lzVen3I/AAAAAAAAHc0/HO\_TvJcuF8oPNuvGZv2L\_nXmtJS8XT5ugCLcBGAs/s1600/Screen%2BShot%2B2017-08-25%2Bat%2B21.23.48.png](https://lh3.googleusercontent.com/-UO65BwWFeBE/WaB5lzVen3I/AAAAAAAAHc0/HO_TvJcuF8oPNuvGZv2L_nXmtJS8XT5ugCLcBGAs/s1600/Screen%2BShot%2B2017-08-25%2Bat%2B21.23.48.png)\>

I know this is not a huge increase but it follows a pattern and is steady  
(will eventually cause the container to reboot due to hitting mem limit).  
In containers with more load it would leak memory faster.

\*What have I tried (without luck) so far:\*

&nbsp;&nbsp;&nbsp;- Logging to file instead of stdout.  
&nbsp;&nbsp;&nbsp;- Manually gc.collect() (as this doesn't release any memory I figure  
&nbsp;&nbsp;&nbsp;that it's not a reference problem but rather that the actual process wont  
&nbsp;&nbsp;&nbsp;release memory to the container once freed by the gc).  
&nbsp;&nbsp;&nbsp;- Removed the dependency injection and calling redis from the service  
&nbsp;&nbsp;&nbsp;itself.  
&nbsp;&nbsp;&nbsp;- Decrease and increase number of workers.  
&nbsp;&nbsp;&nbsp;- Simplify the service to just contain a timer, which logs a message  
&nbsp;&nbsp;&nbsp;every fifth second (still memory increases)

I understand that this might be perceived as to little information to give  
any valuable input. But I sincerely don't know how to proceed. Anyone got  
any ideas or see anything really strange with the example? Anyone  
experienced something similar?

If you came this far, thank you for reading and taking your time! Any help  
would be very appreciated!

Disclaimer: I don't necessarily believe this has to do with Nameko in  
particular but rather some combination of errors/dependencies/bad  
implementation.

---

<div class="post-metadata">

**Author:** ![David\_Szotten](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/david_szotten/32/6_2.png) [@David\_Szotten](https://discourse.nameko.io/u/David_Szotten)\
**Post date:** [August 27, 2017, 9:44am UTC](https://discourse.nameko.io/t/process-not-releasing-memory-to-container/206/2 "2017-08-27T09:44:29Z")

</div>

Hi Johan,

I'm sorry you're having issues. However, there's still quite a lot going on  
in your example, and a few unknowns. Could you try reducing the example  
further? (e.g. removing redis, and even the event dispatching? Do you still  
see the problem? What if you remove nameko entirely and just log stuff in a  
loop?) With a 5 second timer, how long before you see anything happening?

Also, can you reproduce the memory increase without docker? (do you see the  
process memory usage grow over time) If not, what's your docker setup?  
Could you post a complete setup that we can `docker run` (or  
`docker-compose run`)? That may be easier as a github repo

Best,  
David

> **···**
>
> On Saturday, 26 August 2017 15:11:36 UTC+1, Johan Frisell wrote:
> 
> > \*First of a spec of the setup I'm running:\*
> > 
> > &nbsp;&nbsp;&nbsp;- AWS EC2 instances in the bottom layer.  
> > &nbsp;&nbsp;&nbsp;- Kubernetes hosting Docker containers  
> > &nbsp;&nbsp;&nbsp;- Clustered RabbitMQ (two servers) with an ELB in front.  
> > &nbsp;&nbsp;&nbsp;- Redis  
> > &nbsp;&nbsp;&nbsp;- Influx  
> > &nbsp;&nbsp;&nbsp;- Nameko 2.6.0  
> > &nbsp;&nbsp;&nbsp;  
> > \*Problem:\*
> > 
> > In short I'm experiencing memory allocation problems. Where the python  
> > process running Nameko wont release the memory back to the container,  
> > causing the container to reboot when it hits its memory limit (in my case  
> > 512MB). An easy way to reproduce this in my environment is to log some  
> > random messages. The more I log the bigger the leak. However this is not in  
> > any way a proof that the logging is the core problem, but maybe an  
> > indication?
> > 
> > Below is a sample code I use to reproduce the problem. As you might notice  
> > I have a circular event dispatch/handling, this is just for testing  
> > purposes. My "real" code does not have this implementation but still  
> > experience the same problem. Even with the most simple implementation, a  
> > service with a simple logging.info("message") will experience a memory  
> > leak.
> > 
> > \*service.py\*  
> > import logging
> > 
> > from nameko.timer import timer  
> > from nameko.events import event\_handler  
> > from nameko.events import EventDispatcher
> > 
> > from service.redis\_cli import RedisDependency
> > 
> > class Service:  
> > &nbsp;&nbsp;&nbsp;&nbsp;name = "service"  
> > &nbsp;&nbsp;&nbsp;&nbsp;dispatcher = EventDispatcher()  
> > &nbsp;&nbsp;&nbsp;&nbsp;r\_db = RedisDependency()
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;@timer(interval=5)  
> > &nbsp;&nbsp;&nbsp;&nbsp;def log(self):  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info("This is a rather long log message?")  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.dispatcher("publish", {"message": "hello world"})
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;@event\_handler("uc\_sensor\_pipeline", "new\_reading")  
> > &nbsp;&nbsp;&nbsp;&nbsp;def do\_log(self, payload):  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info(payload)
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;@event\_handler("service", "publish")  
> > &nbsp;&nbsp;&nbsp;&nbsp;def handle\_publish(self, payload):  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.r\_db.lpush('logging', '123', payload)  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info(payload['message'])
> > 
> > \*redis\_cli.py\*  
> > from nameko.extensions import DependencyProvider
> > 
> > from redis import StrictRedis
> > 
> > class RedisWrapper:  
> > &nbsp;&nbsp;&nbsp;&nbsp;def \_\_init\_\_(self, client):  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client = client
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;def lpush(self, prefix, key, value):  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;full\_key = self.\_format\_key(prefix, key)  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client.lpush(full\_key, value)  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client.ltrim(full\_key, 0, 9)
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;def get(self, prefix, key):  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;full\_key = self.\_format\_key(prefix, key)  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return self.client.lrange(full\_key, 0, -1)
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;def \_decode(self, item):  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return { k.decode(): v.decode() for k, v in item.items() }
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;def \_format\_key(self, prefix, key):  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return '{0}:{1}'.format(prefix, key)
> > 
> > class RedisDependency(DependencyProvider):  
> > &nbsp;&nbsp;&nbsp;&nbsp;def setup(self):  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client = StrictRedis.from\_url(  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.container.config.get('REDIS\_URI'))
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;def get\_dependency(self, worker\_ctx):  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return RedisWrapper(self.client)
> > 
> > \*config.yml\*  
> > AMQP\_URI:  
> > amqp://${RMQ\_USER:guest}:${RMQ\_PASSWORD:guest}@${RMQ\_HOST:localhost}  
> > REDIS\_URI: redis://myredishost:6379
> > 
> > LOGGING:  
> > &nbsp;&nbsp;&nbsp;&nbsp;version: 1  
> > &nbsp;&nbsp;&nbsp;&nbsp;handlers:  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;console:  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;class: logging.StreamHandler  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file:  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;class: logging.handlers.RotatingFileHandler  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;filename: service.log  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;maxBytes: 1048576  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;backupCount: 3  
> > &nbsp;&nbsp;&nbsp;&nbsp;root:  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;level: INFO  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;handlers: [console]
> > 
> > \*Overview of the mem allocation:\*
> > 
> > \<[https://lh3.googleusercontent.com/-UO65BwWFeBE/WaB5lzVen3I/AAAAAAAAHc0/HO\_TvJcuF8oPNuvGZv2L\_nXmtJS8XT5ugCLcBGAs/s1600/Screen%2BShot%2B2017-08-25%2Bat%2B21.23.48.png&gt](https://lh3.googleusercontent.com/-UO65BwWFeBE/WaB5lzVen3I/AAAAAAAAHc0/HO_TvJcuF8oPNuvGZv2L_nXmtJS8XT5ugCLcBGAs/s1600/Screen%2BShot%2B2017-08-25%2Bat%2B21.23.48.png&gt);
> > 
> > I know this is not a huge increase but it follows a pattern and is steady  
> > (will eventually cause the container to reboot due to hitting mem limit).  
> > In containers with more load it would leak memory faster.
> > 
> > \*What have I tried (without luck) so far:\*
> > 
> > &nbsp;&nbsp;&nbsp;- Logging to file instead of stdout.  
> > &nbsp;&nbsp;&nbsp;- Manually gc.collect() (as this doesn't release any memory I figure  
> > &nbsp;&nbsp;&nbsp;that it's not a reference problem but rather that the actual process wont  
> > &nbsp;&nbsp;&nbsp;release memory to the container once freed by the gc).  
> > &nbsp;&nbsp;&nbsp;- Removed the dependency injection and calling redis from the service  
> > &nbsp;&nbsp;&nbsp;itself.  
> > &nbsp;&nbsp;&nbsp;- Decrease and increase number of workers.  
> > &nbsp;&nbsp;&nbsp;- Simplify the service to just contain a timer, which logs a message  
> > &nbsp;&nbsp;&nbsp;every fifth second (still memory increases)
> > 
> > I understand that this might be perceived as to little information to give  
> > any valuable input. But I sincerely don't know how to proceed. Anyone got  
> > any ideas or see anything really strange with the example? Anyone  
> > experienced something similar?
> > 
> > If you came this far, thank you for reading and taking your time! Any help  
> > would be very appreciated!
> > 
> > Disclaimer: I don't necessarily believe this has to do with Nameko in  
> > particular but rather some combination of errors/dependencies/bad  
> > implementation.

---

<div class="post-metadata">

**Author:** ![Johan\_Frisell](https://avatars.discourse-cdn.com/v4/letter/j/838e76/32.png) [@Johan\_Frisell](https://discourse.nameko.io/u/Johan_Frisell)\
**Post date:** [August 27, 2017, 10:38am UTC](https://discourse.nameko.io/t/process-not-releasing-memory-to-container/206/3 "2017-08-27T10:38:51Z")

</div>

Hi,

Thanks for your quick reply, much appreciated!

I've tried almost all settings concerning stripping down my nameko service:

&nbsp;&nbsp;&nbsp;- Removing the redis dep  
&nbsp;&nbsp;&nbsp;- Removing event dispatcher  
&nbsp;&nbsp;&nbsp;- Removing even handler  
&nbsp;&nbsp;&nbsp;- Combinations of the above

However I'm still experiencing the problem. The only conclusion made so far  
is: the more load the bigger leak, which in my opinion indicates some kind  
of integration problem towards underlying platform, logging not releasing  
memory to the container or something similar.

I've created a small test with just a super simple python script logging  
messages to stdout.

import logging  
from time import sleep

def setup():  
&nbsp;&nbsp;&nbsp;&nbsp;logger = logging.getLogger('spam\_application')  
&nbsp;&nbsp;&nbsp;&nbsp;logger.setLevel(logging.DEBUG)

&nbsp;&nbsp;&nbsp;&nbsp;ch = logging.StreamHandler()  
&nbsp;&nbsp;&nbsp;&nbsp;ch.setLevel(logging.INFO)

&nbsp;&nbsp;&nbsp;&nbsp;fh = logging.FileHandler('spam.log')  
&nbsp;&nbsp;&nbsp;&nbsp;fh.setLevel(logging.DEBUG)

&nbsp;&nbsp;&nbsp;&nbsp;formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s -  
%(message)s')  
&nbsp;&nbsp;&nbsp;&nbsp;ch.setFormatter(formatter)  
&nbsp;&nbsp;&nbsp;&nbsp;fh.setFormatter(formatter)

&nbsp;&nbsp;&nbsp;&nbsp;logger.addHandler(ch)  
&nbsp;&nbsp;&nbsp;&nbsp;# logger.addHandler(fh)

def work():  
&nbsp;&nbsp;&nbsp;&nbsp;logger = logging.getLogger('spam\_application')

&nbsp;&nbsp;&nbsp;&nbsp;while True:  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logger.info("Hello World")  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sleep(1)

if \_\_name\_\_ == "\_\_main\_\_":  
&nbsp;&nbsp;&nbsp;&nbsp;setup()  
&nbsp;&nbsp;&nbsp;&nbsp;work()

As of now, it doesn't leak any memory. But since the mem consumption in  
general is so small for such a script I need to run it a for a bit longer  
before posting a graph.

Do you have any thoughts so far?

> **···**
>
> Den söndag 27 augusti 2017 kl. 11:44:30 UTC+2 skrev David Szotten:
> 
> > Hi Johan,
> > 
> > I'm sorry you're having issues. However, there's still quite a lot going  
> > on in your example, and a few unknowns. Could you try reducing the example  
> > further? (e.g. removing redis, and even the event dispatching? Do you still  
> > see the problem? What if you remove nameko entirely and just log stuff in a  
> > loop?) With a 5 second timer, how long before you see anything happening?
> > 
> > Also, can you reproduce the memory increase without docker? (do you see  
> > the process memory usage grow over time) If not, what's your docker setup?  
> > Could you post a complete setup that we can `docker run` (or  
> > `docker-compose run`)? That may be easier as a github repo
> > 
> > Best,  
> > David
> > 
> > On Saturday, 26 August 2017 15:11:36 UTC+1, Johan Frisell wrote:
> > 
> > > \*First of a spec of the setup I'm running:\*
> > > 
> > > &nbsp;&nbsp;&nbsp;- AWS EC2 instances in the bottom layer.  
> > > &nbsp;&nbsp;&nbsp;- Kubernetes hosting Docker containers  
> > > &nbsp;&nbsp;&nbsp;- Clustered RabbitMQ (two servers) with an ELB in front.  
> > > &nbsp;&nbsp;&nbsp;- Redis  
> > > &nbsp;&nbsp;&nbsp;- Influx  
> > > &nbsp;&nbsp;&nbsp;- Nameko 2.6.0  
> > > &nbsp;&nbsp;&nbsp;  
> > > \*Problem:\*
> > > 
> > > In short I'm experiencing memory allocation problems. Where the python  
> > > process running Nameko wont release the memory back to the container,  
> > > causing the container to reboot when it hits its memory limit (in my case  
> > > 512MB). An easy way to reproduce this in my environment is to log some  
> > > random messages. The more I log the bigger the leak. However this is not in  
> > > any way a proof that the logging is the core problem, but maybe an  
> > > indication?
> > > 
> > > Below is a sample code I use to reproduce the problem. As you might  
> > > notice I have a circular event dispatch/handling, this is just for testing  
> > > purposes. My "real" code does not have this implementation but still  
> > > experience the same problem. Even with the most simple implementation, a  
> > > service with a simple logging.info("message") will experience a memory  
> > > leak.
> > > 
> > > \*service.py\*  
> > > import logging
> > > 
> > > from nameko.timer import timer  
> > > from nameko.events import event\_handler  
> > > from nameko.events import EventDispatcher
> > > 
> > > from service.redis\_cli import RedisDependency
> > > 
> > > class Service:  
> > > &nbsp;&nbsp;&nbsp;&nbsp;name = "service"  
> > > &nbsp;&nbsp;&nbsp;&nbsp;dispatcher = EventDispatcher()  
> > > &nbsp;&nbsp;&nbsp;&nbsp;r\_db = RedisDependency()
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;@timer(interval=5)  
> > > &nbsp;&nbsp;&nbsp;&nbsp;def log(self):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info("This is a rather long log message?")  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.dispatcher("publish", {"message": "hello world"})
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;@event\_handler("uc\_sensor\_pipeline", "new\_reading")  
> > > &nbsp;&nbsp;&nbsp;&nbsp;def do\_log(self, payload):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info(payload)
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;@event\_handler("service", "publish")  
> > > &nbsp;&nbsp;&nbsp;&nbsp;def handle\_publish(self, payload):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.r\_db.lpush('logging', '123', payload)  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info(payload['message'])
> > > 
> > > \*redis\_cli.py\*  
> > > from nameko.extensions import DependencyProvider
> > > 
> > > from redis import StrictRedis
> > > 
> > > class RedisWrapper:  
> > > &nbsp;&nbsp;&nbsp;&nbsp;def \_\_init\_\_(self, client):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client = client
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;def lpush(self, prefix, key, value):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;full\_key = self.\_format\_key(prefix, key)  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client.lpush(full\_key, value)  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client.ltrim(full\_key, 0, 9)
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;def get(self, prefix, key):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;full\_key = self.\_format\_key(prefix, key)  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return self.client.lrange(full\_key, 0, -1)
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;def \_decode(self, item):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return { k.decode(): v.decode() for k, v in item.items() }
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;def \_format\_key(self, prefix, key):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return '{0}:{1}'.format(prefix, key)
> > > 
> > > class RedisDependency(DependencyProvider):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;def setup(self):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client = StrictRedis.from\_url(  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.container.config.get('REDIS\_URI'))
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;def get\_dependency(self, worker\_ctx):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return RedisWrapper(self.client)
> > > 
> > > \*config.yml\*  
> > > AMQP\_URI:  
> > > amqp://${RMQ\_USER:guest}:${RMQ\_PASSWORD:guest}@${RMQ\_HOST:localhost}  
> > > REDIS\_URI: redis://myredishost:6379
> > > 
> > > LOGGING:  
> > > &nbsp;&nbsp;&nbsp;&nbsp;version: 1  
> > > &nbsp;&nbsp;&nbsp;&nbsp;handlers:  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;console:  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;class: logging.StreamHandler  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file:  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;class: logging.handlers.RotatingFileHandler  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;filename: service.log  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;maxBytes: 1048576  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;backupCount: 3  
> > > &nbsp;&nbsp;&nbsp;&nbsp;root:  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;level: INFO  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;handlers: [console]
> > > 
> > > \*Overview of the mem allocation:\*
> > > 
> > > \<[https://lh3.googleusercontent.com/-UO65BwWFeBE/WaB5lzVen3I/AAAAAAAAHc0/HO\_TvJcuF8oPNuvGZv2L\_nXmtJS8XT5ugCLcBGAs/s1600/Screen%2BShot%2B2017-08-25%2Bat%2B21.23.48.png&gt](https://lh3.googleusercontent.com/-UO65BwWFeBE/WaB5lzVen3I/AAAAAAAAHc0/HO_TvJcuF8oPNuvGZv2L_nXmtJS8XT5ugCLcBGAs/s1600/Screen%2BShot%2B2017-08-25%2Bat%2B21.23.48.png&gt);
> > > 
> > > I know this is not a huge increase but it follows a pattern and is steady  
> > > (will eventually cause the container to reboot due to hitting mem limit).  
> > > In containers with more load it would leak memory faster.
> > > 
> > > \*What have I tried (without luck) so far:\*
> > > 
> > > &nbsp;&nbsp;&nbsp;- Logging to file instead of stdout.  
> > > &nbsp;&nbsp;&nbsp;- Manually gc.collect() (as this doesn't release any memory I figure  
> > > &nbsp;&nbsp;&nbsp;that it's not a reference problem but rather that the actual process wont  
> > > &nbsp;&nbsp;&nbsp;release memory to the container once freed by the gc).  
> > > &nbsp;&nbsp;&nbsp;- Removed the dependency injection and calling redis from the service  
> > > &nbsp;&nbsp;&nbsp;itself.  
> > > &nbsp;&nbsp;&nbsp;- Decrease and increase number of workers.  
> > > &nbsp;&nbsp;&nbsp;- Simplify the service to just contain a timer, which logs a message  
> > > &nbsp;&nbsp;&nbsp;every fifth second (still memory increases)
> > > 
> > > I understand that this might be perceived as to little information to  
> > > give any valuable input. But I sincerely don't know how to proceed. Anyone  
> > > got any ideas or see anything really strange with the example? Anyone  
> > > experienced something similar?
> > > 
> > > If you came this far, thank you for reading and taking your time! Any  
> > > help would be very appreciated!
> > > 
> > > Disclaimer: I don't necessarily believe this has to do with Nameko in  
> > > particular but rather some combination of errors/dependencies/bad  
> > > implementation.

---

<div class="post-metadata">

**Author:** ![Johan\_Frisell](https://avatars.discourse-cdn.com/v4/letter/j/838e76/32.png) [@Johan\_Frisell](https://discourse.nameko.io/u/Johan_Frisell)\
**Post date:** [August 27, 2017, 11:06am UTC](https://discourse.nameko.io/t/process-not-releasing-memory-to-container/206/4 "2017-08-27T11:06:10Z")

</div>

I've made another test using minikube (local kubertes ish). The memory  
problem persists, although through cAdvisor I can see that the mem increase  
happens continuously at a steady pace:

\<[https://lh3.googleusercontent.com/-fQDcGhUK4l8/WaKniJUKj6I/AAAAAAAAHdI/iyMRrS7K9Vo8Lp3cjhiSVNd2YxXTx3lQACLcBGAs/s1600/Screen%2BShot%2B2017-08-27%2Bat%2B13.01.35.png&gt](https://lh3.googleusercontent.com/-fQDcGhUK4l8/WaKniJUKj6I/AAAAAAAAHdI/iyMRrS7K9Vo8Lp3cjhiSVNd2YxXTx3lQACLcBGAs/s1600/Screen%2BShot%2B2017-08-27%2Bat%2B13.01.35.png&gt);

> **···**
>
> Den söndag 27 augusti 2017 kl. 11:44:30 UTC+2 skrev David Szotten:
> 
> > Hi Johan,
> > 
> > I'm sorry you're having issues. However, there's still quite a lot going  
> > on in your example, and a few unknowns. Could you try reducing the example  
> > further? (e.g. removing redis, and even the event dispatching? Do you still  
> > see the problem? What if you remove nameko entirely and just log stuff in a  
> > loop?) With a 5 second timer, how long before you see anything happening?
> > 
> > Also, can you reproduce the memory increase without docker? (do you see  
> > the process memory usage grow over time) If not, what's your docker setup?  
> > Could you post a complete setup that we can `docker run` (or  
> > `docker-compose run`)? That may be easier as a github repo
> > 
> > Best,  
> > David
> > 
> > On Saturday, 26 August 2017 15:11:36 UTC+1, Johan Frisell wrote:
> > 
> > > \*First of a spec of the setup I'm running:\*
> > > 
> > > &nbsp;&nbsp;&nbsp;- AWS EC2 instances in the bottom layer.  
> > > &nbsp;&nbsp;&nbsp;- Kubernetes hosting Docker containers  
> > > &nbsp;&nbsp;&nbsp;- Clustered RabbitMQ (two servers) with an ELB in front.  
> > > &nbsp;&nbsp;&nbsp;- Redis  
> > > &nbsp;&nbsp;&nbsp;- Influx  
> > > &nbsp;&nbsp;&nbsp;- Nameko 2.6.0  
> > > &nbsp;&nbsp;&nbsp;  
> > > \*Problem:\*
> > > 
> > > In short I'm experiencing memory allocation problems. Where the python  
> > > process running Nameko wont release the memory back to the container,  
> > > causing the container to reboot when it hits its memory limit (in my case  
> > > 512MB). An easy way to reproduce this in my environment is to log some  
> > > random messages. The more I log the bigger the leak. However this is not in  
> > > any way a proof that the logging is the core problem, but maybe an  
> > > indication?
> > > 
> > > Below is a sample code I use to reproduce the problem. As you might  
> > > notice I have a circular event dispatch/handling, this is just for testing  
> > > purposes. My "real" code does not have this implementation but still  
> > > experience the same problem. Even with the most simple implementation, a  
> > > service with a simple logging.info("message") will experience a memory  
> > > leak.
> > > 
> > > \*service.py\*  
> > > import logging
> > > 
> > > from nameko.timer import timer  
> > > from nameko.events import event\_handler  
> > > from nameko.events import EventDispatcher
> > > 
> > > from service.redis\_cli import RedisDependency
> > > 
> > > class Service:  
> > > &nbsp;&nbsp;&nbsp;&nbsp;name = "service"  
> > > &nbsp;&nbsp;&nbsp;&nbsp;dispatcher = EventDispatcher()  
> > > &nbsp;&nbsp;&nbsp;&nbsp;r\_db = RedisDependency()
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;@timer(interval=5)  
> > > &nbsp;&nbsp;&nbsp;&nbsp;def log(self):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info("This is a rather long log message?")  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.dispatcher("publish", {"message": "hello world"})
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;@event\_handler("uc\_sensor\_pipeline", "new\_reading")  
> > > &nbsp;&nbsp;&nbsp;&nbsp;def do\_log(self, payload):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info(payload)
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;@event\_handler("service", "publish")  
> > > &nbsp;&nbsp;&nbsp;&nbsp;def handle\_publish(self, payload):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.r\_db.lpush('logging', '123', payload)  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info(payload['message'])
> > > 
> > > \*redis\_cli.py\*  
> > > from nameko.extensions import DependencyProvider
> > > 
> > > from redis import StrictRedis
> > > 
> > > class RedisWrapper:  
> > > &nbsp;&nbsp;&nbsp;&nbsp;def \_\_init\_\_(self, client):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client = client
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;def lpush(self, prefix, key, value):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;full\_key = self.\_format\_key(prefix, key)  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client.lpush(full\_key, value)  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client.ltrim(full\_key, 0, 9)
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;def get(self, prefix, key):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;full\_key = self.\_format\_key(prefix, key)  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return self.client.lrange(full\_key, 0, -1)
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;def \_decode(self, item):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return { k.decode(): v.decode() for k, v in item.items() }
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;def \_format\_key(self, prefix, key):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return '{0}:{1}'.format(prefix, key)
> > > 
> > > class RedisDependency(DependencyProvider):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;def setup(self):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client = StrictRedis.from\_url(  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.container.config.get('REDIS\_URI'))
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;def get\_dependency(self, worker\_ctx):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return RedisWrapper(self.client)
> > > 
> > > \*config.yml\*  
> > > AMQP\_URI:  
> > > amqp://${RMQ\_USER:guest}:${RMQ\_PASSWORD:guest}@${RMQ\_HOST:localhost}  
> > > REDIS\_URI: redis://myredishost:6379
> > > 
> > > LOGGING:  
> > > &nbsp;&nbsp;&nbsp;&nbsp;version: 1  
> > > &nbsp;&nbsp;&nbsp;&nbsp;handlers:  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;console:  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;class: logging.StreamHandler  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file:  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;class: logging.handlers.RotatingFileHandler  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;filename: service.log  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;maxBytes: 1048576  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;backupCount: 3  
> > > &nbsp;&nbsp;&nbsp;&nbsp;root:  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;level: INFO  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;handlers: [console]
> > > 
> > > \*Overview of the mem allocation:\*
> > > 
> > > \<[https://lh3.googleusercontent.com/-UO65BwWFeBE/WaB5lzVen3I/AAAAAAAAHc0/HO\_TvJcuF8oPNuvGZv2L\_nXmtJS8XT5ugCLcBGAs/s1600/Screen%2BShot%2B2017-08-25%2Bat%2B21.23.48.png&gt](https://lh3.googleusercontent.com/-UO65BwWFeBE/WaB5lzVen3I/AAAAAAAAHc0/HO_TvJcuF8oPNuvGZv2L_nXmtJS8XT5ugCLcBGAs/s1600/Screen%2BShot%2B2017-08-25%2Bat%2B21.23.48.png&gt);
> > > 
> > > I know this is not a huge increase but it follows a pattern and is steady  
> > > (will eventually cause the container to reboot due to hitting mem limit).  
> > > In containers with more load it would leak memory faster.
> > > 
> > > \*What have I tried (without luck) so far:\*
> > > 
> > > &nbsp;&nbsp;&nbsp;- Logging to file instead of stdout.  
> > > &nbsp;&nbsp;&nbsp;- Manually gc.collect() (as this doesn't release any memory I figure  
> > > &nbsp;&nbsp;&nbsp;that it's not a reference problem but rather that the actual process wont  
> > > &nbsp;&nbsp;&nbsp;release memory to the container once freed by the gc).  
> > > &nbsp;&nbsp;&nbsp;- Removed the dependency injection and calling redis from the service  
> > > &nbsp;&nbsp;&nbsp;itself.  
> > > &nbsp;&nbsp;&nbsp;- Decrease and increase number of workers.  
> > > &nbsp;&nbsp;&nbsp;- Simplify the service to just contain a timer, which logs a message  
> > > &nbsp;&nbsp;&nbsp;every fifth second (still memory increases)
> > > 
> > > I understand that this might be perceived as to little information to  
> > > give any valuable input. But I sincerely don't know how to proceed. Anyone  
> > > got any ideas or see anything really strange with the example? Anyone  
> > > experienced something similar?
> > > 
> > > If you came this far, thank you for reading and taking your time! Any  
> > > help would be very appreciated!
> > > 
> > > Disclaimer: I don't necessarily believe this has to do with Nameko in  
> > > particular but rather some combination of errors/dependencies/bad  
> > > implementation.

---

<div class="post-metadata">

**Author:** ![Johan\_Frisell](https://avatars.discourse-cdn.com/v4/letter/j/838e76/32.png) [@Johan\_Frisell](https://discourse.nameko.io/u/Johan_Frisell)\
**Post date:** [August 27, 2017, 6:13pm UTC](https://discourse.nameko.io/t/process-not-releasing-memory-to-container/206/5 "2017-08-27T18:13:11Z")

</div>

Sorry for spamming this topic, I will take the time to compose a summary  
once I have something valuable to share or maybe even a solution to the  
problem. But I have made some testing without docker at all. Hosting my  
nameko service on a pure EC2 instance with the same configuration as my  
containers (dist, installed deps etc). The process doesn't seem to leak  
memory in this case.

This narrows it down to Nameko in combination with Docker and/or  
Kubernetes. I guess I'll have to host a pure Docker container as well. To  
possibly rule out Kubernetes. Just wanted to give you an update on the  
progress. If you have any ideas this would be most welcome!

Kindest,

> **···**
>
> Den söndag 27 augusti 2017 kl. 11:44:30 UTC+2 skrev David Szotten:
> 
> > Hi Johan,
> > 
> > I'm sorry you're having issues. However, there's still quite a lot going  
> > on in your example, and a few unknowns. Could you try reducing the example  
> > further? (e.g. removing redis, and even the event dispatching? Do you still  
> > see the problem? What if you remove nameko entirely and just log stuff in a  
> > loop?) With a 5 second timer, how long before you see anything happening?
> > 
> > Also, can you reproduce the memory increase without docker? (do you see  
> > the process memory usage grow over time) If not, what's your docker setup?  
> > Could you post a complete setup that we can `docker run` (or  
> > `docker-compose run`)? That may be easier as a github repo
> > 
> > Best,  
> > David
> > 
> > On Saturday, 26 August 2017 15:11:36 UTC+1, Johan Frisell wrote:
> > 
> > > \*First of a spec of the setup I'm running:\*
> > > 
> > > &nbsp;&nbsp;&nbsp;- AWS EC2 instances in the bottom layer.  
> > > &nbsp;&nbsp;&nbsp;- Kubernetes hosting Docker containers  
> > > &nbsp;&nbsp;&nbsp;- Clustered RabbitMQ (two servers) with an ELB in front.  
> > > &nbsp;&nbsp;&nbsp;- Redis  
> > > &nbsp;&nbsp;&nbsp;- Influx  
> > > &nbsp;&nbsp;&nbsp;- Nameko 2.6.0  
> > > &nbsp;&nbsp;&nbsp;  
> > > \*Problem:\*
> > > 
> > > In short I'm experiencing memory allocation problems. Where the python  
> > > process running Nameko wont release the memory back to the container,  
> > > causing the container to reboot when it hits its memory limit (in my case  
> > > 512MB). An easy way to reproduce this in my environment is to log some  
> > > random messages. The more I log the bigger the leak. However this is not in  
> > > any way a proof that the logging is the core problem, but maybe an  
> > > indication?
> > > 
> > > Below is a sample code I use to reproduce the problem. As you might  
> > > notice I have a circular event dispatch/handling, this is just for testing  
> > > purposes. My "real" code does not have this implementation but still  
> > > experience the same problem. Even with the most simple implementation, a  
> > > service with a simple logging.info("message") will experience a memory  
> > > leak.
> > > 
> > > \*service.py\*  
> > > import logging
> > > 
> > > from nameko.timer import timer  
> > > from nameko.events import event\_handler  
> > > from nameko.events import EventDispatcher
> > > 
> > > from service.redis\_cli import RedisDependency
> > > 
> > > class Service:  
> > > &nbsp;&nbsp;&nbsp;&nbsp;name = "service"  
> > > &nbsp;&nbsp;&nbsp;&nbsp;dispatcher = EventDispatcher()  
> > > &nbsp;&nbsp;&nbsp;&nbsp;r\_db = RedisDependency()
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;@timer(interval=5)  
> > > &nbsp;&nbsp;&nbsp;&nbsp;def log(self):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info("This is a rather long log message?")  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.dispatcher("publish", {"message": "hello world"})
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;@event\_handler("uc\_sensor\_pipeline", "new\_reading")  
> > > &nbsp;&nbsp;&nbsp;&nbsp;def do\_log(self, payload):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info(payload)
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;@event\_handler("service", "publish")  
> > > &nbsp;&nbsp;&nbsp;&nbsp;def handle\_publish(self, payload):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.r\_db.lpush('logging', '123', payload)  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info(payload['message'])
> > > 
> > > \*redis\_cli.py\*  
> > > from nameko.extensions import DependencyProvider
> > > 
> > > from redis import StrictRedis
> > > 
> > > class RedisWrapper:  
> > > &nbsp;&nbsp;&nbsp;&nbsp;def \_\_init\_\_(self, client):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client = client
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;def lpush(self, prefix, key, value):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;full\_key = self.\_format\_key(prefix, key)  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client.lpush(full\_key, value)  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client.ltrim(full\_key, 0, 9)
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;def get(self, prefix, key):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;full\_key = self.\_format\_key(prefix, key)  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return self.client.lrange(full\_key, 0, -1)
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;def \_decode(self, item):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return { k.decode(): v.decode() for k, v in item.items() }
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;def \_format\_key(self, prefix, key):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return '{0}:{1}'.format(prefix, key)
> > > 
> > > class RedisDependency(DependencyProvider):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;def setup(self):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client = StrictRedis.from\_url(  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.container.config.get('REDIS\_URI'))
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;def get\_dependency(self, worker\_ctx):  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return RedisWrapper(self.client)
> > > 
> > > \*config.yml\*  
> > > AMQP\_URI:  
> > > amqp://${RMQ\_USER:guest}:${RMQ\_PASSWORD:guest}@${RMQ\_HOST:localhost}  
> > > REDIS\_URI: redis://myredishost:6379
> > > 
> > > LOGGING:  
> > > &nbsp;&nbsp;&nbsp;&nbsp;version: 1  
> > > &nbsp;&nbsp;&nbsp;&nbsp;handlers:  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;console:  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;class: logging.StreamHandler  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file:  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;class: logging.handlers.RotatingFileHandler  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;filename: service.log  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;maxBytes: 1048576  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;backupCount: 3  
> > > &nbsp;&nbsp;&nbsp;&nbsp;root:  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;level: INFO  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;handlers: [console]
> > > 
> > > \*Overview of the mem allocation:\*
> > > 
> > > \<[https://lh3.googleusercontent.com/-UO65BwWFeBE/WaB5lzVen3I/AAAAAAAAHc0/HO\_TvJcuF8oPNuvGZv2L\_nXmtJS8XT5ugCLcBGAs/s1600/Screen%2BShot%2B2017-08-25%2Bat%2B21.23.48.png&gt](https://lh3.googleusercontent.com/-UO65BwWFeBE/WaB5lzVen3I/AAAAAAAAHc0/HO_TvJcuF8oPNuvGZv2L_nXmtJS8XT5ugCLcBGAs/s1600/Screen%2BShot%2B2017-08-25%2Bat%2B21.23.48.png&gt);
> > > 
> > > I know this is not a huge increase but it follows a pattern and is steady  
> > > (will eventually cause the container to reboot due to hitting mem limit).  
> > > In containers with more load it would leak memory faster.
> > > 
> > > \*What have I tried (without luck) so far:\*
> > > 
> > > &nbsp;&nbsp;&nbsp;- Logging to file instead of stdout.  
> > > &nbsp;&nbsp;&nbsp;- Manually gc.collect() (as this doesn't release any memory I figure  
> > > &nbsp;&nbsp;&nbsp;that it's not a reference problem but rather that the actual process wont  
> > > &nbsp;&nbsp;&nbsp;release memory to the container once freed by the gc).  
> > > &nbsp;&nbsp;&nbsp;- Removed the dependency injection and calling redis from the service  
> > > &nbsp;&nbsp;&nbsp;itself.  
> > > &nbsp;&nbsp;&nbsp;- Decrease and increase number of workers.  
> > > &nbsp;&nbsp;&nbsp;- Simplify the service to just contain a timer, which logs a message  
> > > &nbsp;&nbsp;&nbsp;every fifth second (still memory increases)
> > > 
> > > I understand that this might be perceived as to little information to  
> > > give any valuable input. But I sincerely don't know how to proceed. Anyone  
> > > got any ideas or see anything really strange with the example? Anyone  
> > > experienced something similar?
> > > 
> > > If you came this far, thank you for reading and taking your time! Any  
> > > help would be very appreciated!
> > > 
> > > Disclaimer: I don't necessarily believe this has to do with Nameko in  
> > > particular but rather some combination of errors/dependencies/bad  
> > > implementation.

---

<div class="post-metadata">

**Author:** ![Johan\_Frisell](https://avatars.discourse-cdn.com/v4/letter/j/838e76/32.png) [@Johan\_Frisell](https://discourse.nameko.io/u/Johan_Frisell)\
**Post date:** [August 27, 2017, 12:31pm UTC](https://discourse.nameko.io/t/process-not-releasing-memory-to-container/206/6 "2017-08-27T12:31:17Z")

</div>

I can now confirm that no leak exists with the simple python script. Only  
when i incorporate a nameko service.

> **···**
>
> Den söndag 27 augusti 2017 kl. 12:38:52 UTC+2 skrev Johan Frisell:
> 
> > Hi,
> > 
> > Thanks for your quick reply, much appreciated!
> > 
> > I've tried almost all settings concerning stripping down my nameko service:
> > 
> > &nbsp;&nbsp;&nbsp;- Removing the redis dep  
> > &nbsp;&nbsp;&nbsp;- Removing event dispatcher  
> > &nbsp;&nbsp;&nbsp;- Removing even handler  
> > &nbsp;&nbsp;&nbsp;- Combinations of the above
> > 
> > However I'm still experiencing the problem. The only conclusion made so  
> > far is: the more load the bigger leak, which in my opinion indicates some  
> > kind of integration problem towards underlying platform, logging not  
> > releasing memory to the container or something similar.
> > 
> > I've created a small test with just a super simple python script logging  
> > messages to stdout.
> > 
> > import logging  
> > from time import sleep
> > 
> > def setup():  
> > &nbsp;&nbsp;&nbsp;&nbsp;logger = logging.getLogger('spam\_application')  
> > &nbsp;&nbsp;&nbsp;&nbsp;logger.setLevel(logging.DEBUG)
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;ch = logging.StreamHandler()  
> > &nbsp;&nbsp;&nbsp;&nbsp;ch.setLevel(logging.INFO)
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;fh = logging.FileHandler('spam.log')  
> > &nbsp;&nbsp;&nbsp;&nbsp;fh.setLevel(logging.DEBUG)
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s  
> > - %(message)s')  
> > &nbsp;&nbsp;&nbsp;&nbsp;ch.setFormatter(formatter)  
> > &nbsp;&nbsp;&nbsp;&nbsp;fh.setFormatter(formatter)
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;logger.addHandler(ch)  
> > &nbsp;&nbsp;&nbsp;&nbsp;# logger.addHandler(fh)
> > 
> > def work():  
> > &nbsp;&nbsp;&nbsp;&nbsp;logger = logging.getLogger('spam\_application')
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;while True:  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logger.info("Hello World")  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sleep(1)
> > 
> > if \_\_name\_\_ == "\_\_main\_\_":  
> > &nbsp;&nbsp;&nbsp;&nbsp;setup()  
> > &nbsp;&nbsp;&nbsp;&nbsp;work()
> > 
> > As of now, it doesn't leak any memory. But since the mem consumption in  
> > general is so small for such a script I need to run it a for a bit longer  
> > before posting a graph.
> > 
> > Do you have any thoughts so far?
> > 
> > Den söndag 27 augusti 2017 kl. 11:44:30 UTC+2 skrev David Szotten:
> > 
> > > Hi Johan,
> > > 
> > > I'm sorry you're having issues. However, there's still quite a lot going  
> > > on in your example, and a few unknowns. Could you try reducing the example  
> > > further? (e.g. removing redis, and even the event dispatching? Do you still  
> > > see the problem? What if you remove nameko entirely and just log stuff in a  
> > > loop?) With a 5 second timer, how long before you see anything happening?
> > > 
> > > Also, can you reproduce the memory increase without docker? (do you see  
> > > the process memory usage grow over time) If not, what's your docker setup?  
> > > Could you post a complete setup that we can `docker run` (or  
> > > `docker-compose run`)? That may be easier as a github repo
> > > 
> > > Best,  
> > > David
> > > 
> > > On Saturday, 26 August 2017 15:11:36 UTC+1, Johan Frisell wrote:
> > > 
> > > > \*First of a spec of the setup I'm running:\*
> > > > 
> > > > &nbsp;&nbsp;&nbsp;- AWS EC2 instances in the bottom layer.  
> > > > &nbsp;&nbsp;&nbsp;- Kubernetes hosting Docker containers  
> > > > &nbsp;&nbsp;&nbsp;- Clustered RabbitMQ (two servers) with an ELB in front.  
> > > > &nbsp;&nbsp;&nbsp;- Redis  
> > > > &nbsp;&nbsp;&nbsp;- Influx  
> > > > &nbsp;&nbsp;&nbsp;- Nameko 2.6.0  
> > > > &nbsp;&nbsp;&nbsp;  
> > > > \*Problem:\*
> > > > 
> > > > In short I'm experiencing memory allocation problems. Where the python  
> > > > process running Nameko wont release the memory back to the container,  
> > > > causing the container to reboot when it hits its memory limit (in my case  
> > > > 512MB). An easy way to reproduce this in my environment is to log some  
> > > > random messages. The more I log the bigger the leak. However this is not in  
> > > > any way a proof that the logging is the core problem, but maybe an  
> > > > indication?
> > > > 
> > > > Below is a sample code I use to reproduce the problem. As you might  
> > > > notice I have a circular event dispatch/handling, this is just for testing  
> > > > purposes. My "real" code does not have this implementation but still  
> > > > experience the same problem. Even with the most simple implementation, a  
> > > > service with a simple logging.info("message") will experience a memory  
> > > > leak.
> > > > 
> > > > \*service.py\*  
> > > > import logging
> > > > 
> > > > from nameko.timer import timer  
> > > > from nameko.events import event\_handler  
> > > > from nameko.events import EventDispatcher
> > > > 
> > > > from service.redis\_cli import RedisDependency
> > > > 
> > > > class Service:  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;name = "service"  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;dispatcher = EventDispatcher()  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;r\_db = RedisDependency()
> > > > 
> > > > &nbsp;&nbsp;&nbsp;&nbsp;@timer(interval=5)  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;def log(self):  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info("This is a rather long log message?")  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.dispatcher("publish", {"message": "hello world"})
> > > > 
> > > > &nbsp;&nbsp;&nbsp;&nbsp;@event\_handler("uc\_sensor\_pipeline", "new\_reading")  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;def do\_log(self, payload):  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info(payload)
> > > > 
> > > > &nbsp;&nbsp;&nbsp;&nbsp;@event\_handler("service", "publish")  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;def handle\_publish(self, payload):  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.r\_db.lpush('logging', '123', payload)  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info(payload['message'])
> > > > 
> > > > \*redis\_cli.py\*  
> > > > from nameko.extensions import DependencyProvider
> > > > 
> > > > from redis import StrictRedis
> > > > 
> > > > class RedisWrapper:  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;def \_\_init\_\_(self, client):  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client = client
> > > > 
> > > > &nbsp;&nbsp;&nbsp;&nbsp;def lpush(self, prefix, key, value):  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;full\_key = self.\_format\_key(prefix, key)  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client.lpush(full\_key, value)  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client.ltrim(full\_key, 0, 9)
> > > > 
> > > > &nbsp;&nbsp;&nbsp;&nbsp;def get(self, prefix, key):  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;full\_key = self.\_format\_key(prefix, key)  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return self.client.lrange(full\_key, 0, -1)
> > > > 
> > > > &nbsp;&nbsp;&nbsp;&nbsp;def \_decode(self, item):  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return { k.decode(): v.decode() for k, v in item.items() }
> > > > 
> > > > &nbsp;&nbsp;&nbsp;&nbsp;def \_format\_key(self, prefix, key):  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return '{0}:{1}'.format(prefix, key)
> > > > 
> > > > class RedisDependency(DependencyProvider):  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;def setup(self):  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client = StrictRedis.from\_url(  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.container.config.get('REDIS\_URI'))
> > > > 
> > > > &nbsp;&nbsp;&nbsp;&nbsp;def get\_dependency(self, worker\_ctx):  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return RedisWrapper(self.client)
> > > > 
> > > > \*config.yml\*  
> > > > AMQP\_URI:  
> > > > amqp://${RMQ\_USER:guest}:${RMQ\_PASSWORD:guest}@${RMQ\_HOST:localhost}  
> > > > REDIS\_URI: redis://myredishost:6379
> > > > 
> > > > LOGGING:  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;version: 1  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;handlers:  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;console:  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;class: logging.StreamHandler  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file:  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;class: logging.handlers.RotatingFileHandler  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;filename: service.log  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;maxBytes: 1048576  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;backupCount: 3  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;root:  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;level: INFO  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;handlers: [console]
> > > > 
> > > > \*Overview of the mem allocation:\*
> > > > 
> > > > \<[https://lh3.googleusercontent.com/-UO65BwWFeBE/WaB5lzVen3I/AAAAAAAAHc0/HO\_TvJcuF8oPNuvGZv2L\_nXmtJS8XT5ugCLcBGAs/s1600/Screen%2BShot%2B2017-08-25%2Bat%2B21.23.48.png&gt](https://lh3.googleusercontent.com/-UO65BwWFeBE/WaB5lzVen3I/AAAAAAAAHc0/HO_TvJcuF8oPNuvGZv2L_nXmtJS8XT5ugCLcBGAs/s1600/Screen%2BShot%2B2017-08-25%2Bat%2B21.23.48.png&gt);
> > > > 
> > > > I know this is not a huge increase but it follows a pattern and is  
> > > > steady (will eventually cause the container to reboot due to hitting mem  
> > > > limit). In containers with more load it would leak memory faster.
> > > > 
> > > > \*What have I tried (without luck) so far:\*
> > > > 
> > > > &nbsp;&nbsp;&nbsp;- Logging to file instead of stdout.  
> > > > &nbsp;&nbsp;&nbsp;- Manually gc.collect() (as this doesn't release any memory I figure  
> > > > &nbsp;&nbsp;&nbsp;that it's not a reference problem but rather that the actual process wont  
> > > > &nbsp;&nbsp;&nbsp;release memory to the container once freed by the gc).  
> > > > &nbsp;&nbsp;&nbsp;- Removed the dependency injection and calling redis from the  
> > > > &nbsp;&nbsp;&nbsp;service itself.  
> > > > &nbsp;&nbsp;&nbsp;- Decrease and increase number of workers.  
> > > > &nbsp;&nbsp;&nbsp;- Simplify the service to just contain a timer, which logs a message  
> > > > &nbsp;&nbsp;&nbsp;every fifth second (still memory increases)
> > > > 
> > > > I understand that this might be perceived as to little information to  
> > > > give any valuable input. But I sincerely don't know how to proceed. Anyone  
> > > > got any ideas or see anything really strange with the example? Anyone  
> > > > experienced something similar?
> > > > 
> > > > If you came this far, thank you for reading and taking your time! Any  
> > > > help would be very appreciated!
> > > > 
> > > > Disclaimer: I don't necessarily believe this has to do with Nameko in  
> > > > particular but rather some combination of errors/dependencies/bad  
> > > > implementation.

---

<div class="post-metadata">

**Author:** ![mattbennett](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/mattbennett/32/13_2.png) [@mattbennett](https://discourse.nameko.io/u/mattbennett)\
**Post date:** [August 28, 2017, 6:44am UTC](https://discourse.nameko.io/t/process-not-releasing-memory-to-container/206/7 "2017-08-28T06:44:27Z")

</div>

Hi Johan,

Thanks for keeping us up to date with your investigations. Two useful looks  
I've found really helpful for finding memory leaks in the past are Pympler  
\<[Identifying memory leaks — Pympler 0.5 documentation](https://pythonhosted.org/Pympler/muppy.html#muppy&gt); and Objgraph  
\<[https://mg.pov.lt/objgraph/&gt;\](https://mg.pov.lt/objgraph/&gt;%5C).

Use the Nameko backdoor (which deserves its own page in the docs, but  
doesn't have one) to get a REPL inside the running process:

$ nameko run service --backdoor 3000  
...

$ nameko backdoor 3000

> > > from pympler import muppy  
> > > all\_objects = muppy.get\_objects() # will take a while  
> > > from pympler import summary  
> > > sum = summary.summarize(all\_objects)  
> > > summary.print\_(sum)

...

Matt.

> **···**
>
> On Sunday, August 27, 2017 at 7:13:12 PM UTC+1, Johan Frisell wrote:
> 
> > Sorry for spamming this topic, I will take the time to compose a summary  
> > once I have something valuable to share or maybe even a solution to the  
> > problem. But I have made some testing without docker at all. Hosting my  
> > nameko service on a pure EC2 instance with the same configuration as my  
> > containers (dist, installed deps etc). The process doesn't seem to leak  
> > memory in this case.
> > 
> > This narrows it down to Nameko in combination with Docker and/or  
> > Kubernetes. I guess I'll have to host a pure Docker container as well. To  
> > possibly rule out Kubernetes. Just wanted to give you an update on the  
> > progress. If you have any ideas this would be most welcome!
> > 
> > Kindest,
> > 
> > Den söndag 27 augusti 2017 kl. 11:44:30 UTC+2 skrev David Szotten:
> > 
> > > Hi Johan,
> > > 
> > > I'm sorry you're having issues. However, there's still quite a lot going  
> > > on in your example, and a few unknowns. Could you try reducing the example  
> > > further? (e.g. removing redis, and even the event dispatching? Do you still  
> > > see the problem? What if you remove nameko entirely and just log stuff in a  
> > > loop?) With a 5 second timer, how long before you see anything happening?
> > > 
> > > Also, can you reproduce the memory increase without docker? (do you see  
> > > the process memory usage grow over time) If not, what's your docker setup?  
> > > Could you post a complete setup that we can `docker run` (or  
> > > `docker-compose run`)? That may be easier as a github repo
> > > 
> > > Best,  
> > > David
> > > 
> > > On Saturday, 26 August 2017 15:11:36 UTC+1, Johan Frisell wrote:
> > > 
> > > > \*First of a spec of the setup I'm running:\*
> > > > 
> > > > &nbsp;&nbsp;&nbsp;- AWS EC2 instances in the bottom layer.  
> > > > &nbsp;&nbsp;&nbsp;- Kubernetes hosting Docker containers  
> > > > &nbsp;&nbsp;&nbsp;- Clustered RabbitMQ (two servers) with an ELB in front.  
> > > > &nbsp;&nbsp;&nbsp;- Redis  
> > > > &nbsp;&nbsp;&nbsp;- Influx  
> > > > &nbsp;&nbsp;&nbsp;- Nameko 2.6.0  
> > > > &nbsp;&nbsp;&nbsp;  
> > > > \*Problem:\*
> > > > 
> > > > In short I'm experiencing memory allocation problems. Where the python  
> > > > process running Nameko wont release the memory back to the container,  
> > > > causing the container to reboot when it hits its memory limit (in my case  
> > > > 512MB). An easy way to reproduce this in my environment is to log some  
> > > > random messages. The more I log the bigger the leak. However this is not in  
> > > > any way a proof that the logging is the core problem, but maybe an  
> > > > indication?
> > > > 
> > > > Below is a sample code I use to reproduce the problem. As you might  
> > > > notice I have a circular event dispatch/handling, this is just for testing  
> > > > purposes. My "real" code does not have this implementation but still  
> > > > experience the same problem. Even with the most simple implementation, a  
> > > > service with a simple logging.info("message") will experience a memory  
> > > > leak.
> > > > 
> > > > \*service.py\*  
> > > > import logging
> > > > 
> > > > from nameko.timer import timer  
> > > > from nameko.events import event\_handler  
> > > > from nameko.events import EventDispatcher
> > > > 
> > > > from service.redis\_cli import RedisDependency
> > > > 
> > > > class Service:  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;name = "service"  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;dispatcher = EventDispatcher()  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;r\_db = RedisDependency()
> > > > 
> > > > &nbsp;&nbsp;&nbsp;&nbsp;@timer(interval=5)  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;def log(self):  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info("This is a rather long log message?")  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.dispatcher("publish", {"message": "hello world"})
> > > > 
> > > > &nbsp;&nbsp;&nbsp;&nbsp;@event\_handler("uc\_sensor\_pipeline", "new\_reading")  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;def do\_log(self, payload):  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info(payload)
> > > > 
> > > > &nbsp;&nbsp;&nbsp;&nbsp;@event\_handler("service", "publish")  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;def handle\_publish(self, payload):  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.r\_db.lpush('logging', '123', payload)  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info(payload['message'])
> > > > 
> > > > \*redis\_cli.py\*  
> > > > from nameko.extensions import DependencyProvider
> > > > 
> > > > from redis import StrictRedis
> > > > 
> > > > class RedisWrapper:  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;def \_\_init\_\_(self, client):  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client = client
> > > > 
> > > > &nbsp;&nbsp;&nbsp;&nbsp;def lpush(self, prefix, key, value):  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;full\_key = self.\_format\_key(prefix, key)  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client.lpush(full\_key, value)  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client.ltrim(full\_key, 0, 9)
> > > > 
> > > > &nbsp;&nbsp;&nbsp;&nbsp;def get(self, prefix, key):  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;full\_key = self.\_format\_key(prefix, key)  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return self.client.lrange(full\_key, 0, -1)
> > > > 
> > > > &nbsp;&nbsp;&nbsp;&nbsp;def \_decode(self, item):  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return { k.decode(): v.decode() for k, v in item.items() }
> > > > 
> > > > &nbsp;&nbsp;&nbsp;&nbsp;def \_format\_key(self, prefix, key):  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return '{0}:{1}'.format(prefix, key)
> > > > 
> > > > class RedisDependency(DependencyProvider):  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;def setup(self):  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client = StrictRedis.from\_url(  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.container.config.get('REDIS\_URI'))
> > > > 
> > > > &nbsp;&nbsp;&nbsp;&nbsp;def get\_dependency(self, worker\_ctx):  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return RedisWrapper(self.client)
> > > > 
> > > > \*config.yml\*  
> > > > AMQP\_URI:  
> > > > amqp://${RMQ\_USER:guest}:${RMQ\_PASSWORD:guest}@${RMQ\_HOST:localhost}  
> > > > REDIS\_URI: redis://myredishost:6379
> > > > 
> > > > LOGGING:  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;version: 1  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;handlers:  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;console:  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;class: logging.StreamHandler  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file:  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;class: logging.handlers.RotatingFileHandler  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;filename: service.log  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;maxBytes: 1048576  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;backupCount: 3  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;root:  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;level: INFO  
> > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;handlers: [console]
> > > > 
> > > > \*Overview of the mem allocation:\*
> > > > 
> > > > \<[https://lh3.googleusercontent.com/-UO65BwWFeBE/WaB5lzVen3I/AAAAAAAAHc0/HO\_TvJcuF8oPNuvGZv2L\_nXmtJS8XT5ugCLcBGAs/s1600/Screen%2BShot%2B2017-08-25%2Bat%2B21.23.48.png&gt](https://lh3.googleusercontent.com/-UO65BwWFeBE/WaB5lzVen3I/AAAAAAAAHc0/HO_TvJcuF8oPNuvGZv2L_nXmtJS8XT5ugCLcBGAs/s1600/Screen%2BShot%2B2017-08-25%2Bat%2B21.23.48.png&gt);
> > > > 
> > > > I know this is not a huge increase but it follows a pattern and is  
> > > > steady (will eventually cause the container to reboot due to hitting mem  
> > > > limit). In containers with more load it would leak memory faster.
> > > > 
> > > > \*What have I tried (without luck) so far:\*
> > > > 
> > > > &nbsp;&nbsp;&nbsp;- Logging to file instead of stdout.  
> > > > &nbsp;&nbsp;&nbsp;- Manually gc.collect() (as this doesn't release any memory I figure  
> > > > &nbsp;&nbsp;&nbsp;that it's not a reference problem but rather that the actual process wont  
> > > > &nbsp;&nbsp;&nbsp;release memory to the container once freed by the gc).  
> > > > &nbsp;&nbsp;&nbsp;- Removed the dependency injection and calling redis from the  
> > > > &nbsp;&nbsp;&nbsp;service itself.  
> > > > &nbsp;&nbsp;&nbsp;- Decrease and increase number of workers.  
> > > > &nbsp;&nbsp;&nbsp;- Simplify the service to just contain a timer, which logs a message  
> > > > &nbsp;&nbsp;&nbsp;every fifth second (still memory increases)
> > > > 
> > > > I understand that this might be perceived as to little information to  
> > > > give any valuable input. But I sincerely don't know how to proceed. Anyone  
> > > > got any ideas or see anything really strange with the example? Anyone  
> > > > experienced something similar?
> > > > 
> > > > If you came this far, thank you for reading and taking your time! Any  
> > > > help would be very appreciated!
> > > > 
> > > > Disclaimer: I don't necessarily believe this has to do with Nameko in  
> > > > particular but rather some combination of errors/dependencies/bad  
> > > > implementation.

---

<div class="post-metadata">

**Author:** ![Johan\_Frisell](https://avatars.discourse-cdn.com/v4/letter/j/838e76/32.png) [@Johan\_Frisell](https://discourse.nameko.io/u/Johan_Frisell)\
**Post date:** [September 5, 2017, 7:15am UTC](https://discourse.nameko.io/t/process-not-releasing-memory-to-container/206/8 "2017-09-05T07:15:14Z")

</div>

Just wanted to get back in touch. I've not solved the problem yet, a sprint  
just came in the way 🙂

Tried the backdoor, however I can't get it to work on in my docker  
container. I get

netcat: invalid option -- '-'

whenever I try to connect to the backdoor. Probably some underlying  
dependency missing in my container (ubuntu:16:04 based with rlwrap and  
netcat installed), I'll get back as soon as I have anything to report.

Kindest,  
Johan

> **···**
>
> Den måndag 28 augusti 2017 kl. 08:44:28 UTC+2 skrev Matt Yule-Bennett:
> 
> > Hi Johan,
> > 
> > Thanks for keeping us up to date with your investigations. Two useful  
> > looks I've found really helpful for finding memory leaks in the past are  
> > Pympler \<[Identifying memory leaks — Pympler 0.5 documentation](https://pythonhosted.org/Pympler/muppy.html#muppy&gt); and Objgraph  
> > \<[https://mg.pov.lt/objgraph/&gt;\](https://mg.pov.lt/objgraph/&gt;%5C).
> > 
> > Use the Nameko backdoor (which deserves its own page in the docs, but  
> > doesn't have one) to get a REPL inside the running process:
> > 
> > $ nameko run service --backdoor 3000  
> > ...
> > 
> > $ nameko backdoor 3000  
> > \>\>\> from pympler import muppy  
> > \>\>\> all\_objects = muppy.get\_objects() # will take a while  
> > \>\>\> from pympler import summary  
> > \>\>\> sum = summary.summarize(all\_objects)  
> > \>\>\> summary.print\_(sum)  
> > ...
> > 
> > Matt.
> > 
> > On Sunday, August 27, 2017 at 7:13:12 PM UTC+1, Johan Frisell wrote:
> > 
> > > Sorry for spamming this topic, I will take the time to compose a summary  
> > > once I have something valuable to share or maybe even a solution to the  
> > > problem. But I have made some testing without docker at all. Hosting my  
> > > nameko service on a pure EC2 instance with the same configuration as my  
> > > containers (dist, installed deps etc). The process doesn't seem to leak  
> > > memory in this case.
> > > 
> > > This narrows it down to Nameko in combination with Docker and/or  
> > > Kubernetes. I guess I'll have to host a pure Docker container as well. To  
> > > possibly rule out Kubernetes. Just wanted to give you an update on the  
> > > progress. If you have any ideas this would be most welcome!
> > > 
> > > Kindest,
> > > 
> > > Den söndag 27 augusti 2017 kl. 11:44:30 UTC+2 skrev David Szotten:
> > > 
> > > > Hi Johan,
> > > > 
> > > > I'm sorry you're having issues. However, there's still quite a lot going  
> > > > on in your example, and a few unknowns. Could you try reducing the example  
> > > > further? (e.g. removing redis, and even the event dispatching? Do you still  
> > > > see the problem? What if you remove nameko entirely and just log stuff in a  
> > > > loop?) With a 5 second timer, how long before you see anything happening?
> > > > 
> > > > Also, can you reproduce the memory increase without docker? (do you see  
> > > > the process memory usage grow over time) If not, what's your docker setup?  
> > > > Could you post a complete setup that we can `docker run` (or  
> > > > `docker-compose run`)? That may be easier as a github repo
> > > > 
> > > > Best,  
> > > > David
> > > > 
> > > > On Saturday, 26 August 2017 15:11:36 UTC+1, Johan Frisell wrote:
> > > > 
> > > > > \*First of a spec of the setup I'm running:\*
> > > > > 
> > > > > &nbsp;&nbsp;&nbsp;- AWS EC2 instances in the bottom layer.  
> > > > > &nbsp;&nbsp;&nbsp;- Kubernetes hosting Docker containers  
> > > > > &nbsp;&nbsp;&nbsp;- Clustered RabbitMQ (two servers) with an ELB in front.  
> > > > > &nbsp;&nbsp;&nbsp;- Redis  
> > > > > &nbsp;&nbsp;&nbsp;- Influx  
> > > > > &nbsp;&nbsp;&nbsp;- Nameko 2.6.0  
> > > > > &nbsp;&nbsp;&nbsp;  
> > > > > \*Problem:\*
> > > > > 
> > > > > In short I'm experiencing memory allocation problems. Where the python  
> > > > > process running Nameko wont release the memory back to the container,  
> > > > > causing the container to reboot when it hits its memory limit (in my case  
> > > > > 512MB). An easy way to reproduce this in my environment is to log some  
> > > > > random messages. The more I log the bigger the leak. However this is not in  
> > > > > any way a proof that the logging is the core problem, but maybe an  
> > > > > indication?
> > > > > 
> > > > > Below is a sample code I use to reproduce the problem. As you might  
> > > > > notice I have a circular event dispatch/handling, this is just for testing  
> > > > > purposes. My "real" code does not have this implementation but still  
> > > > > experience the same problem. Even with the most simple implementation, a  
> > > > > service with a simple logging.info("message") will experience a memory  
> > > > > leak.
> > > > > 
> > > > > \*service.py\*  
> > > > > import logging
> > > > > 
> > > > > from nameko.timer import timer  
> > > > > from nameko.events import event\_handler  
> > > > > from nameko.events import EventDispatcher
> > > > > 
> > > > > from service.redis\_cli import RedisDependency
> > > > > 
> > > > > class Service:  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;name = "service"  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;dispatcher = EventDispatcher()  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;r\_db = RedisDependency()
> > > > > 
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;@timer(interval=5)  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;def log(self):  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info("This is a rather long log message?")  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.dispatcher("publish", {"message": "hello world"})
> > > > > 
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;@event\_handler("uc\_sensor\_pipeline", "new\_reading")  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;def do\_log(self, payload):  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info(payload)
> > > > > 
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;@event\_handler("service", "publish")  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;def handle\_publish(self, payload):  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.r\_db.lpush('logging', '123', payload)  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info(payload['message'])
> > > > > 
> > > > > \*redis\_cli.py\*  
> > > > > from nameko.extensions import DependencyProvider
> > > > > 
> > > > > from redis import StrictRedis
> > > > > 
> > > > > class RedisWrapper:  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;def \_\_init\_\_(self, client):  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client = client
> > > > > 
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;def lpush(self, prefix, key, value):  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;full\_key = self.\_format\_key(prefix, key)  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client.lpush(full\_key, value)  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client.ltrim(full\_key, 0, 9)
> > > > > 
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;def get(self, prefix, key):  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;full\_key = self.\_format\_key(prefix, key)  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return self.client.lrange(full\_key, 0, -1)
> > > > > 
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;def \_decode(self, item):  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return { k.decode(): v.decode() for k, v in item.items() }
> > > > > 
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;def \_format\_key(self, prefix, key):  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return '{0}:{1}'.format(prefix, key)
> > > > > 
> > > > > class RedisDependency(DependencyProvider):  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;def setup(self):  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client = StrictRedis.from\_url(  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.container.config.get('REDIS\_URI'))
> > > > > 
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;def get\_dependency(self, worker\_ctx):  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return RedisWrapper(self.client)
> > > > > 
> > > > > \*config.yml\*  
> > > > > AMQP\_URI:  
> > > > > amqp://${RMQ\_USER:guest}:${RMQ\_PASSWORD:guest}@${RMQ\_HOST:localhost}  
> > > > > REDIS\_URI: redis://myredishost:6379
> > > > > 
> > > > > LOGGING:  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;version: 1  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;handlers:  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;console:  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;class: logging.StreamHandler  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file:  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;class: logging.handlers.RotatingFileHandler  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;filename: service.log  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;maxBytes: 1048576  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;backupCount: 3  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;root:  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;level: INFO  
> > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;handlers: [console]
> > > > > 
> > > > > \*Overview of the mem allocation:\*
> > > > > 
> > > > > \<[https://lh3.googleusercontent.com/-UO65BwWFeBE/WaB5lzVen3I/AAAAAAAAHc0/HO\_TvJcuF8oPNuvGZv2L\_nXmtJS8XT5ugCLcBGAs/s1600/Screen%2BShot%2B2017-08-25%2Bat%2B21.23.48.png&gt](https://lh3.googleusercontent.com/-UO65BwWFeBE/WaB5lzVen3I/AAAAAAAAHc0/HO_TvJcuF8oPNuvGZv2L_nXmtJS8XT5ugCLcBGAs/s1600/Screen%2BShot%2B2017-08-25%2Bat%2B21.23.48.png&gt);
> > > > > 
> > > > > I know this is not a huge increase but it follows a pattern and is  
> > > > > steady (will eventually cause the container to reboot due to hitting mem  
> > > > > limit). In containers with more load it would leak memory faster.
> > > > > 
> > > > > \*What have I tried (without luck) so far:\*
> > > > > 
> > > > > &nbsp;&nbsp;&nbsp;- Logging to file instead of stdout.  
> > > > > &nbsp;&nbsp;&nbsp;- Manually gc.collect() (as this doesn't release any memory I  
> > > > > &nbsp;&nbsp;&nbsp;figure that it's not a reference problem but rather that the actual process  
> > > > > &nbsp;&nbsp;&nbsp;wont release memory to the container once freed by the gc).  
> > > > > &nbsp;&nbsp;&nbsp;- Removed the dependency injection and calling redis from the  
> > > > > &nbsp;&nbsp;&nbsp;service itself.  
> > > > > &nbsp;&nbsp;&nbsp;- Decrease and increase number of workers.  
> > > > > &nbsp;&nbsp;&nbsp;- Simplify the service to just contain a timer, which logs a  
> > > > > &nbsp;&nbsp;&nbsp;message every fifth second (still memory increases)
> > > > > 
> > > > > I understand that this might be perceived as to little information to  
> > > > > give any valuable input. But I sincerely don't know how to proceed. Anyone  
> > > > > got any ideas or see anything really strange with the example? Anyone  
> > > > > experienced something similar?
> > > > > 
> > > > > If you came this far, thank you for reading and taking your time! Any  
> > > > > help would be very appreciated!
> > > > > 
> > > > > Disclaimer: I don't necessarily believe this has to do with Nameko in  
> > > > > particular but rather some combination of errors/dependencies/bad  
> > > > > implementation.

---

<div class="post-metadata">

**Author:** ![mattbennett](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/mattbennett/32/13_2.png) [@mattbennett](https://discourse.nameko.io/u/mattbennett)\
**Post date:** [September 5, 2017, 7:26am UTC](https://discourse.nameko.io/t/process-not-releasing-memory-to-container/206/9 "2017-09-05T07:26:42Z")

</div>

Oh interesting. The backdoor command is just a convenience wrapper around  
rlwrap, netcat and/or telnet. I tend to just use them directly, e.g.  
"rlwrap telnet localhost 3000"

If you have time, please raise an issue on the Github page to report the  
problems with the backdoor command.

> **···**
>
> On Tuesday, September 5, 2017 at 8:15:14 AM UTC+1, Johan Frisell wrote:
> 
> > Just wanted to get back in touch. I've not solved the problem yet, a  
> > sprint just came in the way 🙂
> > 
> > Tried the backdoor, however I can't get it to work on in my docker  
> > container. I get
> > 
> > netcat: invalid option -- '-'
> > 
> > whenever I try to connect to the backdoor. Probably some underlying  
> > dependency missing in my container (ubuntu:16:04 based with rlwrap and  
> > netcat installed), I'll get back as soon as I have anything to report.
> > 
> > Kindest,  
> > Johan
> > 
> > Den måndag 28 augusti 2017 kl. 08:44:28 UTC+2 skrev Matt Yule-Bennett:
> > 
> > > Hi Johan,
> > > 
> > > Thanks for keeping us up to date with your investigations. Two useful  
> > > looks I've found really helpful for finding memory leaks in the past are  
> > > Pympler \<[Identifying memory leaks — Pympler 0.5 documentation](https://pythonhosted.org/Pympler/muppy.html#muppy&gt); and Objgraph  
> > > \<[https://mg.pov.lt/objgraph/&gt;\](https://mg.pov.lt/objgraph/&gt;%5C).
> > > 
> > > Use the Nameko backdoor (which deserves its own page in the docs, but  
> > > doesn't have one) to get a REPL inside the running process:
> > > 
> > > $ nameko run service --backdoor 3000  
> > > ...
> > > 
> > > $ nameko backdoor 3000  
> > > \>\>\> from pympler import muppy  
> > > \>\>\> all\_objects = muppy.get\_objects() # will take a while  
> > > \>\>\> from pympler import summary  
> > > \>\>\> sum = summary.summarize(all\_objects)  
> > > \>\>\> summary.print\_(sum)  
> > > ...
> > > 
> > > Matt.
> > > 
> > > On Sunday, August 27, 2017 at 7:13:12 PM UTC+1, Johan Frisell wrote:
> > > 
> > > > Sorry for spamming this topic, I will take the time to compose a summary  
> > > > once I have something valuable to share or maybe even a solution to the  
> > > > problem. But I have made some testing without docker at all. Hosting my  
> > > > nameko service on a pure EC2 instance with the same configuration as my  
> > > > containers (dist, installed deps etc). The process doesn't seem to leak  
> > > > memory in this case.
> > > > 
> > > > This narrows it down to Nameko in combination with Docker and/or  
> > > > Kubernetes. I guess I'll have to host a pure Docker container as well. To  
> > > > possibly rule out Kubernetes. Just wanted to give you an update on the  
> > > > progress. If you have any ideas this would be most welcome!
> > > > 
> > > > Kindest,
> > > > 
> > > > Den söndag 27 augusti 2017 kl. 11:44:30 UTC+2 skrev David Szotten:
> > > > 
> > > > > Hi Johan,
> > > > > 
> > > > > I'm sorry you're having issues. However, there's still quite a lot  
> > > > > going on in your example, and a few unknowns. Could you try reducing the  
> > > > > example further? (e.g. removing redis, and even the event dispatching? Do  
> > > > > you still see the problem? What if you remove nameko entirely and just log  
> > > > > stuff in a loop?) With a 5 second timer, how long before you see anything  
> > > > > happening?
> > > > > 
> > > > > Also, can you reproduce the memory increase without docker? (do you see  
> > > > > the process memory usage grow over time) If not, what's your docker setup?  
> > > > > Could you post a complete setup that we can `docker run` (or  
> > > > > `docker-compose run`)? That may be easier as a github repo
> > > > > 
> > > > > Best,  
> > > > > David
> > > > > 
> > > > > On Saturday, 26 August 2017 15:11:36 UTC+1, Johan Frisell wrote:
> > > > > 
> > > > > > \*First of a spec of the setup I'm running:\*
> > > > > > 
> > > > > > &nbsp;&nbsp;&nbsp;- AWS EC2 instances in the bottom layer.  
> > > > > > &nbsp;&nbsp;&nbsp;- Kubernetes hosting Docker containers  
> > > > > > &nbsp;&nbsp;&nbsp;- Clustered RabbitMQ (two servers) with an ELB in front.  
> > > > > > &nbsp;&nbsp;&nbsp;- Redis  
> > > > > > &nbsp;&nbsp;&nbsp;- Influx  
> > > > > > &nbsp;&nbsp;&nbsp;- Nameko 2.6.0  
> > > > > > &nbsp;&nbsp;&nbsp;  
> > > > > > \*Problem:\*
> > > > > > 
> > > > > > In short I'm experiencing memory allocation problems. Where the python  
> > > > > > process running Nameko wont release the memory back to the container,  
> > > > > > causing the container to reboot when it hits its memory limit (in my case  
> > > > > > 512MB). An easy way to reproduce this in my environment is to log some  
> > > > > > random messages. The more I log the bigger the leak. However this is not in  
> > > > > > any way a proof that the logging is the core problem, but maybe an  
> > > > > > indication?
> > > > > > 
> > > > > > Below is a sample code I use to reproduce the problem. As you might  
> > > > > > notice I have a circular event dispatch/handling, this is just for testing  
> > > > > > purposes. My "real" code does not have this implementation but still  
> > > > > > experience the same problem. Even with the most simple implementation, a  
> > > > > > service with a simple logging.info("message") will experience a  
> > > > > > memory leak.
> > > > > > 
> > > > > > \*service.py\*  
> > > > > > import logging
> > > > > > 
> > > > > > from nameko.timer import timer  
> > > > > > from nameko.events import event\_handler  
> > > > > > from nameko.events import EventDispatcher
> > > > > > 
> > > > > > from service.redis\_cli import RedisDependency
> > > > > > 
> > > > > > class Service:  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;name = "service"  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;dispatcher = EventDispatcher()  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;r\_db = RedisDependency()
> > > > > > 
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;@timer(interval=5)  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;def log(self):  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info("This is a rather long log message?")  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.dispatcher("publish", {"message": "hello world"})
> > > > > > 
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;@event\_handler("uc\_sensor\_pipeline", "new\_reading")  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;def do\_log(self, payload):  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info(payload)
> > > > > > 
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;@event\_handler("service", "publish")  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;def handle\_publish(self, payload):  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.r\_db.lpush('logging', '123', payload)  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info(payload['message'])
> > > > > > 
> > > > > > \*redis\_cli.py\*  
> > > > > > from nameko.extensions import DependencyProvider
> > > > > > 
> > > > > > from redis import StrictRedis
> > > > > > 
> > > > > > class RedisWrapper:  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;def \_\_init\_\_(self, client):  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client = client
> > > > > > 
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;def lpush(self, prefix, key, value):  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;full\_key = self.\_format\_key(prefix, key)  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client.lpush(full\_key, value)  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client.ltrim(full\_key, 0, 9)
> > > > > > 
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;def get(self, prefix, key):  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;full\_key = self.\_format\_key(prefix, key)  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return self.client.lrange(full\_key, 0, -1)
> > > > > > 
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;def \_decode(self, item):  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return { k.decode(): v.decode() for k, v in item.items() }
> > > > > > 
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;def \_format\_key(self, prefix, key):  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return '{0}:{1}'.format(prefix, key)
> > > > > > 
> > > > > > class RedisDependency(DependencyProvider):  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;def setup(self):  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client = StrictRedis.from\_url(  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.container.config.get('REDIS\_URI'))
> > > > > > 
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;def get\_dependency(self, worker\_ctx):  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return RedisWrapper(self.client)
> > > > > > 
> > > > > > \*config.yml\*  
> > > > > > AMQP\_URI:  
> > > > > > amqp://${RMQ\_USER:guest}:${RMQ\_PASSWORD:guest}@${RMQ\_HOST:localhost}  
> > > > > > REDIS\_URI: redis://myredishost:6379
> > > > > > 
> > > > > > LOGGING:  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;version: 1  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;handlers:  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;console:  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;class: logging.StreamHandler  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file:  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;class: logging.handlers.RotatingFileHandler  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;filename: service.log  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;maxBytes: 1048576  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;backupCount: 3  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;root:  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;level: INFO  
> > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;handlers: [console]
> > > > > > 
> > > > > > \*Overview of the mem allocation:\*
> > > > > > 
> > > > > > \<[https://lh3.googleusercontent.com/-UO65BwWFeBE/WaB5lzVen3I/AAAAAAAAHc0/HO\_TvJcuF8oPNuvGZv2L\_nXmtJS8XT5ugCLcBGAs/s1600/Screen%2BShot%2B2017-08-25%2Bat%2B21.23.48.png&gt](https://lh3.googleusercontent.com/-UO65BwWFeBE/WaB5lzVen3I/AAAAAAAAHc0/HO_TvJcuF8oPNuvGZv2L_nXmtJS8XT5ugCLcBGAs/s1600/Screen%2BShot%2B2017-08-25%2Bat%2B21.23.48.png&gt);
> > > > > > 
> > > > > > I know this is not a huge increase but it follows a pattern and is  
> > > > > > steady (will eventually cause the container to reboot due to hitting mem  
> > > > > > limit). In containers with more load it would leak memory faster.
> > > > > > 
> > > > > > \*What have I tried (without luck) so far:\*
> > > > > > 
> > > > > > &nbsp;&nbsp;&nbsp;- Logging to file instead of stdout.  
> > > > > > &nbsp;&nbsp;&nbsp;- Manually gc.collect() (as this doesn't release any memory I  
> > > > > > &nbsp;&nbsp;&nbsp;figure that it's not a reference problem but rather that the actual process  
> > > > > > &nbsp;&nbsp;&nbsp;wont release memory to the container once freed by the gc).  
> > > > > > &nbsp;&nbsp;&nbsp;- Removed the dependency injection and calling redis from the  
> > > > > > &nbsp;&nbsp;&nbsp;service itself.  
> > > > > > &nbsp;&nbsp;&nbsp;- Decrease and increase number of workers.  
> > > > > > &nbsp;&nbsp;&nbsp;- Simplify the service to just contain a timer, which logs a  
> > > > > > &nbsp;&nbsp;&nbsp;message every fifth second (still memory increases)
> > > > > > 
> > > > > > I understand that this might be perceived as to little information to  
> > > > > > give any valuable input. But I sincerely don't know how to proceed. Anyone  
> > > > > > got any ideas or see anything really strange with the example? Anyone  
> > > > > > experienced something similar?
> > > > > > 
> > > > > > If you came this far, thank you for reading and taking your time! Any  
> > > > > > help would be very appreciated!
> > > > > > 
> > > > > > Disclaimer: I don't necessarily believe this has to do with Nameko in  
> > > > > > particular but rather some combination of errors/dependencies/bad  
> > > > > > implementation.

---

<div class="post-metadata">

**Author:** ![Johan\_Frisell](https://avatars.discourse-cdn.com/v4/letter/j/838e76/32.png) [@Johan\_Frisell](https://discourse.nameko.io/u/Johan_Frisell)\
**Post date:** [September 5, 2017, 12:52pm UTC](https://discourse.nameko.io/t/process-not-releasing-memory-to-container/206/10 "2017-09-05T12:52:26Z")

</div>

I might, just want to make sure where the problem lies. Not sure it has to  
do with the actual backdoor cli command. Checked the sc and it was pretty  
straight forward, not much could go wrong there 🙂 I'll raise an issue if I  
can pin it down to something more specific.

Thanks for the tip!

> **···**
>
> Den tisdag 5 september 2017 kl. 09:26:43 UTC+2 skrev Matt Yule-Bennett:
> 
> > Oh interesting. The backdoor command is just a convenience wrapper around  
> > rlwrap, netcat and/or telnet. I tend to just use them directly, e.g.  
> > "rlwrap telnet localhost 3000"
> > 
> > If you have time, please raise an issue on the Github page to report the  
> > problems with the backdoor command.
> > 
> > On Tuesday, September 5, 2017 at 8:15:14 AM UTC+1, Johan Frisell wrote:
> > 
> > > Just wanted to get back in touch. I've not solved the problem yet, a  
> > > sprint just came in the way 🙂
> > > 
> > > Tried the backdoor, however I can't get it to work on in my docker  
> > > container. I get
> > > 
> > > netcat: invalid option -- '-'
> > > 
> > > whenever I try to connect to the backdoor. Probably some underlying  
> > > dependency missing in my container (ubuntu:16:04 based with rlwrap and  
> > > netcat installed), I'll get back as soon as I have anything to report.
> > > 
> > > Kindest,  
> > > Johan
> > > 
> > > Den måndag 28 augusti 2017 kl. 08:44:28 UTC+2 skrev Matt Yule-Bennett:
> > > 
> > > > Hi Johan,
> > > > 
> > > > Thanks for keeping us up to date with your investigations. Two useful  
> > > > looks I've found really helpful for finding memory leaks in the past are  
> > > > Pympler \<[Identifying memory leaks — Pympler 0.5 documentation](https://pythonhosted.org/Pympler/muppy.html#muppy&gt); and Objgraph  
> > > > \<[https://mg.pov.lt/objgraph/&gt;\](https://mg.pov.lt/objgraph/&gt;%5C).
> > > > 
> > > > Use the Nameko backdoor (which deserves its own page in the docs, but  
> > > > doesn't have one) to get a REPL inside the running process:
> > > > 
> > > > $ nameko run service --backdoor 3000  
> > > > ...
> > > > 
> > > > $ nameko backdoor 3000  
> > > > \>\>\> from pympler import muppy  
> > > > \>\>\> all\_objects = muppy.get\_objects() # will take a while  
> > > > \>\>\> from pympler import summary  
> > > > \>\>\> sum = summary.summarize(all\_objects)  
> > > > \>\>\> summary.print\_(sum)  
> > > > ...
> > > > 
> > > > Matt.
> > > > 
> > > > On Sunday, August 27, 2017 at 7:13:12 PM UTC+1, Johan Frisell wrote:
> > > > 
> > > > > Sorry for spamming this topic, I will take the time to compose a  
> > > > > summary once I have something valuable to share or maybe even a solution to  
> > > > > the problem. But I have made some testing without docker at all. Hosting my  
> > > > > nameko service on a pure EC2 instance with the same configuration as my  
> > > > > containers (dist, installed deps etc). The process doesn't seem to leak  
> > > > > memory in this case.
> > > > > 
> > > > > This narrows it down to Nameko in combination with Docker and/or  
> > > > > Kubernetes. I guess I'll have to host a pure Docker container as well. To  
> > > > > possibly rule out Kubernetes. Just wanted to give you an update on the  
> > > > > progress. If you have any ideas this would be most welcome!
> > > > > 
> > > > > Kindest,
> > > > > 
> > > > > Den söndag 27 augusti 2017 kl. 11:44:30 UTC+2 skrev David Szotten:
> > > > > 
> > > > > > Hi Johan,
> > > > > > 
> > > > > > I'm sorry you're having issues. However, there's still quite a lot  
> > > > > > going on in your example, and a few unknowns. Could you try reducing the  
> > > > > > example further? (e.g. removing redis, and even the event dispatching? Do  
> > > > > > you still see the problem? What if you remove nameko entirely and just log  
> > > > > > stuff in a loop?) With a 5 second timer, how long before you see anything  
> > > > > > happening?
> > > > > > 
> > > > > > Also, can you reproduce the memory increase without docker? (do you  
> > > > > > see the process memory usage grow over time) If not, what's your docker  
> > > > > > setup? Could you post a complete setup that we can `docker run` (or  
> > > > > > `docker-compose run`)? That may be easier as a github repo
> > > > > > 
> > > > > > Best,  
> > > > > > David
> > > > > > 
> > > > > > On Saturday, 26 August 2017 15:11:36 UTC+1, Johan Frisell wrote:
> > > > > > 
> > > > > > > \*First of a spec of the setup I'm running:\*
> > > > > > > 
> > > > > > > &nbsp;&nbsp;&nbsp;- AWS EC2 instances in the bottom layer.  
> > > > > > > &nbsp;&nbsp;&nbsp;- Kubernetes hosting Docker containers  
> > > > > > > &nbsp;&nbsp;&nbsp;- Clustered RabbitMQ (two servers) with an ELB in front.  
> > > > > > > &nbsp;&nbsp;&nbsp;- Redis  
> > > > > > > &nbsp;&nbsp;&nbsp;- Influx  
> > > > > > > &nbsp;&nbsp;&nbsp;- Nameko 2.6.0  
> > > > > > > &nbsp;&nbsp;&nbsp;  
> > > > > > > \*Problem:\*
> > > > > > > 
> > > > > > > In short I'm experiencing memory allocation problems. Where the  
> > > > > > > python process running Nameko wont release the memory back to the  
> > > > > > > container, causing the container to reboot when it hits its memory limit  
> > > > > > > (in my case 512MB). An easy way to reproduce this in my environment is to  
> > > > > > > log some random messages. The more I log the bigger the leak. However this  
> > > > > > > is not in any way a proof that the logging is the core problem, but maybe  
> > > > > > > an indication?
> > > > > > > 
> > > > > > > Below is a sample code I use to reproduce the problem. As you might  
> > > > > > > notice I have a circular event dispatch/handling, this is just for testing  
> > > > > > > purposes. My "real" code does not have this implementation but still  
> > > > > > > experience the same problem. Even with the most simple implementation, a  
> > > > > > > service with a simple logging.info("message") will experience a  
> > > > > > > memory leak.
> > > > > > > 
> > > > > > > \*service.py\*  
> > > > > > > import logging
> > > > > > > 
> > > > > > > from nameko.timer import timer  
> > > > > > > from nameko.events import event\_handler  
> > > > > > > from nameko.events import EventDispatcher
> > > > > > > 
> > > > > > > from service.redis\_cli import RedisDependency
> > > > > > > 
> > > > > > > class Service:  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;name = "service"  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;dispatcher = EventDispatcher()  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;r\_db = RedisDependency()
> > > > > > > 
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;@timer(interval=5)  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;def log(self):  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info("This is a rather long log message?")  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.dispatcher("publish", {"message": "hello world"})
> > > > > > > 
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;@event\_handler("uc\_sensor\_pipeline", "new\_reading")  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;def do\_log(self, payload):  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info(payload)
> > > > > > > 
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;@event\_handler("service", "publish")  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;def handle\_publish(self, payload):  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.r\_db.lpush('logging', '123', payload)  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;logging.info(payload['message'])
> > > > > > > 
> > > > > > > \*redis\_cli.py\*  
> > > > > > > from nameko.extensions import DependencyProvider
> > > > > > > 
> > > > > > > from redis import StrictRedis
> > > > > > > 
> > > > > > > class RedisWrapper:  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;def \_\_init\_\_(self, client):  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client = client
> > > > > > > 
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;def lpush(self, prefix, key, value):  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;full\_key = self.\_format\_key(prefix, key)  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client.lpush(full\_key, value)  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client.ltrim(full\_key, 0, 9)
> > > > > > > 
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;def get(self, prefix, key):  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;full\_key = self.\_format\_key(prefix, key)  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return self.client.lrange(full\_key, 0, -1)
> > > > > > > 
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;def \_decode(self, item):  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return { k.decode(): v.decode() for k, v in item.items() }
> > > > > > > 
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;def \_format\_key(self, prefix, key):  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return '{0}:{1}'.format(prefix, key)
> > > > > > > 
> > > > > > > class RedisDependency(DependencyProvider):  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;def setup(self):  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.client = StrictRedis.from\_url(  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.container.config.get('REDIS\_URI'))
> > > > > > > 
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;def get\_dependency(self, worker\_ctx):  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return RedisWrapper(self.client)
> > > > > > > 
> > > > > > > \*config.yml\*  
> > > > > > > AMQP\_URI:  
> > > > > > > amqp://${RMQ\_USER:guest}:${RMQ\_PASSWORD:guest}@${RMQ\_HOST:localhost}  
> > > > > > > REDIS\_URI: redis://myredishost:6379
> > > > > > > 
> > > > > > > LOGGING:  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;version: 1  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;handlers:  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;console:  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;class: logging.StreamHandler  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file:  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;class: logging.handlers.RotatingFileHandler  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;filename: service.log  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;maxBytes: 1048576  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;backupCount: 3  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;root:  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;level: INFO  
> > > > > > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;handlers: [console]
> > > > > > > 
> > > > > > > \*Overview of the mem allocation:\*
> > > > > > > 
> > > > > > > \<[https://lh3.googleusercontent.com/-UO65BwWFeBE/WaB5lzVen3I/AAAAAAAAHc0/HO\_TvJcuF8oPNuvGZv2L\_nXmtJS8XT5ugCLcBGAs/s1600/Screen%2BShot%2B2017-08-25%2Bat%2B21.23.48.png&gt](https://lh3.googleusercontent.com/-UO65BwWFeBE/WaB5lzVen3I/AAAAAAAAHc0/HO_TvJcuF8oPNuvGZv2L_nXmtJS8XT5ugCLcBGAs/s1600/Screen%2BShot%2B2017-08-25%2Bat%2B21.23.48.png&gt);
> > > > > > > 
> > > > > > > I know this is not a huge increase but it follows a pattern and is  
> > > > > > > steady (will eventually cause the container to reboot due to hitting mem  
> > > > > > > limit). In containers with more load it would leak memory faster.
> > > > > > > 
> > > > > > > \*What have I tried (without luck) so far:\*
> > > > > > > 
> > > > > > > &nbsp;&nbsp;&nbsp;- Logging to file instead of stdout.  
> > > > > > > &nbsp;&nbsp;&nbsp;- Manually gc.collect() (as this doesn't release any memory I  
> > > > > > > &nbsp;&nbsp;&nbsp;figure that it's not a reference problem but rather that the actual process  
> > > > > > > &nbsp;&nbsp;&nbsp;wont release memory to the container once freed by the gc).  
> > > > > > > &nbsp;&nbsp;&nbsp;- Removed the dependency injection and calling redis from the  
> > > > > > > &nbsp;&nbsp;&nbsp;service itself.  
> > > > > > > &nbsp;&nbsp;&nbsp;- Decrease and increase number of workers.  
> > > > > > > &nbsp;&nbsp;&nbsp;- Simplify the service to just contain a timer, which logs a  
> > > > > > > &nbsp;&nbsp;&nbsp;message every fifth second (still memory increases)
> > > > > > > 
> > > > > > > I understand that this might be perceived as to little information to  
> > > > > > > give any valuable input. But I sincerely don't know how to proceed. Anyone  
> > > > > > > got any ideas or see anything really strange with the example? Anyone  
> > > > > > > experienced something similar?
> > > > > > > 
> > > > > > > If you came this far, thank you for reading and taking your time! Any  
> > > > > > > help would be very appreciated!
> > > > > > > 
> > > > > > > Disclaimer: I don't necessarily believe this has to do with Nameko in  
> > > > > > > particular but rather some combination of errors/dependencies/bad  
> > > > > > > implementation.