Too many a_inodes from Entrypoint

1

Short version - an Entrypoint I wrote is creating an a_inode fd on each event trigger, eventually resulting in “OSError: [Errno 24] Too many open files”

My Entrypoint subscribes to data from an external vendor and provides a callback method. When new data arrives, this callback is triggered. Skipping the boring unpacking of the external vendor’s data, here is what i do:

    def handle_message(self, vendor_data):

        d = unpack_vendor_data(vendor_data)

        event = Event()
        handle_result = partial(self.handle_result, event)
        args = (d)
        kwargs = {}

        self.container.spawn_worker(self, args, kwargs, handle_result=handle_result)

        result = event.wait()
        return result

    def handle_result(self, event, worker_ctx, result, exc_info):
        event.send(result, exc_info)
        return result, exc_info

My service method is trivial:

class VendorHandler:

    name = "vendorhandler"
    dispatch = EventDispatcher()

    @vendor_event
    def publish_vendor_msg(self, msg):
        self.dispatch("my_topic", msg)

Monitoring shows each time an event is triggered, a new a_inode item is created and left open:

lsof -p 5318

....
nameko  5318 afischbein  158u  a_inode               0,11        0    8047 [eventpoll]
nameko  5318 afischbein  159u  a_inode               0,11        0    8047 [eventpoll]
nameko  5318 afischbein  160u  a_inode               0,11        0    8047 [eventpoll]

Eventually the file limit is reached and the process exits with:

Exception in event handler:
Traceback (most recent call last):
  File "/home/devadmin/.conda/envs/_bpipepub/lib/python3.6/site-packages/eventlet/hubs/__init__.py", line 111, in get_hub
AttributeError: '_thread._local' object has no attribute 'hub'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/devadmin/.conda/envs/_bpipepub/lib/python3.6/site-packages/blpapi/session.py", line 100, in __dispatchEvent
  File "/home/devadmin/.conda/envs/_bpipepub/lib/python3.6/site-packages/bpipepub/entrypoint.py", line 275, in processEvent
  File "/home/devadmin/.conda/envs/_bpipepub/lib/python3.6/site-packages/bpipepub/entrypoint.py", line 219, in handle_message
  File "/home/devadmin/.conda/envs/_bpipepub/lib/python3.6/site-packages/nameko-2.11.0-py3.6.egg/nameko/containers.py", line 345, in spawn_worker
  File "/home/devadmin/.conda/envs/_bpipepub/lib/python3.6/site-packages/eventlet/greenpool.py", line 78, in spawn
  File "/home/devadmin/.conda/envs/_bpipepub/lib/python3.6/site-packages/eventlet/greenthread.py", line 50, in spawn
  File "/home/devadmin/.conda/envs/_bpipepub/lib/python3.6/site-packages/eventlet/hubs/__init__.py", line 117, in get_hub
  File "/home/devadmin/.conda/envs/_bpipepub/lib/python3.6/site-packages/eventlet/hubs/epolls.py", line 38, in __init__
OSError: [Errno 24] Too many open files

My module environment:

amqp                      2.3.2                      py_1    conda-forge
asn1crypto                0.24.0                   py36_0
backports                 1.0                      py36_1
backports.os              0.1.1                    py36_0
blas                      1.0                         mkl
blpapi                    3.9.2           py36_blpapicpp3.8.18.1_1    conda-forge
ca-certificates           2018.03.07                    0
certifi                   2018.10.15               py36_0
cffi                      1.11.5           py36he75722e_1
chardet                   3.0.4                    py36_1
cryptography              2.3.1            py36h1ba5d50_2
eventlet                  0.23.0                py36_1000    conda-forge
greenlet                  0.4.15           py36h7b6447c_0
idna                      2.7                      py36_0
importlib_metadata        0.6                      py36_0
intel-openmp              2019.0                      118
kombu                     4.2.1                      py_0    conda-forge
libedit                   3.1.20170329         h6b74fdf_2
libffi                    3.2.1                hd88cf55_4
libgcc-ng                 8.2.0                hdf63c60_1
libgfortran-ng            7.3.0                hdf63c60_0
libstdcxx-ng              8.2.0                hdf63c60_1
mkl                       2019.0                      118
mkl_fft                   1.0.6            py36h7dd41cf_0
mkl_random                1.0.1            py36h4414c95_1
mock                      2.0.0                    py36_0
mysqlclient               1.3.12               ubuntu1604    http://uslinuxprod01/repo/python/main
nameko                    2.11.0                   py36_2    http://uslinuxprod01/repo/python/main
ncurses                   6.1                  hf484d3e_0
numpy                     1.15.4           py36h1d66e8a_0
numpy-base                1.15.4           py36h81de0dd_0
openssl                   1.1.1                h7b6447c_0
pandas                    0.20.3           py36h6022372_2
path.py                   11.5.0                   py36_0
pbr                       5.1.0                    py36_0
pip                       18.1                     py36_0
pycparser                 2.19                     py36_0
pyopenssl                 18.0.0                   py36_0
pysocks                   1.6.8                    py36_0
python                    3.6.7                h0371630_0
python-dateutil           2.7.5                    py36_0
pytz                      2018.7                   py36_0
pyyaml                    3.13             py36h14c3975_0
readline                  7.0                  h7b6447c_5
requests                  2.20.0                   py36_0
setuptools                40.5.0                   py36_0
simplejson                3.16.0           py36h14c3975_0
six                       1.11.0                   py36_1
sqlalchemy                1.2.13           py36h7b6447c_0
sqlite                    3.25.2               h7b6447c_0
tk                        8.6.8                hbc83047_0
ujson                     1.35             py36h14c3975_0
urllib3                   1.23                     py36_0
vine                      1.1.4                      py_1    conda-forge
werkzeug                  0.14.1                   py36_0
wheel                     0.32.2                   py36_0
wrapt                     1.10.11          py36h14c3975_2
xz                        5.2.4                h14c3975_4
yaml                      0.1.7                had09818_2
zlib                      1.2.11               ha838bed_2

Linux version:

~$ cat /proc/version
Linux version 4.4.0-96-generic (buildd@lgw01-10) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) ) #119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017

Any support you can provide would be most appreciated!

2

I think it’s the rest of your Entrypoint that is the interesting bit. There’s nothing shown in your snippet here that relates to files… Where does the vendor_data passed to handle_message come from?

But also, how many concurrent workers are you running? When the service crashes, have previous workers definitely terminated or could they still be running?

3

Hi Matt - thanks for the quick response.
Short version - no problem in Nameko, looks like interaction between vendor library and Eventlet - creating new threads from within the vendor library is what appears to cause the file leak. Switching to synchronous/single-threaded mode in the vendor library resolved the issue, so feel free to delete this post if not relevant to Nameko users.

Thanks again.

1 Like