RecoverableConnectionError('connection already closed') and all workers killed!

googlegroup
1

I have an some event handlers on a nameko service which do a long running
task each (5-15mins each).

Seemingly randomly the following set of errors happens:

<https://lh3.googleusercontent.com/-OzkmfMx9hwI/WfrlOPijE7I/AAAAAAAAAAc/s3kz32tIw5gT_K-l85tXezsBcO4BmrH1QCLcBGAs/s1600/bad.png.png>

My service has HEARTBEAT: 3 in the config. HAProxy in front of rabbitmq has
3hr timeout set.

Does anyone know what is going on here?

thanks

2

Which version of Nameko? Please also post a stack trace.

···

On Thursday, November 2, 2017 at 9:35:32 AM UTC, Richard wrote:

I have an some event handlers on a nameko service which do a long running
task each (5-15mins each).

Seemingly randomly the following set of errors happens:

<https://lh3.googleusercontent.com/-OzkmfMx9hwI/WfrlOPijE7I/AAAAAAAAAAc/s3kz32tIw5gT_K-l85tXezsBcO4BmrH1QCLcBGAs/s1600/bad.png.png&gt;

My service has HEARTBEAT: 3 in the config. HAProxy in front of rabbitmq
has 3hr timeout set.

Does anyone know what is going on here?

thanks

3

2.7.0 version

RecoverableConnectionError: connection already closed
  File "bin/nameko_extras", line 11, in <module>
    sys.exit(main())
  File "nameko_extras/cli/main.py", line 24, in main
    run.main(args)
  File "nameko_extras/cli/run.py", line 48, in main
    run(services, config, **kwargs)
  File "nameko/cli/run.py", line 140, in run
    runnlet.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/runners.py", line 96, in wait
    SpawningProxy(self.containers, abort_on_error=True).wait()
  File "nameko/utils/__init__.py", line 184, in spawning_method
    return list(fail_fast_imap(pool, call, self._items))
  File "nameko/utils/__init__.py", line 153, in fail_fast_imap
    eventlet.getcurrent().throw(*exc_info)
  File "nameko/utils/__init__.py", line 132, in handle_result
    thread_result = finished_thread.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/utils/__init__.py", line 181, in call
    return getattr(item, name)(*args, **kwargs)
  File "nameko/containers.py", line 312, in wait
    return self._died.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "nameko/containers.py", line 466, in _handle_thread_exited
    gt.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/containers.py", line 399, in _run_worker
    worker_ctx, result, exc_info)
  File "nameko/messaging.py", line 461, in handle_result
    self.handle_message_processed(message, result, exc_info)
  File "nameko/messaging.py", line 469, in handle_message_processed
    self.queue_consumer.ack_message(message)
  File "nameko/utils/retry.py", line 46, in wrapper
    return wrapped(*args, **kwargs)
  File "nameko/messaging.py", line 315, in ack_message
    message.ack()
  File "kombu/message.py", line 88, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "amqp/channel.py", line 1584, in basic_ack
    self._send_method((60, 80), args)
  File "amqp/abstract_channel.py", line 50, in _send_method
    raise RecoverableConnectionError('connection already closed')

···

On Thursday, November 2, 2017 at 10:48:40 AM UTC, Matt Yule-Bennett wrote:

Which version of Nameko? Please also post a stack trace.

On Thursday, November 2, 2017 at 9:35:32 AM UTC, Richard wrote:

I have an some event handlers on a nameko service which do a long running
task each (5-15mins each).

Seemingly randomly the following set of errors happens:

<https://lh3.googleusercontent.com/-OzkmfMx9hwI/WfrlOPijE7I/AAAAAAAAAAc/s3kz32tIw5gT_K-l85tXezsBcO4BmrH1QCLcBGAs/s1600/bad.png.png&gt;

My service has HEARTBEAT: 3 in the config. HAProxy in front of rabbitmq
has 3hr timeout set.

Does anyone know what is going on here?

thanks

4

Sorry the previous message was missing all of the trace:

RecoverableConnectionError: connection already closed
  File "bin/nameko_extras", line 11, in <module>
    sys.exit(main())
  File "nameko_extras/cli/main.py", line 24, in main
    run.main(args)
  File "nameko_extras/cli/run.py", line 48, in main
    run(services, config, **kwargs)
  File "nameko/cli/run.py", line 140, in run
    runnlet.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/runners.py", line 96, in wait
    SpawningProxy(self.containers, abort_on_error=True).wait()
  File "nameko/utils/__init__.py", line 184, in spawning_method
    return list(fail_fast_imap(pool, call, self._items))
  File "nameko/utils/__init__.py", line 153, in fail_fast_imap
    eventlet.getcurrent().throw(*exc_info)
  File "nameko/utils/__init__.py", line 132, in handle_result
    thread_result = finished_thread.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/utils/__init__.py", line 181, in call
    return getattr(item, name)(*args, **kwargs)
  File "nameko/containers.py", line 312, in wait
    return self._died.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "nameko/containers.py", line 466, in _handle_thread_exited
    gt.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/containers.py", line 399, in _run_worker
    worker_ctx, result, exc_info)
  File "nameko/messaging.py", line 461, in handle_result
    self.handle_message_processed(message, result, exc_info)
  File "nameko/messaging.py", line 469, in handle_message_processed
    self.queue_consumer.ack_message(message)
  File "nameko/utils/retry.py", line 46, in wrapper
    return wrapped(*args, **kwargs)
  File "nameko/messaging.py", line 315, in ack_message
    message.ack()
  File "kombu/message.py", line 88, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "amqp/channel.py", line 1584, in basic_ack
    self._send_method((60, 80), args)
  File "amqp/abstract_channel.py", line 50, in _send_method
    raise RecoverableConnectionError('connection already closed')

···

On Thursday, November 2, 2017 at 10:57:03 AM UTC, Richard wrote:

2.7.0 version

RecoverableConnectionError: connection already closed
  File "bin/nameko_extras", line 11, in <module>
    sys.exit(main())
  File "nameko_extras/cli/main.py", line 24, in main
    run.main(args)
  File "nameko_extras/cli/run.py", line 48, in main
    run(services, config, **kwargs)
  File "nameko/cli/run.py", line 140, in run
    runnlet.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/runners.py", line 96, in wait
    SpawningProxy(self.containers, abort_on_error=True).wait()
  File "nameko/utils/__init__.py", line 184, in spawning_method
    return list(fail_fast_imap(pool, call, self._items))
  File "nameko/utils/__init__.py", line 153, in fail_fast_imap
    eventlet.getcurrent().throw(*exc_info)
  File "nameko/utils/__init__.py", line 132, in handle_result
    thread_result = finished_thread.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/utils/__init__.py", line 181, in call
    return getattr(item, name)(*args, **kwargs)
  File "nameko/containers.py", line 312, in wait
    return self._died.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "nameko/containers.py", line 466, in _handle_thread_exited
    gt.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/containers.py", line 399, in _run_worker
    worker_ctx, result, exc_info)
  File "nameko/messaging.py", line 461, in handle_result
    self.handle_message_processed(message, result, exc_info)
  File "nameko/messaging.py", line 469, in handle_message_processed
    self.queue_consumer.ack_message(message)
  File "nameko/utils/retry.py", line 46, in wrapper
    return wrapped(*args, **kwargs)
  File "nameko/messaging.py", line 315, in ack_message
    message.ack()
  File "kombu/message.py", line 88, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "amqp/channel.py", line 1584, in basic_ack
    self._send_method((60, 80), args)
  File "amqp/abstract_channel.py", line 50, in _send_method
    raise <span style="color: #606;" class="styled-

5

just to rule this out, are these long running tasks hogging the cpu by any
chance (blocking the event loop)?

···

On Thursday, 2 November 2017 10:57:49 UTC, Richard wrote:

Sorry the previous message was missing all of the trace:

RecoverableConnectionError: connection already closed
  File "bin/nameko_extras", line 11, in <module>
    sys.exit(main())
  File "nameko_extras/cli/main.py", line 24, in main
    run.main(args)
  File "nameko_extras/cli/run.py", line 48, in main
    run(services, config, **kwargs)
  File "nameko/cli/run.py", line 140, in run
    runnlet.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/runners.py", line 96, in wait
    SpawningProxy(self.containers, abort_on_error=True).wait()
  File "nameko/utils/__init__.py", line 184, in spawning_method
    return list(fail_fast_imap(pool, call, self._items))
  File "nameko/utils/__init__.py", line 153, in fail_fast_imap
    eventlet.getcurrent().throw(*exc_info)
  File "nameko/utils/__init__.py", line 132, in handle_result
    thread_result = finished_thread.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/utils/__init__.py", line 181, in call
    return getattr(item, name)(*args, **kwargs)
  File "nameko/containers.py", line 312, in wait
    return self._died.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "nameko/containers.py", line 466, in _handle_thread_exited
    gt.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/containers.py", line 399, in _run_worker
    worker_ctx, result, exc_info)
  File "nameko/messaging.py", line 461, in handle_result
    self.handle_message_processed(message, result, exc_info)
  File "nameko/messaging.py", line 469, in handle_message_processed
    self.queue_consumer.ack_message(message)
  File "nameko/utils/retry.py", line 46, in wrapper
    return wrapped(*args, **kwargs)
  File "nameko/messaging.py", line 315, in ack_message
    message.ack()
  File "kombu/message.py", line 88, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "amqp/channel.py", line 1584, in basic_ack
    self._send_method((60, 80), args)
  File "amqp/abstract_channel.py", line 50, in _send_method
    raise RecoverableConnectionError('connection already closed')

On Thursday, November 2, 2017 at 10:57:03 AM UTC, Richard wrote:

2.7.0 version

RecoverableConnectionError: connection already closed
  File "bin/nameko_extras", line 11, in <module>
    sys.exit(main())
  File "nameko_extras/cli/main.py", line 24, in main
    run.main(args)
  File "nameko_extras/cli/run.py", line 48, in main
    run(services, config, **kwargs)
  File "nameko/cli/run.py", line 140, in run
    runnlet.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/runners.py", line 96, in wait
    SpawningProxy(self.containers, abort_on_error=True).wait()
  File "nameko/utils/__init__.py", line 184, in spawning_method
    return list(fail_fast_imap(pool, call, self._items))
  File "nameko/utils/__init__.py", line 153, in fail_fast_imap
    eventlet.getcurrent().throw(*exc_info)
  File "nameko/utils/__init__.py", line 132, in handle_result
    thread_result = finished_thread.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/utils/__init__.py", line 181, in call
    return getattr(item, name)(*args, **kwargs)
  File "nameko/containers.py", line 312, in wait
    return self._died.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "nameko/containers.py", line 466, in _handle_thread_exited
    gt.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/containers.py", line 399, in _run_worker
    worker_ctx, result, exc_info)
  File "nameko/messaging.py", line 461, in handle_result
    self.handle_message_processed(message, result, exc_info)
  File "nameko/messaging.py", line 469, in handle_message_processed
    self.queue_consumer.ack_message(message)
  File "nameko/utils/retry.py", line 46, in wrapper
    return wrapped(*args, **kwargs)
  File "nameko/messaging.py", line 315, in ack_message
    message.ack()
  File "kombu/message.py", line 88, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "amqp/channel.py", line 1584, in basic_ack
    self._send_method((60, 80), args)
  File "amqp/abstract_channel.py", line 50, in _send_method
    raise <span style="color: #606;" class="styled-

6

Average CPU is 70%

···

On Thursday, November 2, 2017 at 11:05:52 AM UTC, David Szotten wrote:

just to rule this out, are these long running tasks hogging the cpu by any
chance (blocking the event loop)?

On Thursday, 2 November 2017 10:57:49 UTC, Richard wrote:

Sorry the previous message was missing all of the trace:

RecoverableConnectionError: connection already closed
  File "bin/nameko_extras", line 11, in <module>
    sys.exit(main())
  File "nameko_extras/cli/main.py", line 24, in main
    run.main(args)
  File "nameko_extras/cli/run.py", line 48, in main
    run(services, config, **kwargs)
  File "nameko/cli/run.py", line 140, in run
    runnlet.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/runners.py", line 96, in wait
    SpawningProxy(self.containers, abort_on_error=True).wait()
  File "nameko/utils/__init__.py", line 184, in spawning_method
    return list(fail_fast_imap(pool, call, self._items))
  File "nameko/utils/__init__.py", line 153, in fail_fast_imap
    eventlet.getcurrent().throw(*exc_info)
  File "nameko/utils/__init__.py", line 132, in handle_result
    thread_result = finished_thread.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/utils/__init__.py", line 181, in call
    return getattr(item, name)(*args, **kwargs)
  File "nameko/containers.py", line 312, in wait
    return self._died.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "nameko/containers.py", line 466, in _handle_thread_exited
    gt.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/containers.py", line 399, in _run_worker
    worker_ctx, result, exc_info)
  File "nameko/messaging.py", line 461, in handle_result
    self.handle_message_processed(message, result, exc_info)
  File "nameko/messaging.py", line 469, in handle_message_processed
    self.queue_consumer.ack_message(message)
  File "nameko/utils/retry.py", line 46, in wrapper
    return wrapped(*args, **kwargs)
  File "nameko/messaging.py", line 315, in ack_message
    message.ack()
  File "kombu/message.py", line 88, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "amqp/channel.py", line 1584, in basic_ack
    self._send_method((60, 80), args)
  File "amqp/abstract_channel.py", line 50, in _send_method
    raise RecoverableConnectionError('connection already closed')

On Thursday, November 2, 2017 at 10:57:03 AM UTC, Richard wrote:

2.7.0 version

RecoverableConnectionError: connection already closed
  File "bin/nameko_extras", line 11, in <module>
    sys.exit(main())
  File "nameko_extras/cli/main.py", line 24, in main
    run.main(args)
  File "nameko_extras/cli/run.py", line 48, in main
    run(services, config, **kwargs)
  File "nameko/cli/run.py", line 140, in run
    runnlet.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/runners.py", line 96, in wait
    SpawningProxy(self.containers, abort_on_error=True).wait()
  File "nameko/utils/__init__.py", line 184, in spawning_method
    return list(fail_fast_imap(pool, call, self._items))
  File "nameko/utils/__init__.py", line 153, in fail_fast_imap
    eventlet.getcurrent().throw(*exc_info)
  File "nameko/utils/__init__.py", line 132, in handle_result
    thread_result = finished_thread.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/utils/__init__.py", line 181, in call
    return getattr(item, name)(*args, **kwargs)
  File "nameko/containers.py", line 312, in wait
    return self._died.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "nameko/containers.py", line 466, in _handle_thread_exited
    gt.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/containers.py", line 399, in _run_worker
    worker_ctx, result, exc_info)
  File "nameko/messaging.py", line 461, in handle_result
    self.handle_message_processed(message, result, exc_info)
  File "nameko/messaging.py", line 469, in handle_message_processed
    self.queue_consumer.ack_message(message)
  File "nameko/utils/retry.py", line 46, in wrapper
    return wrapped(*args, **kwargs)
  File "nameko/messaging.py", line 315, in ack_message
    message.ack()
  File "kombu/message.py", line 88, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "amqp/channel.py", line 1584, in basic_ack
    self._send_method((60, 80), args)
  File "amqp/abstract_channel.py", line 50, in _send_method
    raise <span style="color: #606;" class="styled-

7

Average CPU is 70%

This isn't enough to tell you anything. On a multi-core box you can
completely max out one CPU and still see a low average.

If the workload inside the event handler isn't doing any I/O it is likely
to be stealing the CPU long enough to starve the thread that's sending your
heartbeats. Once two heartbeats are missed the broker will close the
connection from the other end. You'll be able to see this happening if you
check your rabbit logs.

I bet that is what's happening in this case. However, we should handle this
error better, since it is recoverable.

···

On Thursday, November 2, 2017 at 11:09:10 AM UTC, Richard wrote:

On Thursday, November 2, 2017 at 11:05:52 AM UTC, David Szotten wrote:

just to rule this out, are these long running tasks hogging the cpu by
any chance (blocking the event loop)?

On Thursday, 2 November 2017 10:57:49 UTC, Richard wrote:

Sorry the previous message was missing all of the trace:

RecoverableConnectionError: connection already closed
  File "bin/nameko_extras", line 11, in <module>
    sys.exit(main())
  File "nameko_extras/cli/main.py", line 24, in main
    run.main(args)
  File "nameko_extras/cli/run.py", line 48, in main
    run(services, config, **kwargs)
  File "nameko/cli/run.py", line 140, in run
    runnlet.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/runners.py", line 96, in wait
    SpawningProxy(self.containers, abort_on_error=True).wait()
  File "nameko/utils/__init__.py", line 184, in spawning_method
    return list(fail_fast_imap(pool, call, self._items))
  File "nameko/utils/__init__.py", line 153, in fail_fast_imap
    eventlet.getcurrent().throw(*exc_info)
  File "nameko/utils/__init__.py", line 132, in handle_result
    thread_result = finished_thread.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/utils/__init__.py", line 181, in call
    return getattr(item, name)(*args, **kwargs)
  File "nameko/containers.py", line 312, in wait
    return self._died.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "nameko/containers.py", line 466, in _handle_thread_exited
    gt.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/containers.py", line 399, in _run_worker
    worker_ctx, result, exc_info)
  File "nameko/messaging.py", line 461, in handle_result
    self.handle_message_processed(message, result, exc_info)
  File "nameko/messaging.py", line 469, in handle_message_processed
    self.queue_consumer.ack_message(message)
  File "nameko/utils/retry.py", line 46, in wrapper
    return wrapped(*args, **kwargs)
  File "nameko/messaging.py", line 315, in ack_message
    message.ack()
  File "kombu/message.py", line 88, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "amqp/channel.py", line 1584, in basic_ack
    self._send_method((60, 80), args)
  File "amqp/abstract_channel.py", line 50, in _send_method
    raise RecoverableConnectionError('connection already closed')

On Thursday, November 2, 2017 at 10:57:03 AM UTC, Richard wrote:

2.7.0 version

RecoverableConnectionError: connection already closed
  File "bin/nameko_extras", line 11, in <module>
    sys.exit(main())
  File "nameko_extras/cli/main.py", line 24, in main
    run.main(args)
  File "nameko_extras/cli/run.py", line 48, in main
    run(services, config, **kwargs)
  File "nameko/cli/run.py", line 140, in run
    runnlet.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/runners.py", line 96, in wait
    SpawningProxy(self.containers, abort_on_error=True).wait()
  File "nameko/utils/__init__.py", line 184, in spawning_method
    return list(fail_fast_imap(pool, call, self._items))
  File "nameko/utils/__init__.py", line 153, in fail_fast_imap
    eventlet.getcurrent().throw(*exc_info)
  File "nameko/utils/__init__.py", line 132, in handle_result
    thread_result = finished_thread.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/utils/__init__.py", line 181, in call
    return getattr(item, name)(*args, **kwargs)
  File "nameko/containers.py", line 312, in wait
    return self._died.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "nameko/containers.py", line 466, in _handle_thread_exited
    gt.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/containers.py", line 399, in _run_worker
    worker_ctx, result, exc_info)
  File "nameko/messaging.py", line 461, in handle_result
    self.handle_message_processed(message, result, exc_info)
  File "nameko/messaging.py", line 469, in handle_message_processed
    self.queue_consumer.ack_message(message)
  File "nameko/utils/retry.py", line 46, in wrapper
    return wrapped(*args, **kwargs)
  File "nameko/messaging.py", line 315, in ack_message
    message.ack()
  File "kombu/message.py", line 88, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "amqp/channel.py", line 1584, in basic_ack
    self._send_method((60, 80), args)
  File "amqp/abstract_channel.py", line 50, in _send_method
    raise <span style="color: #606;" class="styled-

8

you can check this
using debug – Debugging tools for Eventlet — Eventlet 0.33.0 documentation

···

On Thursday, 2 November 2017 11:15:10 UTC, Matt Yule-Bennett wrote:

On Thursday, November 2, 2017 at 11:09:10 AM UTC, Richard wrote:

Average CPU is 70%

This isn't enough to tell you anything. On a multi-core box you can
completely max out one CPU and still see a low average.

If the workload inside the event handler isn't doing any I/O it is likely
to be stealing the CPU long enough to starve the thread that's sending your
heartbeats. Once two heartbeats are missed the broker will close the
connection from the other end. You'll be able to see this happening if you
check your rabbit logs.

I bet that is what's happening in this case. However, we should handle
this error better, since it is recoverable.

On Thursday, November 2, 2017 at 11:05:52 AM UTC, David Szotten wrote:

just to rule this out, are these long running tasks hogging the cpu by
any chance (blocking the event loop)?

On Thursday, 2 November 2017 10:57:49 UTC, Richard wrote:

Sorry the previous message was missing all of the trace:

RecoverableConnectionError: connection already closed
  File "bin/nameko_extras", line 11, in <module>
    sys.exit(main())
  File "nameko_extras/cli/main.py", line 24, in main
    run.main(args)
  File "nameko_extras/cli/run.py", line 48, in main
    run(services, config, **kwargs)
  File "nameko/cli/run.py", line 140, in run
    runnlet.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/runners.py", line 96, in wait
    SpawningProxy(self.containers, abort_on_error=True).wait()
  File "nameko/utils/__init__.py", line 184, in spawning_method
    return list(fail_fast_imap(pool, call, self._items))
  File "nameko/utils/__init__.py", line 153, in fail_fast_imap
    eventlet.getcurrent().throw(*exc_info)
  File "nameko/utils/__init__.py", line 132, in handle_result
    thread_result = finished_thread.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/utils/__init__.py", line 181, in call
    return getattr(item, name)(*args, **kwargs)
  File "nameko/containers.py", line 312, in wait
    return self._died.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "nameko/containers.py", line 466, in _handle_thread_exited
    gt.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/containers.py", line 399, in _run_worker
    worker_ctx, result, exc_info)
  File "nameko/messaging.py", line 461, in handle_result
    self.handle_message_processed(message, result, exc_info)
  File "nameko/messaging.py", line 469, in handle_message_processed
    self.queue_consumer.ack_message(message)
  File "nameko/utils/retry.py", line 46, in wrapper
    return wrapped(*args, **kwargs)
  File "nameko/messaging.py", line 315, in ack_message
    message.ack()
  File "kombu/message.py", line 88, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "amqp/channel.py", line 1584, in basic_ack
    self._send_method((60, 80), args)
  File "amqp/abstract_channel.py", line 50, in _send_method
    raise RecoverableConnectionError('connection already closed')

On Thursday, November 2, 2017 at 10:57:03 AM UTC, Richard wrote:

2.7.0 version

RecoverableConnectionError: connection already closed
  File "bin/nameko_extras", line 11, in <module>
    sys.exit(main())
  File "nameko_extras/cli/main.py", line 24, in main
    run.main(args)
  File "nameko_extras/cli/run.py", line 48, in main
    run(services, config, **kwargs)
  File "nameko/cli/run.py", line 140, in run
    runnlet.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/runners.py", line 96, in wait
    SpawningProxy(self.containers, abort_on_error=True).wait()
  File "nameko/utils/__init__.py", line 184, in spawning_method
    return list(fail_fast_imap(pool, call, self._items))
  File "nameko/utils/__init__.py", line 153, in fail_fast_imap
    eventlet.getcurrent().throw(*exc_info)
  File "nameko/utils/__init__.py", line 132, in handle_result
    thread_result = finished_thread.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/utils/__init__.py", line 181, in call
    return getattr(item, name)(*args, **kwargs)
  File "nameko/containers.py", line 312, in wait
    return self._died.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "nameko/containers.py", line 466, in _handle_thread_exited
    gt.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/containers.py", line 399, in _run_worker
    worker_ctx, result, exc_info)
  File "nameko/messaging.py", line 461, in handle_result
    self.handle_message_processed(message, result, exc_info)
  File "nameko/messaging.py", line 469, in handle_message_processed
    self.queue_consumer.ack_message(message)
  File "nameko/utils/retry.py", line 46, in wrapper
    return wrapped(*args, **kwargs)
  File "nameko/messaging.py", line 315, in ack_message
    message.ack()
  File "kombu/message.py", line 88, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "amqp/channel.py", line 1584, in basic_ack
    self._send_method((60, 80), args)
  File "amqp/abstract_channel.py", line 50, in _send_method
    raise <span style="color: #606;" class="styled-

9

What will SIGALARM do exactly?

···

On Thursday, November 2, 2017 at 11:16:45 AM UTC, David Szotten wrote:

you can check this using
debug – Debugging tools for Eventlet — Eventlet 0.33.0 documentation

On Thursday, 2 November 2017 11:15:10 UTC, Matt Yule-Bennett wrote:

On Thursday, November 2, 2017 at 11:09:10 AM UTC, Richard wrote:

Average CPU is 70%

This isn't enough to tell you anything. On a multi-core box you can
completely max out one CPU and still see a low average.

If the workload inside the event handler isn't doing any I/O it is likely
to be stealing the CPU long enough to starve the thread that's sending your
heartbeats. Once two heartbeats are missed the broker will close the
connection from the other end. You'll be able to see this happening if you
check your rabbit logs.

I bet that is what's happening in this case. However, we should handle
this error better, since it is recoverable.

On Thursday, November 2, 2017 at 11:05:52 AM UTC, David Szotten wrote:

just to rule this out, are these long running tasks hogging the cpu by
any chance (blocking the event loop)?

On Thursday, 2 November 2017 10:57:49 UTC, Richard wrote:

Sorry the previous message was missing all of the trace:

RecoverableConnectionError: connection already closed
  File "bin/nameko_extras", line 11, in <module>
    sys.exit(main())
  File "nameko_extras/cli/main.py", line 24, in main
    run.main(args)
  File "nameko_extras/cli/run.py", line 48, in main
    run(services, config, **kwargs)
  File "nameko/cli/run.py", line 140, in run
    runnlet.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/runners.py", line 96, in wait
    SpawningProxy(self.containers, abort_on_error=True).wait()
  File "nameko/utils/__init__.py", line 184, in spawning_method
    return list(fail_fast_imap(pool, call, self._items))
  File "nameko/utils/__init__.py", line 153, in fail_fast_imap
    eventlet.getcurrent().throw(*exc_info)
  File "nameko/utils/__init__.py", line 132, in handle_result
    thread_result = finished_thread.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/utils/__init__.py", line 181, in call
    return getattr(item, name)(*args, **kwargs)
  File "nameko/containers.py", line 312, in wait
    return self._died.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "nameko/containers.py", line 466, in _handle_thread_exited
    gt.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/containers.py", line 399, in _run_worker
    worker_ctx, result, exc_info)
  File "nameko/messaging.py", line 461, in handle_result
    self.handle_message_processed(message, result, exc_info)
  File "nameko/messaging.py", line 469, in handle_message_processed
    self.queue_consumer.ack_message(message)
  File "nameko/utils/retry.py", line 46, in wrapper
    return wrapped(*args, **kwargs)
  File "nameko/messaging.py", line 315, in ack_message
    message.ack()
  File "kombu/message.py", line 88, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "amqp/channel.py", line 1584, in basic_ack
    self._send_method((60, 80), args)
  File "amqp/abstract_channel.py", line 50, in _send_method
    raise RecoverableConnectionError('connection already closed')

On Thursday, November 2, 2017 at 10:57:03 AM UTC, Richard wrote:

2.7.0 version

RecoverableConnectionError: connection already closed
  File "bin/nameko_extras", line 11, in <module>
    sys.exit(main())
  File "nameko_extras/cli/main.py", line 24, in main
    run.main(args)
  File "nameko_extras/cli/run.py", line 48, in main
    run(services, config, **kwargs)
  File "nameko/cli/run.py", line 140, in run
    runnlet.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/runners.py", line 96, in wait
    SpawningProxy(self.containers, abort_on_error=True).wait()
  File "nameko/utils/__init__.py", line 184, in spawning_method
    return list(fail_fast_imap(pool, call, self._items))
  File "nameko/utils/__init__.py", line 153, in fail_fast_imap
    eventlet.getcurrent().throw(*exc_info)
  File "nameko/utils/__init__.py", line 132, in handle_result
    thread_result = finished_thread.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/utils/__init__.py", line 181, in call
    return getattr(item, name)(*args, **kwargs)
  File "nameko/containers.py", line 312, in wait
    return self._died.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "nameko/containers.py", line 466, in _handle_thread_exited
    gt.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/containers.py", line 399, in _run_worker
    worker_ctx, result, exc_info)
  File "nameko/messaging.py", line 461, in handle_result
    self.handle_message_processed(message, result, exc_info)
  File "nameko/messaging.py", line 469, in handle_message_processed
    self.queue_consumer.ack_message(message)
  File "nameko/utils/retry.py", line 46, in wrapper
    return wrapped(*args, **kwargs)
  File "nameko/messaging.py", line 315, in ack_message
    message.ack()
  File "kombu/message.py", line 88, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "amqp/channel.py", line 1584, in basic_ack
    self._send_method((60, 80), args)
  File "amqp/abstract_channel.py", line 50, in _send_method
    raise <span style="color: #606;" class="styled-

10

Cannot even build CI for the project with this change

    def alarm_handler(signum, frame): import inspect> raise RuntimeError("Blocking detector ALARMED at" + str(inspect.getframeinfo(frame)))E RuntimeError: Blocking detector ALARMED atTraceback(filename='/usr/local/lib/python3.6/posixpath.py', lineno=82, function='join', code_context=[' if not p:\n'], index=0)

where should debug.hub_blocking_detection(True) be placed?

···

On Thursday, November 2, 2017 at 11:21:48 AM UTC, Richard wrote:

What will SIGALARM do exactly?

On Thursday, November 2, 2017 at 11:16:45 AM UTC, David Szotten wrote:

you can check this using
debug – Debugging tools for Eventlet — Eventlet 0.33.0 documentation

On Thursday, 2 November 2017 11:15:10 UTC, Matt Yule-Bennett wrote:

On Thursday, November 2, 2017 at 11:09:10 AM UTC, Richard wrote:

Average CPU is 70%

This isn't enough to tell you anything. On a multi-core box you can
completely max out one CPU and still see a low average.

If the workload inside the event handler isn't doing any I/O it is
likely to be stealing the CPU long enough to starve the thread that's
sending your heartbeats. Once two heartbeats are missed the broker will
close the connection from the other end. You'll be able to see this
happening if you check your rabbit logs.

I bet that is what's happening in this case. However, we should handle
this error better, since it is recoverable.

On Thursday, November 2, 2017 at 11:05:52 AM UTC, David Szotten wrote:

just to rule this out, are these long running tasks hogging the cpu by
any chance (blocking the event loop)?

On Thursday, 2 November 2017 10:57:49 UTC, Richard wrote:

Sorry the previous message was missing all of the trace:

RecoverableConnectionError: connection already closed
  File "bin/nameko_extras", line 11, in <module>
    sys.exit(main())
  File "nameko_extras/cli/main.py", line 24, in main
    run.main(args)
  File "nameko_extras/cli/run.py", line 48, in main
    run(services, config, **kwargs)
  File "nameko/cli/run.py", line 140, in run
    runnlet.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/runners.py", line 96, in wait
    SpawningProxy(self.containers, abort_on_error=True).wait()
  File "nameko/utils/__init__.py", line 184, in spawning_method
    return list(fail_fast_imap(pool, call, self._items))
  File "nameko/utils/__init__.py", line 153, in fail_fast_imap
    eventlet.getcurrent().throw(*exc_info)
  File "nameko/utils/__init__.py", line 132, in handle_result
    thread_result = finished_thread.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/utils/__init__.py", line 181, in call
    return getattr(item, name)(*args, **kwargs)
  File "nameko/containers.py", line 312, in wait
    return self._died.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "nameko/containers.py", line 466, in _handle_thread_exited
    gt.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/containers.py", line 399, in _run_worker
    worker_ctx, result, exc_info)
  File "nameko/messaging.py", line 461, in handle_result
    self.handle_message_processed(message, result, exc_info)
  File "nameko/messaging.py", line 469, in handle_message_processed
    self.queue_consumer.ack_message(message)
  File "nameko/utils/retry.py", line 46, in wrapper
    return wrapped(*args, **kwargs)
  File "nameko/messaging.py", line 315, in ack_message
    message.ack()
  File "kombu/message.py", line 88, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "amqp/channel.py", line 1584, in basic_ack
    self._send_method((60, 80), args)
  File "amqp/abstract_channel.py", line 50, in _send_method
    raise RecoverableConnectionError('connection already closed')

On Thursday, November 2, 2017 at 10:57:03 AM UTC, Richard wrote:

2.7.0 version

RecoverableConnectionError: connection already closed
  File "bin/nameko_extras", line 11, in <module>
    sys.exit(main())
  File "nameko_extras/cli/main.py", line 24, in main
    run.main(args)
  File "nameko_extras/cli/run.py", line 48, in main
    run(services, config, **kwargs)
  File "nameko/cli/run.py", line 140, in run
    runnlet.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/runners.py", line 96, in wait
    SpawningProxy(self.containers, abort_on_error=True).wait()
  File "nameko/utils/__init__.py", line 184, in spawning_method
    return list(fail_fast_imap(pool, call, self._items))
  File "nameko/utils/__init__.py", line 153, in fail_fast_imap
    eventlet.getcurrent().throw(*exc_info)
  File "nameko/utils/__init__.py", line 132, in handle_result
    thread_result = finished_thread.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/utils/__init__.py", line 181, in call
    return getattr(item, name)(*args, **kwargs)
  File "nameko/containers.py", line 312, in wait
    return self._died.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "nameko/containers.py", line 466, in _handle_thread_exited
    gt.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/containers.py", line 399, in _run_worker
    worker_ctx, result, exc_info)
  File "nameko/messaging.py", line 461, in handle_result
    self.handle_message_processed(message, result, exc_info)
  File "nameko/messaging.py", line 469, in handle_message_processed
    self.queue_consumer.ack_message(message)
  File "nameko/utils/retry.py", line 46, in wrapper
    return wrapped(*args, **kwargs)
  File "nameko/messaging.py", line 315, in ack_message
    message.ack()
  File "kombu/message.py", line 88, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "amqp/channel.py", line 1584, in basic_ack
    self._send_method((60, 80), args)
  File "amqp/abstract_channel.py", line 50, in _send_method
    raise <span style="color: #606;" class="styled-

11

Ok, i placed "debug.hub_blocking_detection(True)"
in the event handler method, ran the app, no error raised

···

On Thursday, November 2, 2017 at 11:31:59 AM UTC, Richard wrote:

Cannot even build CI for the project with this change

    def alarm_handler(signum, frame): import inspect> raise RuntimeError("Blocking detector ALARMED at" + str(inspect.getframeinfo(frame)))E RuntimeError: Blocking detector ALARMED atTraceback(filename='/usr/local/lib/python3.6/posixpath.py', lineno=82, function='join', code_context=[' if not p:\n'], index=0)

where should debug.hub_blocking_detection(True) be placed?

On Thursday, November 2, 2017 at 11:21:48 AM UTC, Richard wrote:

What will SIGALARM do exactly?

On Thursday, November 2, 2017 at 11:16:45 AM UTC, David Szotten wrote:

you can check this using
debug – Debugging tools for Eventlet — Eventlet 0.33.0 documentation

On Thursday, 2 November 2017 11:15:10 UTC, Matt Yule-Bennett wrote:

On Thursday, November 2, 2017 at 11:09:10 AM UTC, Richard wrote:

Average CPU is 70%

This isn't enough to tell you anything. On a multi-core box you can
completely max out one CPU and still see a low average.

If the workload inside the event handler isn't doing any I/O it is
likely to be stealing the CPU long enough to starve the thread that's
sending your heartbeats. Once two heartbeats are missed the broker will
close the connection from the other end. You'll be able to see this
happening if you check your rabbit logs.

I bet that is what's happening in this case. However, we should handle
this error better, since it is recoverable.

On Thursday, November 2, 2017 at 11:05:52 AM UTC, David Szotten wrote:

just to rule this out, are these long running tasks hogging the cpu
by any chance (blocking the event loop)?

On Thursday, 2 November 2017 10:57:49 UTC, Richard wrote:

Sorry the previous message was missing all of the trace:

RecoverableConnectionError: connection already closed
  File "bin/nameko_extras", line 11, in <module>
    sys.exit(main())
  File "nameko_extras/cli/main.py", line 24, in main
    run.main(args)
  File "nameko_extras/cli/run.py", line 48, in main
    run(services, config, **kwargs)
  File "nameko/cli/run.py", line 140, in run
    runnlet.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/runners.py", line 96, in wait
    SpawningProxy(self.containers, abort_on_error=True).wait()
  File "nameko/utils/__init__.py", line 184, in spawning_method
    return list(fail_fast_imap(pool, call, self._items))
  File "nameko/utils/__init__.py", line 153, in fail_fast_imap
    eventlet.getcurrent().throw(*exc_info)
  File "nameko/utils/__init__.py", line 132, in handle_result
    thread_result = finished_thread.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/utils/__init__.py", line 181, in call
    return getattr(item, name)(*args, **kwargs)
  File "nameko/containers.py", line 312, in wait
    return self._died.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "nameko/containers.py", line 466, in _handle_thread_exited
    gt.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/containers.py", line 399, in _run_worker
    worker_ctx, result, exc_info)
  File "nameko/messaging.py", line 461, in handle_result
    self.handle_message_processed(message, result, exc_info)
  File "nameko/messaging.py", line 469, in handle_message_processed
    self.queue_consumer.ack_message(message)
  File "nameko/utils/retry.py", line 46, in wrapper
    return wrapped(*args, **kwargs)
  File "nameko/messaging.py", line 315, in ack_message
    message.ack()
  File "kombu/message.py", line 88, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "amqp/channel.py", line 1584, in basic_ack
    self._send_method((60, 80), args)
  File "amqp/abstract_channel.py", line 50, in _send_method
    raise RecoverableConnectionError('connection already closed')

On Thursday, November 2, 2017 at 10:57:03 AM UTC, Richard wrote:

2.7.0 version

RecoverableConnectionError: connection already closed
  File "bin/nameko_extras", line 11, in <module>
    sys.exit(main())
  File "nameko_extras/cli/main.py", line 24, in main
    run.main(args)
  File "nameko_extras/cli/run.py", line 48, in main
    run(services, config, **kwargs)
  File "nameko/cli/run.py", line 140, in run
    runnlet.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/runners.py", line 96, in wait
    SpawningProxy(self.containers, abort_on_error=True).wait()
  File "nameko/utils/__init__.py", line 184, in spawning_method
    return list(fail_fast_imap(pool, call, self._items))
  File "nameko/utils/__init__.py", line 153, in fail_fast_imap
    eventlet.getcurrent().throw(*exc_info)
  File "nameko/utils/__init__.py", line 132, in handle_result
    thread_result = finished_thread.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/utils/__init__.py", line 181, in call
    return getattr(item, name)(*args, **kwargs)
  File "nameko/containers.py", line 312, in wait
    return self._died.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "nameko/containers.py", line 466, in _handle_thread_exited
    gt.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/containers.py", line 399, in _run_worker
    worker_ctx, result, exc_info)
  File "nameko/messaging.py", line 461, in handle_result
    self.handle_message_processed(message, result, exc_info)
  File "nameko/messaging.py", line 469, in handle_message_processed
    self.queue_consumer.ack_message(message)
  File "nameko/utils/retry.py", line 46, in wrapper
    return wrapped(*args, **kwargs)
  File "nameko/messaging.py", line 315, in ack_message
    message.ack()
  File "kombu/message.py", line 88, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "amqp/channel.py", line 1584, in basic_ack
    self._send_method((60, 80), args)
  File "amqp/abstract_channel.py", line 50, in _send_method
    raise <span style="color: #606;" class="styled-

12

Placing at the top of the service module also gives the same result when
running the app. no error raised

···

On Thursday, November 2, 2017 at 11:38:02 AM UTC, Richard wrote:

Ok, i placed "debug.hub_blocking_detection(True)"
in the event handler method, ran the app, no error raised

On Thursday, November 2, 2017 at 11:31:59 AM UTC, Richard wrote:

Cannot even build CI for the project with this change

    def alarm_handler(signum, frame): import inspect> raise RuntimeError("Blocking detector ALARMED at" + str(inspect.getframeinfo(frame)))E RuntimeError: Blocking detector ALARMED atTraceback(filename='/usr/local/lib/python3.6/posixpath.py', lineno=82, function='join', code_context=[' if not p:\n'], index=0)

where should debug.hub_blocking_detection(True) be placed?

On Thursday, November 2, 2017 at 11:21:48 AM UTC, Richard wrote:

What will SIGALARM do exactly?

On Thursday, November 2, 2017 at 11:16:45 AM UTC, David Szotten wrote:

you can check this using
debug – Debugging tools for Eventlet — Eventlet 0.33.0 documentation

On Thursday, 2 November 2017 11:15:10 UTC, Matt Yule-Bennett wrote:

On Thursday, November 2, 2017 at 11:09:10 AM UTC, Richard wrote:

Average CPU is 70%

This isn't enough to tell you anything. On a multi-core box you can
completely max out one CPU and still see a low average.

If the workload inside the event handler isn't doing any I/O it is
likely to be stealing the CPU long enough to starve the thread that's
sending your heartbeats. Once two heartbeats are missed the broker will
close the connection from the other end. You'll be able to see this
happening if you check your rabbit logs.

I bet that is what's happening in this case. However, we should handle
this error better, since it is recoverable.

On Thursday, November 2, 2017 at 11:05:52 AM UTC, David Szotten wrote:

just to rule this out, are these long running tasks hogging the cpu
by any chance (blocking the event loop)?

On Thursday, 2 November 2017 10:57:49 UTC, Richard wrote:

Sorry the previous message was missing all of the trace:

RecoverableConnectionError: connection already closed
  File "bin/nameko_extras", line 11, in <module>
    sys.exit(main())
  File "nameko_extras/cli/main.py", line 24, in main
    run.main(args)
  File "nameko_extras/cli/run.py", line 48, in main
    run(services, config, **kwargs)
  File "nameko/cli/run.py", line 140, in run
    runnlet.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/runners.py", line 96, in wait
    SpawningProxy(self.containers, abort_on_error=True).wait()
  File "nameko/utils/__init__.py", line 184, in spawning_method
    return list(fail_fast_imap(pool, call, self._items))
  File "nameko/utils/__init__.py", line 153, in fail_fast_imap
    eventlet.getcurrent().throw(*exc_info)
  File "nameko/utils/__init__.py", line 132, in handle_result
    thread_result = finished_thread.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/utils/__init__.py", line 181, in call
    return getattr(item, name)(*args, **kwargs)
  File "nameko/containers.py", line 312, in wait
    return self._died.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "nameko/containers.py", line 466, in _handle_thread_exited
    gt.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/containers.py", line 399, in _run_worker
    worker_ctx, result, exc_info)
  File "nameko/messaging.py", line 461, in handle_result
    self.handle_message_processed(message, result, exc_info)
  File "nameko/messaging.py", line 469, in handle_message_processed
    self.queue_consumer.ack_message(message)
  File "nameko/utils/retry.py", line 46, in wrapper
    return wrapped(*args, **kwargs)
  File "nameko/messaging.py", line 315, in ack_message
    message.ack()
  File "kombu/message.py", line 88, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "amqp/channel.py", line 1584, in basic_ack
    self._send_method((60, 80), args)
  File "amqp/abstract_channel.py", line 50, in _send_method
    raise RecoverableConnectionError('connection already closed')

On Thursday, November 2, 2017 at 10:57:03 AM UTC, Richard wrote:

2.7.0 version

RecoverableConnectionError: connection already closed
  File "bin/nameko_extras", line 11, in <module>
    sys.exit(main())
  File "nameko_extras/cli/main.py", line 24, in main
    run.main(args)
  File "nameko_extras/cli/run.py", line 48, in main
    run(services, config, **kwargs)
  File "nameko/cli/run.py", line 140, in run
    runnlet.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/runners.py", line 96, in wait
    SpawningProxy(self.containers, abort_on_error=True).wait()
  File "nameko/utils/__init__.py", line 184, in spawning_method
    return list(fail_fast_imap(pool, call, self._items))
  File "nameko/utils/__init__.py", line 153, in fail_fast_imap
    eventlet.getcurrent().throw(*exc_info)
  File "nameko/utils/__init__.py", line 132, in handle_result
    thread_result = finished_thread.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/utils/__init__.py", line 181, in call
    return getattr(item, name)(*args, **kwargs)
  File "nameko/containers.py", line 312, in wait
    return self._died.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "nameko/containers.py", line 466, in _handle_thread_exited
    gt.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/containers.py", line 399, in _run_worker
    worker_ctx, result, exc_info)
  File "nameko/messaging.py", line 461, in handle_result
    self.handle_message_processed(message, result, exc_info)
  File "nameko/messaging.py", line 469, in handle_message_processed
    self.queue_consumer.ack_message(message)
  File "nameko/utils/retry.py", line 46, in wrapper
    return wrapped(*args, **kwargs)
  File "nameko/messaging.py", line 315, in ack_message
    message.ack()
  File "kombu/message.py", line 88, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "amqp/channel.py", line 1584, in basic_ack
    self._send_method((60, 80), args)
  File "amqp/abstract_channel.py", line 50, in _send_method
    raise <span style="color: #606;" class="styled-

13

fixed this, was either lowering some cpu use or raising the missed
heartbeat threshold from 2 to 10 on the rabbitmq side. will find out which
change exactly tomorrow when we put the heartbeat threshold back to 2
missed.

···

On Thursday, November 2, 2017 at 11:39:23 AM UTC, Richard wrote:

Placing at the top of the service module also gives the same result when
running the app. no error raised

On Thursday, November 2, 2017 at 11:38:02 AM UTC, Richard wrote:

Ok, i placed "debug.hub_blocking_detection(True)"
in the event handler method, ran the app, no error raised

On Thursday, November 2, 2017 at 11:31:59 AM UTC, Richard wrote:

Cannot even build CI for the project with this change

    def alarm_handler(signum, frame): import inspect> raise RuntimeError("Blocking detector ALARMED at" + str(inspect.getframeinfo(frame)))E RuntimeError: Blocking detector ALARMED atTraceback(filename='/usr/local/lib/python3.6/posixpath.py', lineno=82, function='join', code_context=[' if not p:\n'], index=0)

where should debug.hub_blocking_detection(True) be placed?

On Thursday, November 2, 2017 at 11:21:48 AM UTC, Richard wrote:

What will SIGALARM do exactly?

On Thursday, November 2, 2017 at 11:16:45 AM UTC, David Szotten wrote:

you can check this using
debug – Debugging tools for Eventlet — Eventlet 0.33.0 documentation

On Thursday, 2 November 2017 11:15:10 UTC, Matt Yule-Bennett wrote:

On Thursday, November 2, 2017 at 11:09:10 AM UTC, Richard wrote:

Average CPU is 70%

This isn't enough to tell you anything. On a multi-core box you can
completely max out one CPU and still see a low average.

If the workload inside the event handler isn't doing any I/O it is
likely to be stealing the CPU long enough to starve the thread that's
sending your heartbeats. Once two heartbeats are missed the broker will
close the connection from the other end. You'll be able to see this
happening if you check your rabbit logs.

I bet that is what's happening in this case. However, we should
handle this error better, since it is recoverable.

On Thursday, November 2, 2017 at 11:05:52 AM UTC, David Szotten >>>>>>> wrote:

just to rule this out, are these long running tasks hogging the cpu
by any chance (blocking the event loop)?

On Thursday, 2 November 2017 10:57:49 UTC, Richard wrote:

Sorry the previous message was missing all of the trace:

RecoverableConnectionError: connection already closed
  File "bin/nameko_extras", line 11, in <module>
    sys.exit(main())
  File "nameko_extras/cli/main.py", line 24, in main
    run.main(args)
  File "nameko_extras/cli/run.py", line 48, in main
    run(services, config, **kwargs)
  File "nameko/cli/run.py", line 140, in run
    runnlet.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/runners.py", line 96, in wait
    SpawningProxy(self.containers, abort_on_error=True).wait()
  File "nameko/utils/__init__.py", line 184, in spawning_method
    return list(fail_fast_imap(pool, call, self._items))
  File "nameko/utils/__init__.py", line 153, in fail_fast_imap
    eventlet.getcurrent().throw(*exc_info)
  File "nameko/utils/__init__.py", line 132, in handle_result
    thread_result = finished_thread.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/utils/__init__.py", line 181, in call
    return getattr(item, name)(*args, **kwargs)
  File "nameko/containers.py", line 312, in wait
    return self._died.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "nameko/containers.py", line 466, in _handle_thread_exited
    gt.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/containers.py", line 399, in _run_worker
    worker_ctx, result, exc_info)
  File "nameko/messaging.py", line 461, in handle_result
    self.handle_message_processed(message, result, exc_info)
  File "nameko/messaging.py", line 469, in handle_message_processed
    self.queue_consumer.ack_message(message)
  File "nameko/utils/retry.py", line 46, in wrapper
    return wrapped(*args, **kwargs)
  File "nameko/messaging.py", line 315, in ack_message
    message.ack()
  File "kombu/message.py", line 88, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "amqp/channel.py", line 1584, in basic_ack
    self._send_method((60, 80), args)
  File "amqp/abstract_channel.py", line 50, in _send_method
    raise RecoverableConnectionError('connection already closed')

On Thursday, November 2, 2017 at 10:57:03 AM UTC, Richard wrote:

2.7.0 version

RecoverableConnectionError: connection already closed
  File "bin/nameko_extras", line 11, in <module>
    sys.exit(main())
  File "nameko_extras/cli/main.py", line 24, in main
    run.main(args)
  File "nameko_extras/cli/run.py", line 48, in main
    run(services, config, **kwargs)
  File "nameko/cli/run.py", line 140, in run
    runnlet.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/runners.py", line 96, in wait
    SpawningProxy(self.containers, abort_on_error=True).wait()
  File "nameko/utils/__init__.py", line 184, in spawning_method
    return list(fail_fast_imap(pool, call, self._items))
  File "nameko/utils/__init__.py", line 153, in fail_fast_imap
    eventlet.getcurrent().throw(*exc_info)
  File "nameko/utils/__init__.py", line 132, in handle_result
    thread_result = finished_thread.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/utils/__init__.py", line 181, in call
    return getattr(item, name)(*args, **kwargs)
  File "nameko/containers.py", line 312, in wait
    return self._died.wait()
  File "eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "eventlet/hubs/hub.py", line 295, in switch
    return self.greenlet.switch()
  File "nameko/containers.py", line 466, in _handle_thread_exited
    gt.wait()
  File "eventlet/greenthread.py", line 179, in wait
    return self._exit_event.wait()
  File "eventlet/event.py", line 125, in wait
    current.throw(*self._exc)
  File "eventlet/greenthread.py", line 218, in main
    result = function(*args, **kwargs)
  File "nameko/containers.py", line 399, in _run_worker
    worker_ctx, result, exc_info)
  File "nameko/messaging.py", line 461, in handle_result
    self.handle_message_processed(message, result, exc_info)
  File "nameko/messaging.py", line 469, in handle_message_processed
    self.queue_consumer.ack_message(message)
  File "nameko/utils/retry.py", line 46, in wrapper
    return wrapped(*args, **kwargs)
  File "nameko/messaging.py", line 315, in ack_message
    message.ack()
  File "kombu/message.py", line 88, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "amqp/channel.py", line 1584, in basic_ack
    self._send_method((60, 80), args)
  File "amqp/abstract_channel.py", line 50, in _send_method
    raise <span style="color: #606;" class="styled-