Nameko tests failing with Connection to broker lost

I cloned the nameko repo recently and I am trying run the tests locally. I am however getting the following error all the time.

(env) travelling_devsman@Josephs-MacBook-Pro nameko % pytest test -x -v  --timeout 30 --timeout_method=thread
================================================================================================= test session starts ==================================================================================================
platform darwin -- Python 3.8.5, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /Users/travelling_devsman/Desktop/programming/opensource/nameko/env/bin/python
cachedir: .pytest_cache
rootdir: /Users/travelling_devsman/Desktop/programming/opensource/nameko, configfile: setup.cfg
plugins: nameko-sqlalchemy-1.5.0, timeout-1.4.2, nameko-2.12.0
timeout: 30.0s
timeout method: thread
timeout func_only: False
collected 655 items                                                                                                                                                                                                    

test/test_broker.py::test_idle_disconnect PASSED                                                                                                                                                                 [  0%]
test/test_broker.py::test_proxy_disconnect_with_active_worker FAILED                                                                                                                                             [  0%]

======================================================================================================= FAILURES =======================================================================================================
_______________________________________________________________________________________ test_proxy_disconnect_with_active_worker _______________________________________________________________________________________

container_factory = <function container_factory.<locals>.make_container at 0x10da8d160>, rabbit_manager = <nameko.testing.rabbit.Client object at 0x10da490d0>
rabbit_config = {'AMQP_URI': 'pyamqp://guest:guest@localhost:5672/nameko_test_ogrjfmurka', 'username': 'guest', 'vhost': 'nameko_test_ogrjfmurka'}

    def test_proxy_disconnect_with_active_worker(
            container_factory, rabbit_manager, rabbit_config):
        """ Break the connection to rabbit while a service's queue consumer and
        rabbit while the service has an in-flight rpc request (i.e. it is waiting
        on a reply).
        """
        # ExampleService is the target; ProxyService has the rpc_proxy;
        proxy_container = container_factory(ProxyService, rabbit_config)
        example_container = container_factory(ExampleService, rabbit_config)
    
        proxy_container.start()
    
        # get proxyservice's queue consumer connection while we know it's the
        # only active connection
        vhost = rabbit_config['vhost']
        connections = get_rabbit_connections(vhost, rabbit_manager)
>       assert len(connections) == 1
E       assert 0 == 1
E         +0
E         -1

test/test_broker.py:116: AssertionError
------------------------------------------------------------------------------------------------ Captured log teardown -------------------------------------------------------------------------------------------------
WARNING  kombu.mixins:mixins.py:180 Connection to broker lost, trying to re-establish connection...
Traceback (most recent call last):
  File "/Users/travelling_devsman/Desktop/programming/opensource/nameko/env/lib/python3.8/site-packages/kombu/mixins.py", line 175, in run
    for _ in self.consume(limit=None, **kwargs):
  File "/Users/travelling_devsman/Desktop/programming/opensource/nameko/env/lib/python3.8/site-packages/kombu/mixins.py", line 197, in consume
    conn.drain_events(timeout=safety_interval)
  File "/Users/travelling_devsman/Desktop/programming/opensource/nameko/env/lib/python3.8/site-packages/kombu/connection.py", line 324, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/Users/travelling_devsman/Desktop/programming/opensource/nameko/env/lib/python3.8/site-packages/kombu/transport/pyamqp.py", line 103, in drain_events
    return connection.drain_events(**kwargs)
  File "/Users/travelling_devsman/Desktop/programming/opensource/nameko/env/lib/python3.8/site-packages/amqp/connection.py", line 508, in drain_events
    while not self.blocking_read(timeout):
  File "/Users/travelling_devsman/Desktop/programming/opensource/nameko/env/lib/python3.8/site-packages/amqp/connection.py", line 514, in blocking_read
    return self.on_inbound_frame(frame)
  File "/Users/travelling_devsman/Desktop/programming/opensource/nameko/env/lib/python3.8/site-packages/amqp/method_framing.py", line 55, in on_frame
    callback(channel, method_sig, buf, None)
  File "/Users/travelling_devsman/Desktop/programming/opensource/nameko/env/lib/python3.8/site-packages/amqp/connection.py", line 520, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
  File "/Users/travelling_devsman/Desktop/programming/opensource/nameko/env/lib/python3.8/site-packages/amqp/abstract_channel.py", line 145, in dispatch_method
    listener(*args)
  File "/Users/travelling_devsman/Desktop/programming/opensource/nameko/env/lib/python3.8/site-packages/amqp/channel.py", line 1453, in _on_basic_cancel
    raise ConsumerCancelled(consumer_tag, spec.Basic.Cancel)
amqp.exceptions.ConsumerCancelled: Basic.cancel: (0) None4
=============================================================================================== short test summary info ================================================================================================
FAILED test/test_broker.py::test_proxy_disconnect_with_active_worker - assert 0 == 1

Am I missing anything?

Edit: I have rabbitmq running locally.

I think that test is a bit unreliable. I recall seeing it fail occasionally in CI too – and a retry fixes it.

Is this the only test that fails for you locally?

Hi @mattbennett,

Thanks for your quick response.

I just ran it without the -x flag and looks like there are a number of them failing (I guess most of them are related to the above issue).

There is one of them that times out though.

test/test_broker.py::test_idle_disconnect PASSED                                                                                                                                                                 [  0%]
test/test_broker.py::test_proxy_disconnect_with_active_worker FAILED                                                                                                                                             [  0%]
test/test_broker.py::test_service_disconnect_with_active_async_worker FAILED                                                                                                                                     [  0%]
test/test_broker.py::test_service_disconnect_with_active_rpc_worker FAILED                                                                                                                                       [  0%]
test/test_broker.py::test_service_disconnect_with_active_rpc_worker_via_service_proxy FAILED                                                                                                                     [  0%]
test/test_call_id_stack.py::test_worker_context_gets_stack PASSED                                                                                                                                                [  0%]
test/test_call_id_stack.py::test_short_call_stack PASSED                                                                                                                                                         [  1%]
test/test_call_id_stack.py::test_call_id_stack PASSED                                                                                                                                                            [  1%]
test/test_call_id_stack.py::test_call_id_over_events PASSED                                                                                                                                                      [  1%]
test/test_call_id_stack.py::TestImmediateParentCallId::test_with_parent PASSED                                                                                                                                   [  1%]
test/test_call_id_stack.py::TestImmediateParentCallId::test_without_parent PASSED                                                                                                                                [  1%]
test/test_call_id_stack.py::TestOriginCallId::test_with_origin PASSED                                                                                                                                            [  1%]
test/test_call_id_stack.py::TestOriginCallId::test_without_origin PASSED                                                                                                                                         [  1%]
test/test_container.py::test_collects_extensions PASSED                                                                                                                                                          [  2%]
test/test_container.py::test_starts_extensions PASSED                                                                                                                                                            [  2%]
test/test_container.py::test_stops_extensions PASSED                                                                                                                                                             [  2%]
test/test_container.py::test_stops_entrypoints_before_dependency_providers PASSED                                                                                                                                [  2%]
test/test_container.py::test_worker_life_cycle PASSED                                                                                                                                                            [  2%]
test/test_container.py::test_wait_waits_for_container_stopped PASSED                                                                                                                                             [  2%]
test/test_container.py::test_container_doesnt_exhaust_max_workers PASSED                                                                                                                                         [  3%]
test/test_container.py::test_stop_already_stopped PASSED                                                                                                                                                         [  3%]
test/test_container.py::test_kill_already_stopped PASSED                                                                                                                                                         [  3%]
test/test_container.py::test_kill_container_with_managed_threads PASSED                                                                                                                                          [  3%]
test/test_container.py::test_kill_container_with_active_workers PASSED                                                                                                                                           [  3%]
test/test_container.py::test_handle_killed_worker PASSED                                                                                                                                                         [  3%]
test/test_container.py::test_spawned_thread_kills_container PASSED                                                                                                                                               [  3%]
test/test_container.py::test_spawned_thread_causes_container_to_kill_other_thread PASSED                                                                                                                         [  4%]
test/test_container.py::test_container_only_killed_once PASSED                                                                                                                                                   [  4%]
test/test_container.py::test_container_stop_kills_remaining_managed_threads PASSED                                                                                                                               [  4%]
test/test_container.py::test_container_kill_kills_remaining_managed_threads PASSED                                                                                                                               [  4%]
test/test_container.py::test_kill_bad_dependency PASSED                                                                                                                                                          [  4%]
test/test_container.py::test_stop_during_kill PASSED                                                                                                                                                             [  4%]
test/test_container.py::test_get_service_name PASSED                                                                                                                                                             [  5%]
test/test_container.py::test_logging_managed_threads PASSED                                                                                                                                                      [  5%]
test/test_contextdata.py::test_get_custom_context_value PASSED                                                                                                                                                   [  5%]
test/test_contextdata.py::test_get_unset_value PASSED                                                                                                                                                            [  5%]
test/test_contextdata.py::test_get_builtin_dependencies[auth_token-auth_token] PASSED                                                                                                                            [  5%]
test/test_contextdata.py::test_get_builtin_dependencies[language-language] PASSED                                                                                                                                [  5%]
test/test_contextdata.py::test_get_builtin_dependencies[user_id-user_id] PASSED                                                                                                                                  [  5%]
test/test_contextdata.py::test_get_builtin_dependencies[user_agent-user_agent] PASSED                                                                                                                            [  6%]
test/test_dependency_providers.py::TestConfig::test_get_config_value PASSED                                                                                                                                      [  6%]
test/test_entrypoints.py::TestDecorator::test_decorator_without_args PASSED                                                                                                                                      [  6%]
test/test_entrypoints.py::TestDecorator::test_decorator_with_args PASSED                                                                                                                                         [  6%]
test/test_entrypoints.py::TestDecorator::test_decorator_with_kwargs PASSED                                                                                                                                       [  6%]
test/test_entrypoints.py::TestExpectedExceptions::test_expected_exceptions PASSED                                                                                                                                [  6%]
test/test_entrypoints.py::TestSensitiveArguments::test_sensitive_arguments PASSED                                                                                                                                [  7%]
test/test_entrypoints.py::TestSensitiveArguments::test_sensitive_variables_backwards_compat PASSED                                                                                                               [  7%]
test/test_errors.py::test_error_in_worker PASSED                                                                                                                                                                 [  7%]
test/test_errors.py::test_expected_error_in_worker PASSED                                                                                                                                                        [  7%]
test/test_errors.py::test_error_in_remote_worker PASSED                                                                                                                                                          [  7%]
test/test_errors.py::test_handle_result_error PASSED                                                                                                                                                             [  7%]
test/test_errors.py::test_dependency_call_lifecycle_errors[worker_setup] PASSED                                                                                                                                  [  7%]
test/test_errors.py::test_dependency_call_lifecycle_errors[worker_result] PASSED                                                                                                                                 [  8%]
test/test_errors.py::test_dependency_call_lifecycle_errors[worker_teardown] PASSED                                                                                                                               [  8%]
test/test_errors.py::test_runner_catches_container_errors PASSED                                                                                                                                                 [  8%]
test/test_errors.py::test_graceful_stop_on_one_container_error PASSED                                                                                                                                            [  8%]
test/test_events.py::test_event_dispatcher PASSED                                                                                                                                                                [  8%]
test/test_events.py::test_event_handler PASSED                                                                                                                                                                   [  8%]
test/test_events.py::TestReliableDeliveryEventHandlerConfigurationError::test_raises_with_default_broadcast_identity PASSED                                                                                      [  9%]
test/test_events.py::TestReliableDeliveryEventHandlerConfigurationError::test_no_raise_with_custom_identity PASSED                                                                                               [  9%]
test/test_events.py::test_service_pooled_events FAILED                                                                                                                                                           [  9%]
test/test_events.py::test_service_pooled_events_multiple_handlers FAILED                                                                                                                                         [  9%]
test/test_events.py::test_singleton_events FAILED                                                                                                                                                                [  9%]
test/test_events.py::test_broadcast_events FAILED                                                                                                                                                                [  9%]
test/test_events.py::test_requeue_on_error FAILED                                                                                                                                                                [  9%]
test/test_events.py::test_reliable_delivery FAILED                                                                                                                                                               [ 10%]
test/test_events.py::test_unreliable_delivery FAILED                                                                                                                                                             [ 10%]
test/test_events.py::test_custom_event_handler PASSED                                                                                                                                                            [ 10%]
test/test_events.py::test_dispatch_to_rabbit FAILED                                                                                                                                                              [ 10%]
test/test_events.py::TestBackwardsCompatClassAttrs::test_attrs_are_applied_as_defaults[retry-False] PASSED                                                                                                       [ 10%]
test/test_events.py::TestBackwardsCompatClassAttrs::test_attrs_are_applied_as_defaults[retry_policy-value1] PASSED                                                                                               [ 10%]
test/test_events.py::TestBackwardsCompatClassAttrs::test_attrs_are_applied_as_defaults[use_confirms-False] PASSED                                                                                                [ 10%]
test/test_events.py::TestConfigurability::test_regular_parameters[delivery_mode] PASSED                                                                                                                          [ 11%]
test/test_events.py::TestConfigurability::test_regular_parameters[mandatory] PASSED                                                                                                                              [ 11%]
test/test_events.py::TestConfigurability::test_regular_parameters[priority] PASSED                                                                                                                               [ 11%]
test/test_events.py::TestConfigurability::test_regular_parameters[expiration] PASSED                                                                                                                             [ 11%]
test/test_events.py::TestConfigurability::test_regular_parameters[serializer] PASSED                                                                                                                             [ 11%]
test/test_events.py::TestConfigurability::test_regular_parameters[compression] PASSED                                                                                                                            [ 11%]
test/test_events.py::TestConfigurability::test_regular_parameters[retry] PASSED                                                                                                                                  [ 12%]
test/test_events.py::TestConfigurability::test_regular_parameters[retry_policy] PASSED                                                                                                                           [ 12%]
test/test_events.py::TestConfigurability::test_regular_parameters[correlation_id] PASSED                                                                                                                         [ 12%]
test/test_events.py::TestConfigurability::test_regular_parameters[user_id] PASSED                                                                                                                                [ 12%]
test/test_events.py::TestConfigurability::test_regular_parameters[bogus_param] PASSED                                                                                                                            [ 12%]
test/test_events.py::TestConfigurability::test_headers PASSED                                                                                                                                                    [ 12%]
test/test_events.py::TestConfigurability::test_restricted_parameters PASSED                                                                                                                                      [ 12%]
test/test_events.py::TestSSL::test_event_handler_over_ssl[True] 
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Stack of MainThread (4699737536) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  File "/Users/travelling_devsman/Desktop/programming/opensource/nameko/env/lib/python3.8/site-packages/eventlet/green/thread.py", line 42, in __thread_body
    func(*args, **kwargs)
  File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/Users/travelling_devsman/Desktop/programming/opensource/nameko/env/lib/python3.8/site-packages/eventlet/green/thread.py", line 63, in wrap_bootstrap_inner
    bootstrap_inner()
  File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 1254, in run
    self.function(*self.args, **self.kwargs)
  File "/Users/travelling_devsman/Desktop/programming/opensource/nameko/env/lib/python3.8/site-packages/pytest_timeout.py", line 410, in timeout_timer
    dump_stacks()
  File "/Users/travelling_devsman/Desktop/programming/opensource/nameko/env/lib/python3.8/site-packages/pytest_timeout.py", line 433, in dump_stacks
    write("".join(traceback.format_stack(frame)))

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Sorry for the late response @josewails.

I suspect this is a combination of the versions of RabbitMQ and Kombu that you’re using. The Nameko testsuite runs against an old version of RabbitMQ (3.6.6). It should probably be updated to be something more recent.

Just to confirm, these particular tests will fail with RabbitMQ 3.6.7 and above.

Nameko in general works fine with modern RabbitMQ versions, but these tests are exploiting a particular way that the RabbitMQ management interface used to work. There is a utility in the v3 RC branch which can be used as an alternative, and these tests just need a small refactor to use it.