Heartbeat with nameko

1

What is the better way to do heartbeat with nameko for HA?
I have a service which tries to run heartbeat to another service. Services exist on different nodes.
If the remote service doesn’t exist from start, nameko throws exception which can be handled. But if the service existed and goes away, the queue keeps on existing and the heartbeat messages are queued up. I don’t want the message queue up as I dont know when the peer will be alive.
I am not able to make the queue non-durable with nameko. Also AMQP heartbeat cannot be directly used through nameko/khombu am I right here?
Is there any way to for the messages not to queue up?

Currently using rpc for heartbeat

2

Not completely sure what you’re trying to achieve here.

AMQP has its own heartbeat which Nameko uses on consumer connections (not publishers I’m afraid). It is used to detect connectivity problems and results in an exception being raised if the connection fails – Nameko catches it and tries to reestablish the connection. Note that this is not a peer-to-peer (i.e. client to server) check, because AMQP connections are always between a peer and a broker.

Nameko services shouldn’t really care about whether they have peers that are “up”, at least if they’re doing normal RPC. When you have multiple instances of service running the same RPC entrypoints (and connected to the same RabbitMQ broker), they all pool together. When you make a request one of the instances will answer, and you don’t control which.

When you stop an instance, the RPC request queue remains on the broker so that clients can continue to make calls, decoupled from the state of the backend. This allows you to do rolling updates and tolerate short service outages like a service restart.

It would be nice if the RPC entrypoint allowed you to specify a queue expiry, so that you can choose whether or not to enable this behaviour. Unfortunately this is not possible at present. There is an item to address this on the roadmap here.

3

Thanks a lot for your response Matt!
My use case is HA system. The redundant service/card may come online at a later point in time.(User has a choice, user may/ may not install redundant card)
In that case if I keep on queuing the rpc messages, it takes up resources.
I want to run heartbeat to know if peer is available. With current mechanism the heartbeat messages are getting queued up as well. I wanted this particular queue to be non durable.

4

Sorry for the slow response. I’m still not completely sure what you want to achieve. There are two ways I could interpret it:

  1. The service/card that the user may choose to install runs the same service as the non-redundant one (i.e. the same Nameko service class). In this case you don’t need to do anything – the redundant card will connect to the broker and consume messages if it is installed. There are no extra queues, only an extra consumer. This is what I would describe as an “HA” setup.

  2. The service/card that the user may choose to install runs a different service. In this case you need to know at the application level whether or not to route messages to it. In this case, adding an application-level heartbeat would make sense. Unfortunately (as you’ve identified) the queue needs to be non-durable/expiring in this case, and Nameko does not currently support this with the RPC extensions.

If you wanted to build this two-way heartbeat and use expiring queues, you’d have to construct your own extensions on top of the Publisher and Consumer (in nameko.messaging) which do support passing arguments (like expiry) when creating the queues.