# RPC-extension and heartbeat if my service doing long task (above 20min)

**URL:** <https://discourse.nameko.io/t/rpc-extension-and-heartbeat-if-my-service-doing-long-task-above-20min/248>\
**Category:** googlegroup\
**Created:** [April 2, 2018, 9:31am UTC](https://discourse.nameko.io/t/rpc-extension-and-heartbeat-if-my-service-doing-long-task-above-20min/248 "2018-04-02T09:31:16Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tsoy](https://avatars.discourse-cdn.com/v4/letter/t/839c29/32.png) [@Tsoy](https://discourse.nameko.io/u/Tsoy)\
**Post date:** [April 2, 2018, 9:31am UTC](https://discourse.nameko.io/t/rpc-extension-and-heartbeat-if-my-service-doing-long-task-above-20min/248/1 "2018-04-02T09:31:16Z")

</div>

The error lies in the fact that the received task from the queue returns to  
the queue back some time later, if you look at the logs there is an error  
associated with the timeouts.  
Here are the detailed logs and question, how do I increase the count of  
heartbeats?

> > TRACE

2018-03-07 15:14:07,102 - WARNING - Connection to broker lost, trying to re-establish  
connection...  
Traceback (most recent call last):  
File "/home/etl/.local/lib/python3.5/site-packages/kombu/mixins.py", line  
199, in consume  
conn.drain\_events(timeout=safety\_interval)  
File "/home/etl/.local/lib/python3.5/site-packages/kombu/connection.py",  
line 288, in drain\_events  
return self.transport.drain\_events(self.connection, \*\*kwargs)  
File  
"/home/etl/.local/lib/python3.5/site-packages/kombu/transport/pyamqp.py",  
line 95, in drain\_events  
return connection.drain\_events(\*\*kwargs)  
File "/home/etl/.local/lib/python3.5/site-packages/amqp/connection.py",  
line 303, in drain\_events  
chanmap, None, timeout=timeout,  
File "/home/etl/.local/lib/python3.5/site-packages/amqp/connection.py",  
line 366, in \_wait\_multiple  
channel, method\_sig, args, content = read\_timeout(timeout)  
File "/home/etl/.local/lib/python3.5/site-packages/amqp/connection.py",  
line 337, in read\_timeout  
return self.method\_reader.read\_method()  
File "/home/etl/.local/lib/python3.5/site-packages/amqp/method\_framing.py",  
line 189, in read\_method  
raise m  
File "/home/etl/.local/lib/python3.5/site-packages/amqp/method\_framing.py",  
line 107, in \_next\_method  
frame\_type, channel, payload = read\_frame()  
File "/home/etl/.local/lib/python3.5/site-packages/amqp/transport.py",  
line 154, in read\_frame  
frame\_header = read(7, True)  
File "/home/etl/.local/lib/python3.5/site-packages/amqp/transport.py",  
line 277, in \_read  
s = recv(n - len(rbuf))  
File  
"/home/etl/.local/lib/python3.5/site-packages/eventlet/greenio/base.py",  
line 360, in recv  
return self.\_recv\_loop(self.fd.recv, b'', bufsize, flags)  
File  
"/home/etl/.local/lib/python3.5/site-packages/eventlet/greenio/base.py",  
line 354, in \_recv\_loop  
self.\_read\_trampoline()  
File  
"/home/etl/.local/lib/python3.5/site-packages/eventlet/greenio/base.py",  
line 325, in \_read\_trampoline  
timeout\_exc=socket\_timeout('timed out'))  
File  
"/home/etl/.local/lib/python3.5/site-packages/eventlet/greenio/base.py",  
line 207, in \_trampoline  
mark\_as\_closed=self.\_mark\_as\_closed)  
File  
"/home/etl/.local/lib/python3.5/site-packages/eventlet/hubs/\_\_init\_\_.py",  
line 163, in trampoline  
return hub.switch()  
File "/home/etl/.local/lib/python3.5/site-packages/eventlet/hubs/hub.py",  
line 295, in switch  
return self.greenlet.switch()  
socket.timeout: timed out

Thanks all!

---

<div class="post-metadata">

**Author:** ![Jukowitz](https://avatars.discourse-cdn.com/v4/letter/j/ba8739/32.png) [@Jukowitz](https://discourse.nameko.io/u/Jukowitz)\
**Post date:** [April 2, 2018, 4:24pm UTC](https://discourse.nameko.io/t/rpc-extension-and-heartbeat-if-my-service-doing-long-task-above-20min/248/2 "2018-04-02T16:24:55Z")

</div>

I had a similar issue.

The default is to send a heartbeat every 60 seconds, with 3 retries.

You can increase the heartbeat timeout with the `HEARTBEAT` config value.

If your 'long task' happens to be a SubProcess call (like mine was) you  
should make sure you import the 'greened' Subprocess module (Subprocess is  
not monkey-patched automatically, you have to specifically import it).

Geoff

> **···**
>
> On Monday, April 2, 2018 at 2:31:16 AM UTC-7, ts...@atlas.ru wrote:
> 
> > The error lies in the fact that the received task from the queue returns  
> > to the queue back some time later, if you look at the logs there is an  
> > error associated with the timeouts.  
> > Here are the detailed logs and question, how do I increase the count of  
> > heartbeats?
> > 
> > \>\> TRACE
> > 
> > 2018-03-07 15:14:07,102 - WARNING - Connection to broker lost, trying to  
> > re-establish connection...  
> > Traceback (most recent call last):  
> > File "/home/etl/.local/lib/python3.5/site-packages/kombu/mixins.py",  
> > line 199, in consume  
> > conn.drain\_events(timeout=safety\_interval)  
> > File "/home/etl/.local/lib/python3.5/site-packages/kombu/connection.py",  
> > line 288, in drain\_events  
> > return self.transport.drain\_events(self.connection, \*\*kwargs)  
> > File  
> > "/home/etl/.local/lib/python3.5/site-packages/kombu/transport/pyamqp.py",  
> > line 95, in drain\_events  
> > return connection.drain\_events(\*\*kwargs)  
> > File "/home/etl/.local/lib/python3.5/site-packages/amqp/connection.py",  
> > line 303, in drain\_events  
> > chanmap, None, timeout=timeout,  
> > File "/home/etl/.local/lib/python3.5/site-packages/amqp/connection.py",  
> > line 366, in \_wait\_multiple  
> > channel, method\_sig, args, content = read\_timeout(timeout)  
> > File "/home/etl/.local/lib/python3.5/site-packages/amqp/connection.py",  
> > line 337, in read\_timeout  
> > return self.method\_reader.read\_method()  
> > File  
> > "/home/etl/.local/lib/python3.5/site-packages/amqp/method\_framing.py",  
> > line 189, in read\_method  
> > raise m  
> > File  
> > "/home/etl/.local/lib/python3.5/site-packages/amqp/method\_framing.py",  
> > line 107, in \_next\_method  
> > frame\_type, channel, payload = read\_frame()  
> > File "/home/etl/.local/lib/python3.5/site-packages/amqp/transport.py",  
> > line 154, in read\_frame  
> > frame\_header = read(7, True)  
> > File "/home/etl/.local/lib/python3.5/site-packages/amqp/transport.py",  
> > line 277, in \_read  
> > s = recv(n - len(rbuf))  
> > File  
> > "/home/etl/.local/lib/python3.5/site-packages/eventlet/greenio/base.py",  
> > line 360, in recv  
> > return self.\_recv\_loop(self.fd.recv, b'', bufsize, flags)  
> > File  
> > "/home/etl/.local/lib/python3.5/site-packages/eventlet/greenio/base.py",  
> > line 354, in \_recv\_loop  
> > self.\_read\_trampoline()  
> > File  
> > "/home/etl/.local/lib/python3.5/site-packages/eventlet/greenio/base.py",  
> > line 325, in \_read\_trampoline  
> > timeout\_exc=socket\_timeout('timed out'))  
> > File  
> > "/home/etl/.local/lib/python3.5/site-packages/eventlet/greenio/base.py",  
> > line 207, in \_trampoline  
> > mark\_as\_closed=self.\_mark\_as\_closed)  
> > File  
> > "/home/etl/.local/lib/python3.5/site-packages/eventlet/hubs/\_\_init\_\_.py",  
> > line 163, in trampoline  
> > return hub.switch()  
> > File "/home/etl/.local/lib/python3.5/site-packages/eventlet/hubs/hub.py",  
> > line 295, in switch  
> > return self.greenlet.switch()  
> > socket.timeout: timed out
> > 
> > Thanks all!

---

<div class="post-metadata">

**Author:** ![Alexey\_Zf](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@Alexey\_Zf](https://discourse.nameko.io/u/Alexey_Zf)\
**Post date:** [April 12, 2018, 9:41am UTC](https://discourse.nameko.io/t/rpc-extension-and-heartbeat-if-my-service-doing-long-task-above-20min/248/3 "2018-04-12T09:41:06Z")

</div>

Yeap! Thanks for advice! I fixed it with eventlet.

> **···**
>
> On Monday, April 2, 2018 at 7:24:55 PM UTC+3, juko...@gmail.com wrote:
> 
> > I had a similar issue.
> > 
> > The default is to send a heartbeat every 60 seconds, with 3 retries.
> > 
> > You can increase the heartbeat timeout with the `HEARTBEAT` config value.
> > 
> > If your 'long task' happens to be a SubProcess call (like mine was) you  
> > should make sure you import the 'greened' Subprocess module (Subprocess is  
> > not monkey-patched automatically, you have to specifically import it).
> > 
> > Geoff
> > 
> > On Monday, April 2, 2018 at 2:31:16 AM UTC-7, ts...@atlas.ru wrote:
> > 
> > > The error lies in the fact that the received task from the queue returns  
> > > to the queue back some time later, if you look at the logs there is an  
> > > error associated with the timeouts.  
> > > Here are the detailed logs and question, how do I increase the count of  
> > > heartbeats?
> > > 
> > > \>\> TRACE
> > > 
> > > 2018-03-07 15:14:07,102 - WARNING - Connection to broker lost, trying to  
> > > re-establish connection...  
> > > Traceback (most recent call last):  
> > > File "/home/etl/.local/lib/python3.5/site-packages/kombu/mixins.py",  
> > > line 199, in consume  
> > > conn.drain\_events(timeout=safety\_interval)  
> > > File "/home/etl/.local/lib/python3.5/site-packages/kombu/connection.py",  
> > > line 288, in drain\_events  
> > > return self.transport.drain\_events(self.connection, \*\*kwargs)  
> > > File  
> > > "/home/etl/.local/lib/python3.5/site-packages/kombu/transport/pyamqp.py",  
> > > line 95, in drain\_events  
> > > return connection.drain\_events(\*\*kwargs)  
> > > File "/home/etl/.local/lib/python3.5/site-packages/amqp/connection.py",  
> > > line 303, in drain\_events  
> > > chanmap, None, timeout=timeout,  
> > > File "/home/etl/.local/lib/python3.5/site-packages/amqp/connection.py",  
> > > line 366, in \_wait\_multiple  
> > > channel, method\_sig, args, content = read\_timeout(timeout)  
> > > File "/home/etl/.local/lib/python3.5/site-packages/amqp/connection.py",  
> > > line 337, in read\_timeout  
> > > return self.method\_reader.read\_method()  
> > > File  
> > > "/home/etl/.local/lib/python3.5/site-packages/amqp/method\_framing.py",  
> > > line 189, in read\_method  
> > > raise m  
> > > File  
> > > "/home/etl/.local/lib/python3.5/site-packages/amqp/method\_framing.py",  
> > > line 107, in \_next\_method  
> > > frame\_type, channel, payload = read\_frame()  
> > > File "/home/etl/.local/lib/python3.5/site-packages/amqp/transport.py",  
> > > line 154, in read\_frame  
> > > frame\_header = read(7, True)  
> > > File "/home/etl/.local/lib/python3.5/site-packages/amqp/transport.py",  
> > > line 277, in \_read  
> > > s = recv(n - len(rbuf))  
> > > File  
> > > "/home/etl/.local/lib/python3.5/site-packages/eventlet/greenio/base.py",  
> > > line 360, in recv  
> > > return self.\_recv\_loop(self.fd.recv, b'', bufsize, flags)  
> > > File  
> > > "/home/etl/.local/lib/python3.5/site-packages/eventlet/greenio/base.py",  
> > > line 354, in \_recv\_loop  
> > > self.\_read\_trampoline()  
> > > File  
> > > "/home/etl/.local/lib/python3.5/site-packages/eventlet/greenio/base.py",  
> > > line 325, in \_read\_trampoline  
> > > timeout\_exc=socket\_timeout('timed out'))  
> > > File  
> > > "/home/etl/.local/lib/python3.5/site-packages/eventlet/greenio/base.py",  
> > > line 207, in \_trampoline  
> > > mark\_as\_closed=self.\_mark\_as\_closed)  
> > > File  
> > > "/home/etl/.local/lib/python3.5/site-packages/eventlet/hubs/\_\_init\_\_.py",  
> > > line 163, in trampoline  
> > > return hub.switch()  
> > > File "/home/etl/.local/lib/python3.5/site-packages/eventlet/hubs/hub.py"  
> > > , line 295, in switch  
> > > return self.greenlet.switch()  
> > > socket.timeout: timed out
> > > 
> > > Thanks all!

---

<div class="post-metadata">

**Author:** ![Davis\_Kirkendall](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/davis_kirkendall/32/18_2.png) [@Davis\_Kirkendall](https://discourse.nameko.io/u/Davis_Kirkendall)\
**Post date:** [May 23, 2018, 3:54pm UTC](https://discourse.nameko.io/t/rpc-extension-and-heartbeat-if-my-service-doing-long-task-above-20min/248/4 "2018-05-23T15:54:21Z")

</div>

I'm having problems with setting the "HEARTBEAT" parameter. It seems like  
kombu (or the underlying amqp lib) uses the minimum of server-heartbeat and  
client-heartbeat. So if the server heartbeat is set to 30 seconds, we  
can't increase the heartbeat to 5 minutes for example.

Is there any (functional) reason that the client heartbeat can't be larger  
than the server heartbeat?

btw. This issue here: [RPC erroneously restarted if connection between worker and broker times out · Issue #477 · nameko/nameko · GitHub](https://github.com/nameko/nameko/issues/477) deals  
with the same problem and has some more solutions (unfortunately the tpool  
solution does not work for all cases, hence the heartbeat question above).

Davis

> **···**
>
> Am Montag, 2. April 2018 18:24:55 UTC+2 schrieb juko...@gmail.com:
> 
> > I had a similar issue.
> > 
> > The default is to send a heartbeat every 60 seconds, with 3 retries.
> > 
> > You can increase the heartbeat timeout with the `HEARTBEAT` config value.
> > 
> > If your 'long task' happens to be a SubProcess call (like mine was) you  
> > should make sure you import the 'greened' Subprocess module (Subprocess is  
> > not monkey-patched automatically, you have to specifically import it).
> > 
> > Geoff
> > 
> > On Monday, April 2, 2018 at 2:31:16 AM UTC-7, ts...@atlas.ru wrote:
> > 
> > > The error lies in the fact that the received task from the queue returns  
> > > to the queue back some time later, if you look at the logs there is an  
> > > error associated with the timeouts.  
> > > Here are the detailed logs and question, how do I increase the count of  
> > > heartbeats?
> > > 
> > > \>\> TRACE
> > > 
> > > 2018-03-07 15:14:07,102 - WARNING - Connection to broker lost, trying to  
> > > re-establish connection...  
> > > Traceback (most recent call last):  
> > > File "/home/etl/.local/lib/python3.5/site-packages/kombu/mixins.py",  
> > > line 199, in consume  
> > > conn.drain\_events(timeout=safety\_interval)  
> > > File "/home/etl/.local/lib/python3.5/site-packages/kombu/connection.py",  
> > > line 288, in drain\_events  
> > > return self.transport.drain\_events(self.connection, \*\*kwargs)  
> > > File  
> > > "/home/etl/.local/lib/python3.5/site-packages/kombu/transport/pyamqp.py",  
> > > line 95, in drain\_events  
> > > return connection.drain\_events(\*\*kwargs)  
> > > File "/home/etl/.local/lib/python3.5/site-packages/amqp/connection.py",  
> > > line 303, in drain\_events  
> > > chanmap, None, timeout=timeout,  
> > > File "/home/etl/.local/lib/python3.5/site-packages/amqp/connection.py",  
> > > line 366, in \_wait\_multiple  
> > > channel, method\_sig, args, content = read\_timeout(timeout)  
> > > File "/home/etl/.local/lib/python3.5/site-packages/amqp/connection.py",  
> > > line 337, in read\_timeout  
> > > return self.method\_reader.read\_method()  
> > > File  
> > > "/home/etl/.local/lib/python3.5/site-packages/amqp/method\_framing.py",  
> > > line 189, in read\_method  
> > > raise m  
> > > File  
> > > "/home/etl/.local/lib/python3.5/site-packages/amqp/method\_framing.py",  
> > > line 107, in \_next\_method  
> > > frame\_type, channel, payload = read\_frame()  
> > > File "/home/etl/.local/lib/python3.5/site-packages/amqp/transport.py",  
> > > line 154, in read\_frame  
> > > frame\_header = read(7, True)  
> > > File "/home/etl/.local/lib/python3.5/site-packages/amqp/transport.py",  
> > > line 277, in \_read  
> > > s = recv(n - len(rbuf))  
> > > File  
> > > "/home/etl/.local/lib/python3.5/site-packages/eventlet/greenio/base.py",  
> > > line 360, in recv  
> > > return self.\_recv\_loop(self.fd.recv, b'', bufsize, flags)  
> > > File  
> > > "/home/etl/.local/lib/python3.5/site-packages/eventlet/greenio/base.py",  
> > > line 354, in \_recv\_loop  
> > > self.\_read\_trampoline()  
> > > File  
> > > "/home/etl/.local/lib/python3.5/site-packages/eventlet/greenio/base.py",  
> > > line 325, in \_read\_trampoline  
> > > timeout\_exc=socket\_timeout('timed out'))  
> > > File  
> > > "/home/etl/.local/lib/python3.5/site-packages/eventlet/greenio/base.py",  
> > > line 207, in \_trampoline  
> > > mark\_as\_closed=self.\_mark\_as\_closed)  
> > > File  
> > > "/home/etl/.local/lib/python3.5/site-packages/eventlet/hubs/\_\_init\_\_.py",  
> > > line 163, in trampoline  
> > > return hub.switch()  
> > > File "/home/etl/.local/lib/python3.5/site-packages/eventlet/hubs/hub.py"  
> > > , line 295, in switch  
> > > return self.greenlet.switch()  
> > > socket.timeout: timed out
> > > 
> > > Thanks all!

---

<div class="post-metadata">

**Author:** ![mattbennett](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/mattbennett/32/13_2.png) [@mattbennett](https://discourse.nameko.io/u/mattbennett)\
**Post date:** [July 16, 2018, 8:24am UTC](https://discourse.nameko.io/t/rpc-extension-and-heartbeat-if-my-service-doing-long-task-above-20min/248/5 "2018-07-16T08:24:51Z")

</div>

Davis,

Sorry, I completely missed this post.

The server setting the max heartbeat is a RabbitMQ design. See  
[https://www.rabbitmq.com/heartbeats.html#heartbeats-timeout:](https://www.rabbitmq.com/heartbeats.html#heartbeats-timeout:)

> Note that in case RabbitMQ server has a non-zero heartbeat timeout

configured (which is the default in versions starting with 3.6.x), the  
client can only lower the value but not increase it.

However, a service method that blocks for more than a few seconds is a bad  
idea. As I described in [RPC erroneously restarted if connection between worker and broker times out · Issue #477 · nameko/nameko · GitHub](https://github.com/nameko/nameko/issues/477), this  
is just part of how implicitly yielding threads work. You must yield  
occasionally otherwise any other threads will starve (which is what's  
happening to the AMQP consumer thread here)

In this situation you generally have two choices:

1. Explicitly yield. If you've got some code that's purely CPU bound with  
no I/O, make sure insert an eventlet.yield() occasionally  
2. If you can't insert a yield, for example because the blocking code is in  
some library that you don't control, use the tpool (this is exactly what  
tpool is for)

> **···**
>
> On Wednesday, 23 May 2018 16:54:21 UTC+1, Davis Kirkendall wrote:
> 
> > I'm having problems with setting the "HEARTBEAT" parameter. It seems like  
> > kombu (or the underlying amqp lib) uses the minimum of server-heartbeat and  
> > client-heartbeat. So if the server heartbeat is set to 30 seconds, we  
> > can't increase the heartbeat to 5 minutes for example.
> > 
> > Is there any (functional) reason that the client heartbeat can't be larger  
> > than the server heartbeat?
> > 
> > btw. This issue here: [RPC erroneously restarted if connection between worker and broker times out · Issue #477 · nameko/nameko · GitHub](https://github.com/nameko/nameko/issues/477)  
> > \<[Redirect Notice](https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fnameko%2Fnameko%2Fissues%2F477&sa=D&sntz=1&usg=AFQjCNHRmGc5gE32CldFhrnLNz3XZ-ZXhg&gt);  
> > deals with the same problem and has some more solutions (unfortunately the  
> > tpool solution does not work for all cases, hence the heartbeat question  
> > above).
> > 
> > Davis
> > 
> > Am Montag, 2. April 2018 18:24:55 UTC+2 schrieb juko...@gmail.com:
> > 
> > > I had a similar issue.
> > > 
> > > The default is to send a heartbeat every 60 seconds, with 3 retries.
> > > 
> > > You can increase the heartbeat timeout with the `HEARTBEAT` config value.
> > > 
> > > If your 'long task' happens to be a SubProcess call (like mine was) you  
> > > should make sure you import the 'greened' Subprocess module (Subprocess is  
> > > not monkey-patched automatically, you have to specifically import it).
> > > 
> > > Geoff
> > > 
> > > On Monday, April 2, 2018 at 2:31:16 AM UTC-7, ts...@atlas.ru wrote:
> > > 
> > > > The error lies in the fact that the received task from the queue returns  
> > > > to the queue back some time later, if you look at the logs there is an  
> > > > error associated with the timeouts.  
> > > > Here are the detailed logs and question, how do I increase the count of  
> > > > heartbeats?
> > > > 
> > > > \>\> TRACE
> > > > 
> > > > 2018-03-07 15:14:07,102 - WARNING - Connection to broker lost, trying  
> > > > to re-establish connection...  
> > > > Traceback (most recent call last):  
> > > > File "/home/etl/.local/lib/python3.5/site-packages/kombu/mixins.py",  
> > > > line 199, in consume  
> > > > conn.drain\_events(timeout=safety\_interval)  
> > > > File "/home/etl/.local/lib/python3.5/site-packages/kombu/connection.py"  
> > > > , line 288, in drain\_events  
> > > > return self.transport.drain\_events(self.connection, \*\*kwargs)  
> > > > File  
> > > > "/home/etl/.local/lib/python3.5/site-packages/kombu/transport/pyamqp.py"  
> > > > , line 95, in drain\_events  
> > > > return connection.drain\_events(\*\*kwargs)  
> > > > File "/home/etl/.local/lib/python3.5/site-packages/amqp/connection.py",  
> > > > line 303, in drain\_events  
> > > > chanmap, None, timeout=timeout,  
> > > > File "/home/etl/.local/lib/python3.5/site-packages/amqp/connection.py",  
> > > > line 366, in \_wait\_multiple  
> > > > channel, method\_sig, args, content = read\_timeout(timeout)  
> > > > File "/home/etl/.local/lib/python3.5/site-packages/amqp/connection.py",  
> > > > line 337, in read\_timeout  
> > > > return self.method\_reader.read\_method()  
> > > > File  
> > > > "/home/etl/.local/lib/python3.5/site-packages/amqp/method\_framing.py",  
> > > > line 189, in read\_method  
> > > > raise m  
> > > > File  
> > > > "/home/etl/.local/lib/python3.5/site-packages/amqp/method\_framing.py",  
> > > > line 107, in \_next\_method  
> > > > frame\_type, channel, payload = read\_frame()  
> > > > File "/home/etl/.local/lib/python3.5/site-packages/amqp/transport.py",  
> > > > line 154, in read\_frame  
> > > > frame\_header = read(7, True)  
> > > > File "/home/etl/.local/lib/python3.5/site-packages/amqp/transport.py",  
> > > > line 277, in \_read  
> > > > s = recv(n - len(rbuf))  
> > > > File  
> > > > "/home/etl/.local/lib/python3.5/site-packages/eventlet/greenio/base.py",  
> > > > line 360, in recv  
> > > > return self.\_recv\_loop(self.fd.recv, b'', bufsize, flags)  
> > > > File  
> > > > "/home/etl/.local/lib/python3.5/site-packages/eventlet/greenio/base.py",  
> > > > line 354, in \_recv\_loop  
> > > > self.\_read\_trampoline()  
> > > > File  
> > > > "/home/etl/.local/lib/python3.5/site-packages/eventlet/greenio/base.py",  
> > > > line 325, in \_read\_trampoline  
> > > > timeout\_exc=socket\_timeout('timed out'))  
> > > > File  
> > > > "/home/etl/.local/lib/python3.5/site-packages/eventlet/greenio/base.py",  
> > > > line 207, in \_trampoline  
> > > > mark\_as\_closed=self.\_mark\_as\_closed)  
> > > > File  
> > > > "/home/etl/.local/lib/python3.5/site-packages/eventlet/hubs/\_\_init\_\_.py"  
> > > > , line 163, in trampoline  
> > > > return hub.switch()  
> > > > File  
> > > > "/home/etl/.local/lib/python3.5/site-packages/eventlet/hubs/hub.py",  
> > > > line 295, in switch  
> > > > return self.greenlet.switch()  
> > > > socket.timeout: timed out
> > > > 
> > > > Thanks all!

---

<div class="post-metadata">

**Author:** ![nixiam](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/nixiam/32/166_2.png) [@nixiam](https://discourse.nameko.io/u/nixiam)\
**Post date:** [January 29, 2022, 6:41pm UTC](https://discourse.nameko.io/t/rpc-extension-and-heartbeat-if-my-service-doing-long-task-above-20min/248/6 "2022-01-29T18:41:44Z")

</div>

Hi,

i have same issue with training a ML model.

After train ended (about 1h) i received the error

```auto
Connection to broker lost, trying to re-establish connection...
Traceback (most recent call last):
  File "/Users/.../lib/python3.8/site-packages/kombu/mixins.py", line 193, in consume
    conn.drain_events(timeout=safety_interval)
  File "/Users/.../lib/python3.8/site-packages/kombu/connection.py", line 317, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/Users/.../lib/python3.8/site-packages/kombu/transport/pyamqp.py", line 169, in drain_events
    return connection.drain_events(**kwargs)
  File "/Users/.../lib/python3.8/site-packages/amqp/connection.py", line 522, in drain_events
    while not self.blocking_read(timeout):
  File "/Users/.../lib/python3.8/site-packages/amqp/connection.py", line 527, in blocking_read
    frame = self.transport.read_frame()
  File "/Users/.../lib/python3.8/site-packages/amqp/transport.py", line 310, in read_frame
    frame_header = read(7, True)
  File "/Users/.../lib/python3.8/site-packages/amqp/transport.py", line 639, in _read
    s = recv(n - len(rbuf))
  File "/Users/.../lib/python3.8/site-packages/eventlet/greenio/base.py", line 370, in recv
    return self._recv_loop(self.fd.recv, b'', bufsize, flags)
  File "/Users/.../lib/python3.8/site-packages/eventlet/greenio/base.py", line 364, in _recv_loop
    self._read_trampoline()
  File "/Users/.../lib/python3.8/site-packages/eventlet/greenio/base.py", line 332, in _read_trampoline
    self._trampoline(
  File "/Users/.../lib/python3.8/site-packages/eventlet/greenio/base.py", line 211, in _trampoline
    return trampoline(fd, read=read, write=write, timeout=timeout,
  File "/Users/.../lib/python3.8/site-packages/eventlet/hubs/ __init__.py", line 159, in trampoline
    return hub.switch()
  File "/Users/t.../lib/python3.8/site-packages/eventlet/hubs/hub.py", line 313, in switch
    return self.greenlet.switch()
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/.../lib/python3.8/site-packages/kombu/mixins.py", line 171, in run
    for _ in self.consume(limit=None, **kwargs):
  File "/Users/.../lib/python3.8/site-packages/kombu/mixins.py", line 195, in consume
    conn.heartbeat_check()
  File "/Users/.../lib/python3.8/site-packages/kombu/connection.py", line 306, in heartbeat_check
    return self.transport.heartbeat_check(self.connection, rate=rate)
  File "/Users/.../lib/python3.8/site-packages/kombu/transport/pyamqp.py", line 220, in heartbeat_check
    return connection.heartbeat_tick(rate=rate)
  File "/Users/.../lib/python3.8/site-packages/amqp/connection.py", line 767, in heartbeat_tick
    raise ConnectionForced('Too many heartbeats missed')
amqp.exceptions.ConnectionForced: Too many heartbeats missed

```

Can you tell me how you fixed it?