# Making nameko not stop when runtime errors occur in dependencies

**URL:** <https://discourse.nameko.io/t/making-nameko-not-stop-when-runtime-errors-occur-in-dependencies/128>\
**Category:** googlegroup\
**Created:** [April 7, 2016, 11:45am UTC](https://discourse.nameko.io/t/making-nameko-not-stop-when-runtime-errors-occur-in-dependencies/128 "2016-04-07T11:45:53Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tsachi\_Shuval](https://avatars.discourse-cdn.com/v4/letter/t/5e9695/32.png) [@Tsachi\_Shuval](https://discourse.nameko.io/u/Tsachi_Shuval)\
**Post date:** [April 7, 2016, 11:45am UTC](https://discourse.nameko.io/t/making-nameko-not-stop-when-runtime-errors-occur-in-dependencies/128/1 "2016-04-07T11:45:53Z")

</div>

Hi,

Seems like a runtime error in a dependency stops nameko (you can "raise"  
somethind in a get\_dependency() method to test it).

This is a very BAD scenario.  
So, other than try..except the code block, is there a way to prevent nameko  
from stopping?

Thanks.

---

<div class="post-metadata">

**Author:** ![David\_Szotten](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/david_szotten/32/6_2.png) [@David\_Szotten](https://discourse.nameko.io/u/David_Szotten)\
**Post date:** [April 7, 2016, 12:05pm UTC](https://discourse.nameko.io/t/making-nameko-not-stop-when-runtime-errors-occur-in-dependencies/128/2 "2016-04-07T12:05:51Z")

</div>

Hi,

`get_dependency` is responsible for creating the dependency to be injected  
into the service. if this fails, the service won't have its dependency  
(which it "depends on") and the implication is that the service can't run  
correctly, so the container terminates.

what would be your preferred outcome? the service running without that  
particular dependency? perhaps you could provide some more details around  
your particular use-case to better help me understand

Best,  
David

> **···**
>
> On Thursday, 7 April 2016 12:45:53 UTC+1, tsachi...@gmail.com wrote:
> 
> > Hi,
> > 
> > Seems like a runtime error in a dependency stops nameko (you can "raise"  
> > somethind in a get\_dependency() method to test it).
> > 
> > This is a very BAD scenario.  
> > So, other than try..except the code block, is there a way to prevent  
> > nameko from stopping?
> > 
> > Thanks.

---

<div class="post-metadata">

**Author:** ![Tsachi\_Shuval](https://avatars.discourse-cdn.com/v4/letter/t/5e9695/32.png) [@Tsachi\_Shuval](https://discourse.nameko.io/u/Tsachi_Shuval)\
**Post date:** [April 7, 2016, 12:17pm UTC](https://discourse.nameko.io/t/making-nameko-not-stop-when-runtime-errors-occur-in-dependencies/128/3 "2016-04-07T12:17:11Z")

</div>

Hi David,

Thanks for the prompt reply.

There is no special use case. It happened to me while my Mongo DB threw an  
exception (the get\_dependency returns a Mongo DB connection object).  
I agree that these kind of operations (IO, etc) should be try..except'ed,  
but still, since nameko is a core service which other components rely upon,  
as far as I'm concerned it should "never fail".

Imagine that Tornado/Apache/nginx/RabbitMQ/etc simply stop due to an  
internal error. This is unacceptable.

IMHO, if a dependency throws an error, the current worker context should  
quit, but the service should still be up and running.  
This error maybe temporary (e.g. timeout, malformed input data to the  
worker-\>injected dependency, etc), so why fail completely?

Don't you agree?

> **···**
>
> On Thursday, April 7, 2016 at 3:05:51 PM UTC+3, David Szotten wrote:
> 
> > Hi,
> > 
> > `get_dependency` is responsible for creating the dependency to be injected  
> > into the service. if this fails, the service won't have its dependency  
> > (which it "depends on") and the implication is that the service can't run  
> > correctly, so the container terminates.
> > 
> > what would be your preferred outcome? the service running without that  
> > particular dependency? perhaps you could provide some more details around  
> > your particular use-case to better help me understand
> > 
> > Best,  
> > David
> > 
> > On Thursday, 7 April 2016 12:45:53 UTC+1, tsachi...@gmail.com wrote:
> > 
> > > Hi,
> > > 
> > > Seems like a runtime error in a dependency stops nameko (you can "raise"  
> > > somethind in a get\_dependency() method to test it).
> > > 
> > > This is a very BAD scenario.  
> > > So, other than try..except the code block, is there a way to prevent  
> > > nameko from stopping?
> > > 
> > > Thanks.

---

<div class="post-metadata">

**Author:** ![David\_Szotten](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/david_szotten/32/6_2.png) [@David\_Szotten](https://discourse.nameko.io/u/David_Szotten)\
**Post date:** [April 7, 2016, 12:30pm UTC](https://discourse.nameko.io/t/making-nameko-not-stop-when-runtime-errors-occur-in-dependencies/128/4 "2016-04-07T12:30:42Z")

</div>

Hi,

Nameko can't tell the difference between e.g. connection error  
(recoverable) and say configuration error (e.g. bad mongo hostname, not  
recoverable). What was the nature of the mongo exception in your case? if  
you occasionally fail to connect due to temporary errors, you might want to  
add retry behaviour inside your dependency.

We tended to find that we get better mileage from running nameko using a  
supervisor process (onefinestay uses supervisord), and letting that restart  
the whole process on that type of error

D

> **···**
>
> On Thursday, 7 April 2016 13:17:12 UTC+1, tsachi...@gmail.com wrote:
> 
> > Hi David,
> > 
> > Thanks for the prompt reply.
> > 
> > There is no special use case. It happened to me while my Mongo DB threw an  
> > exception (the get\_dependency returns a Mongo DB connection object).  
> > I agree that these kind of operations (IO, etc) should be try..except'ed,  
> > but still, since nameko is a core service which other components rely upon,  
> > as far as I'm concerned it should "never fail".
> > 
> > Imagine that Tornado/Apache/nginx/RabbitMQ/etc simply stop due to an  
> > internal error. This is unacceptable.
> > 
> > IMHO, if a dependency throws an error, the current worker context should  
> > quit, but the service should still be up and running.  
> > This error maybe temporary (e.g. timeout, malformed input data to the  
> > worker-\>injected dependency, etc), so why fail completely?
> > 
> > Don't you agree?
> > 
> > On Thursday, April 7, 2016 at 3:05:51 PM UTC+3, David Szotten wrote:
> > 
> > > Hi,
> > > 
> > > `get_dependency` is responsible for creating the dependency to be  
> > > injected into the service. if this fails, the service won't have its  
> > > dependency (which it "depends on") and the implication is that the service  
> > > can't run correctly, so the container terminates.
> > > 
> > > what would be your preferred outcome? the service running without that  
> > > particular dependency? perhaps you could provide some more details around  
> > > your particular use-case to better help me understand
> > > 
> > > Best,  
> > > David
> > > 
> > > On Thursday, 7 April 2016 12:45:53 UTC+1, tsachi...@gmail.com wrote:
> > > 
> > > > Hi,
> > > > 
> > > > Seems like a runtime error in a dependency stops nameko (you can "raise"  
> > > > somethind in a get\_dependency() method to test it).
> > > > 
> > > > This is a very BAD scenario.  
> > > > So, other than try..except the code block, is there a way to prevent  
> > > > nameko from stopping?
> > > > 
> > > > Thanks.

---

<div class="post-metadata">

**Author:** ![mattbennett](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/mattbennett/32/13_2.png) [@mattbennett](https://discourse.nameko.io/u/mattbennett)\
**Post date:** [April 10, 2016, 8:26pm UTC](https://discourse.nameko.io/t/making-nameko-not-stop-when-runtime-errors-occur-in-dependencies/128/5 "2016-04-10T20:26:21Z")

</div>

This behaviour is intentional, but I can understand why it’s confusing.

As David explained, Nameko can’t tell whether errors are transient or  
fatal, so it does the “safe” thing and assumes the error is unrecoverable.  
The only time this isn’t true is during the worker lifecycle (i.e. when  
executing a service method) because an error there can be attributed to the  
active worker.

There is a valid argument that errors specifically in `get_dependency` are  
also tied to a particular worker. It might be more intuitive to catch them  
and return an appropriate error to the caller (if there is one). The caller  
could then retry if appropriate, and the service would stay up in the mean  
time. This also doesn’t preclude a DependencyProvider from catching  
exceptions and retrying itself, if it wanted to hide transient errors from  
the caller.

However, it wouldn't change the fact that users should “expect failures”  
and run Nameko under a process manager though. The Nameko process will  
still exit, for example, if an entrypoint can’t serialise the result of a  
service method and hasn’t been made appropriately robust. Unlike  
`get_dependency` there’s nothing we can do in Nameko core to protect  
against that kind of error.

What do you think David?

> **···**
>
> On Thursday, April 7, 2016 at 1:30:42 PM UTC+1, David Szotten wrote:
> 
> > Hi,
> > 
> > Nameko can't tell the difference between e.g. connection error  
> > (recoverable) and say configuration error (e.g. bad mongo hostname, not  
> > recoverable). What was the nature of the mongo exception in your case? if  
> > you occasionally fail to connect due to temporary errors, you might want to  
> > add retry behaviour inside your dependency.
> > 
> > We tended to find that we get better mileage from running nameko using a  
> > supervisor process (onefinestay uses supervisord), and letting that restart  
> > the whole process on that type of error
> > 
> > D
> > 
> > On Thursday, 7 April 2016 13:17:12 UTC+1, tsachi...@gmail.com wrote:
> > 
> > > Hi David,
> > > 
> > > Thanks for the prompt reply.
> > > 
> > > There is no special use case. It happened to me while my Mongo DB threw  
> > > an exception (the get\_dependency returns a Mongo DB connection object).  
> > > I agree that these kind of operations (IO, etc) should be try..except'ed,  
> > > but still, since nameko is a core service which other components rely upon,  
> > > as far as I'm concerned it should "never fail".
> > > 
> > > Imagine that Tornado/Apache/nginx/RabbitMQ/etc simply stop due to an  
> > > internal error. This is unacceptable.
> > > 
> > > IMHO, if a dependency throws an error, the current worker context should  
> > > quit, but the service should still be up and running.  
> > > This error maybe temporary (e.g. timeout, malformed input data to the  
> > > worker-\>injected dependency, etc), so why fail completely?
> > > 
> > > Don't you agree?
> > > 
> > > On Thursday, April 7, 2016 at 3:05:51 PM UTC+3, David Szotten wrote:
> > > 
> > > > Hi,
> > > > 
> > > > `get_dependency` is responsible for creating the dependency to be  
> > > > injected into the service. if this fails, the service won't have its  
> > > > dependency (which it "depends on") and the implication is that the service  
> > > > can't run correctly, so the container terminates.
> > > > 
> > > > what would be your preferred outcome? the service running without that  
> > > > particular dependency? perhaps you could provide some more details around  
> > > > your particular use-case to better help me understand
> > > > 
> > > > Best,  
> > > > David
> > > > 
> > > > On Thursday, 7 April 2016 12:45:53 UTC+1, tsachi...@gmail.com wrote:
> > > > 
> > > > > Hi,
> > > > > 
> > > > > Seems like a runtime error in a dependency stops nameko (you can  
> > > > > "raise" somethind in a get\_dependency() method to test it).
> > > > > 
> > > > > This is a very BAD scenario.  
> > > > > So, other than try..except the code block, is there a way to prevent  
> > > > > nameko from stopping?
> > > > > 
> > > > > Thanks.

---

<div class="post-metadata">

**Author:** ![David\_Szotten](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/david_szotten/32/6_2.png) [@David\_Szotten](https://discourse.nameko.io/u/David_Szotten)\
**Post date:** [April 11, 2016, 8:51pm UTC](https://discourse.nameko.io/t/making-nameko-not-stop-when-runtime-errors-occur-in-dependencies/128/6 "2016-04-11T20:51:32Z")

</div>

Hi,

Describing the behaviour and our reasons above, the existing behaviour does  
feel a bit unsatisfactory.

Rather than crashing, it might make sense for calls to return some kind of  
"server error" response (a la 500 errors in http).

While nameko can't "protect" against dependencies failing or unserialisable  
responses, we could do something other than crash the process. It doesn't  
seem unreasonable to me to expect the runner to survive such things and  
"use a supervisor" (as was my original advice) seems like a bit of a cop  
out.

I haven't thought about all implications (e.g. would "extension errors" ack  
or nack amqp messages? does each entrypoint decide?) nor about how  
difficult this would be to implement, nor about backwards compat. but the  
general idea might be something we want to think about

best,  
d

> **···**
>
> On Sunday, 10 April 2016 21:26:22 UTC+1, Matt Bennett wrote:
> 
> > This behaviour is intentional, but I can understand why it’s confusing.
> > 
> > As David explained, Nameko can’t tell whether errors are transient or  
> > fatal, so it does the “safe” thing and assumes the error is unrecoverable.  
> > The only time this isn’t true is during the worker lifecycle (i.e. when  
> > executing a service method) because an error there can be attributed to the  
> > active worker.
> > 
> > There is a valid argument that errors specifically in `get_dependency` are  
> > also tied to a particular worker. It might be more intuitive to catch them  
> > and return an appropriate error to the caller (if there is one). The caller  
> > could then retry if appropriate, and the service would stay up in the mean  
> > time. This also doesn’t preclude a DependencyProvider from catching  
> > exceptions and retrying itself, if it wanted to hide transient errors from  
> > the caller.
> > 
> > However, it wouldn't change the fact that users should “expect failures”  
> > and run Nameko under a process manager though. The Nameko process will  
> > still exit, for example, if an entrypoint can’t serialise the result of a  
> > service method and hasn’t been made appropriately robust. Unlike  
> > `get_dependency` there’s nothing we can do in Nameko core to protect  
> > against that kind of error.
> > 
> > What do you think David?
> > 
> > On Thursday, April 7, 2016 at 1:30:42 PM UTC+1, David Szotten wrote:
> > 
> > > Hi,
> > > 
> > > Nameko can't tell the difference between e.g. connection error  
> > > (recoverable) and say configuration error (e.g. bad mongo hostname, not  
> > > recoverable). What was the nature of the mongo exception in your case? if  
> > > you occasionally fail to connect due to temporary errors, you might want to  
> > > add retry behaviour inside your dependency.
> > > 
> > > We tended to find that we get better mileage from running nameko using a  
> > > supervisor process (onefinestay uses supervisord), and letting that restart  
> > > the whole process on that type of error
> > > 
> > > D
> > > 
> > > On Thursday, 7 April 2016 13:17:12 UTC+1, tsachi...@gmail.com wrote:
> > > 
> > > > Hi David,
> > > > 
> > > > Thanks for the prompt reply.
> > > > 
> > > > There is no special use case. It happened to me while my Mongo DB threw  
> > > > an exception (the get\_dependency returns a Mongo DB connection object).  
> > > > I agree that these kind of operations (IO, etc) should be  
> > > > try..except'ed, but still, since nameko is a core service which other  
> > > > components rely upon, as far as I'm concerned it should "never fail".
> > > > 
> > > > Imagine that Tornado/Apache/nginx/RabbitMQ/etc simply stop due to an  
> > > > internal error. This is unacceptable.
> > > > 
> > > > IMHO, if a dependency throws an error, the current worker context should  
> > > > quit, but the service should still be up and running.  
> > > > This error maybe temporary (e.g. timeout, malformed input data to the  
> > > > worker-\>injected dependency, etc), so why fail completely?
> > > > 
> > > > Don't you agree?
> > > > 
> > > > On Thursday, April 7, 2016 at 3:05:51 PM UTC+3, David Szotten wrote:
> > > > 
> > > > > Hi,
> > > > > 
> > > > > `get_dependency` is responsible for creating the dependency to be  
> > > > > injected into the service. if this fails, the service won't have its  
> > > > > dependency (which it "depends on") and the implication is that the service  
> > > > > can't run correctly, so the container terminates.
> > > > > 
> > > > > what would be your preferred outcome? the service running without that  
> > > > > particular dependency? perhaps you could provide some more details around  
> > > > > your particular use-case to better help me understand
> > > > > 
> > > > > Best,  
> > > > > David
> > > > > 
> > > > > On Thursday, 7 April 2016 12:45:53 UTC+1, tsachi...@gmail.com wrote:
> > > > > 
> > > > > > Hi,
> > > > > > 
> > > > > > Seems like a runtime error in a dependency stops nameko (you can  
> > > > > > "raise" somethind in a get\_dependency() method to test it).
> > > > > > 
> > > > > > This is a very BAD scenario.  
> > > > > > So, other than try..except the code block, is there a way to prevent  
> > > > > > nameko from stopping?
> > > > > > 
> > > > > > Thanks.

---

<div class="post-metadata">

**Author:** ![Tsachi\_Shuval](https://avatars.discourse-cdn.com/v4/letter/t/5e9695/32.png) [@Tsachi\_Shuval](https://discourse.nameko.io/u/Tsachi_Shuval)\
**Post date:** [April 12, 2016, 7:49am UTC](https://discourse.nameko.io/t/making-nameko-not-stop-when-runtime-errors-occur-in-dependencies/128/7 "2016-04-12T07:49:17Z")

</div>

Hi,

I agree with you guys that there is no alternative to good coding  
practices, exception handling, input sanitation and process monitoring.  
Having said that, from the discussion so far, it looks like crashing the  
server is almost by design!? "if there's an error in X then nameko will  
\*crash\*"

Of course IO operations in the dependency will generally be try..except'd,  
but imagine a calculation block which happens to throw a division by zero.  
Or maybe I'm relying on an external package (openCV, PIL, whatever) that  
threw an error due to bad arguments passed to one of its functions.  
Should the service crash in such cases?

IMO, nameko should catch these errors, raise exception or return an 'error'  
response, but never crash.  
Of course I am not involved in this project, so I can't tell if it's doable  
(conceptually, practically, etc) and I don't know the impact of  
implementing it.

Anyway, thank you guys for discussing this subject. Would love to hear your  
thoughts and any updates.

Best  
T

> **···**
>
> On Monday, April 11, 2016 at 11:51:33 PM UTC+3, David Szotten wrote:
> 
> > Hi,
> > 
> > Describing the behaviour and our reasons above, the existing behaviour  
> > does feel a bit unsatisfactory.
> > 
> > Rather than crashing, it might make sense for calls to return some kind of  
> > "server error" response (a la 500 errors in http).
> > 
> > While nameko can't "protect" against dependencies failing or  
> > unserialisable responses, we could do something other than crash the  
> > process. It doesn't seem unreasonable to me to expect the runner to survive  
> > such things and "use a supervisor" (as was my original advice) seems like a  
> > bit of a cop out.
> > 
> > I haven't thought about all implications (e.g. would "extension errors"  
> > ack or nack amqp messages? does each entrypoint decide?) nor about how  
> > difficult this would be to implement, nor about backwards compat. but the  
> > general idea might be something we want to think about
> > 
> > best,  
> > d
> > 
> > On Sunday, 10 April 2016 21:26:22 UTC+1, Matt Bennett wrote:
> > 
> > > This behaviour is intentional, but I can understand why it’s confusing.
> > > 
> > > As David explained, Nameko can’t tell whether errors are transient or  
> > > fatal, so it does the “safe” thing and assumes the error is unrecoverable.  
> > > The only time this isn’t true is during the worker lifecycle (i.e. when  
> > > executing a service method) because an error there can be attributed to the  
> > > active worker.
> > > 
> > > There is a valid argument that errors specifically in `get_dependency`  
> > > are also tied to a particular worker. It might be more intuitive to catch  
> > > them and return an appropriate error to the caller (if there is one). The  
> > > caller could then retry if appropriate, and the service would stay up in  
> > > the mean time. This also doesn’t preclude a DependencyProvider from  
> > > catching exceptions and retrying itself, if it wanted to hide transient  
> > > errors from the caller.
> > > 
> > > However, it wouldn't change the fact that users should “expect failures”  
> > > and run Nameko under a process manager though. The Nameko process will  
> > > still exit, for example, if an entrypoint can’t serialise the result of a  
> > > service method and hasn’t been made appropriately robust. Unlike  
> > > `get_dependency` there’s nothing we can do in Nameko core to protect  
> > > against that kind of error.
> > > 
> > > What do you think David?
> > > 
> > > On Thursday, April 7, 2016 at 1:30:42 PM UTC+1, David Szotten wrote:
> > > 
> > > > Hi,
> > > > 
> > > > Nameko can't tell the difference between e.g. connection error  
> > > > (recoverable) and say configuration error (e.g. bad mongo hostname, not  
> > > > recoverable). What was the nature of the mongo exception in your case? if  
> > > > you occasionally fail to connect due to temporary errors, you might want to  
> > > > add retry behaviour inside your dependency.
> > > > 
> > > > We tended to find that we get better mileage from running nameko using a  
> > > > supervisor process (onefinestay uses supervisord), and letting that restart  
> > > > the whole process on that type of error
> > > > 
> > > > D
> > > > 
> > > > On Thursday, 7 April 2016 13:17:12 UTC+1, tsachi...@gmail.com wrote:
> > > > 
> > > > > Hi David,
> > > > > 
> > > > > Thanks for the prompt reply.
> > > > > 
> > > > > There is no special use case. It happened to me while my Mongo DB threw  
> > > > > an exception (the get\_dependency returns a Mongo DB connection object).  
> > > > > I agree that these kind of operations (IO, etc) should be  
> > > > > try..except'ed, but still, since nameko is a core service which other  
> > > > > components rely upon, as far as I'm concerned it should "never fail".
> > > > > 
> > > > > Imagine that Tornado/Apache/nginx/RabbitMQ/etc simply stop due to an  
> > > > > internal error. This is unacceptable.
> > > > > 
> > > > > IMHO, if a dependency throws an error, the current worker context  
> > > > > should quit, but the service should still be up and running.  
> > > > > This error maybe temporary (e.g. timeout, malformed input data to the  
> > > > > worker-\>injected dependency, etc), so why fail completely?
> > > > > 
> > > > > Don't you agree?
> > > > > 
> > > > > On Thursday, April 7, 2016 at 3:05:51 PM UTC+3, David Szotten wrote:
> > > > > 
> > > > > > Hi,
> > > > > > 
> > > > > > `get_dependency` is responsible for creating the dependency to be  
> > > > > > injected into the service. if this fails, the service won't have its  
> > > > > > dependency (which it "depends on") and the implication is that the service  
> > > > > > can't run correctly, so the container terminates.
> > > > > > 
> > > > > > what would be your preferred outcome? the service running without that  
> > > > > > particular dependency? perhaps you could provide some more details around  
> > > > > > your particular use-case to better help me understand
> > > > > > 
> > > > > > Best,  
> > > > > > David
> > > > > > 
> > > > > > On Thursday, 7 April 2016 12:45:53 UTC+1, tsachi...@gmail.com wrote:
> > > > > > 
> > > > > > > Hi,
> > > > > > > 
> > > > > > > Seems like a runtime error in a dependency stops nameko (you can  
> > > > > > > "raise" somethind in a get\_dependency() method to test it).
> > > > > > > 
> > > > > > > This is a very BAD scenario.  
> > > > > > > So, other than try..except the code block, is there a way to prevent  
> > > > > > > nameko from stopping?
> > > > > > > 
> > > > > > > Thanks.

---

<div class="post-metadata">

**Author:** ![David\_Szotten](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.nameko.io/david_szotten/32/6_2.png) [@David\_Szotten](https://discourse.nameko.io/u/David_Szotten)\
**Post date:** [April 12, 2016, 11:37am UTC](https://discourse.nameko.io/t/making-nameko-not-stop-when-runtime-errors-occur-in-dependencies/128/8 "2016-04-12T11:37:02Z")

</div>

Just to be clear, the situation isn't quite that dire. Nameko makes a  
distinction between service code and "framework/plumbing" (core +  
extensions) code. Uncaught exceptions in service code \_are\_ caught by the  
framework and result in error responses

This discussion is about the possibility of changing the behaviour of  
exceptions in extensions and elsewhere, e.g. from get\_dependency, and other  
places that are tied to a worker lifecycle from the former to the latter  
behaviour. For certain errors i do think crashing is the correct behaviour,  
e.g. during a "setup" phase. E.g. if my app can't resolve the hostname of  
my db server, i want it to crash at startup, not fail when serving its  
first (and all subsequent) requests

d

> **···**
>
> On Tuesday, 12 April 2016 08:49:18 UTC+1, tsachi...@gmail.com wrote:
> 
> > Hi,
> > 
> > I agree with you guys that there is no alternative to good coding  
> > practices, exception handling, input sanitation and process monitoring.  
> > Having said that, from the discussion so far, it looks like crashing the  
> > server is almost by design!? "if there's an error in X then nameko will  
> > \*crash\*"
> > 
> > Of course IO operations in the dependency will generally be try..except'd,  
> > but imagine a calculation block which happens to throw a division by zero.  
> > Or maybe I'm relying on an external package (openCV, PIL, whatever) that  
> > threw an error due to bad arguments passed to one of its functions.  
> > Should the service crash in such cases?
> > 
> > IMO, nameko should catch these errors, raise exception or return an  
> > 'error' response, but never crash.  
> > Of course I am not involved in this project, so I can't tell if it's  
> > doable (conceptually, practically, etc) and I don't know the impact of  
> > implementing it.
> > 
> > Anyway, thank you guys for discussing this subject. Would love to hear  
> > your thoughts and any updates.
> > 
> > Best  
> > T
> > 
> > On Monday, April 11, 2016 at 11:51:33 PM UTC+3, David Szotten wrote:
> > 
> > > Hi,
> > > 
> > > Describing the behaviour and our reasons above, the existing behaviour  
> > > does feel a bit unsatisfactory.
> > > 
> > > Rather than crashing, it might make sense for calls to return some kind  
> > > of "server error" response (a la 500 errors in http).
> > > 
> > > While nameko can't "protect" against dependencies failing or  
> > > unserialisable responses, we could do something other than crash the  
> > > process. It doesn't seem unreasonable to me to expect the runner to survive  
> > > such things and "use a supervisor" (as was my original advice) seems like a  
> > > bit of a cop out.
> > > 
> > > I haven't thought about all implications (e.g. would "extension errors"  
> > > ack or nack amqp messages? does each entrypoint decide?) nor about how  
> > > difficult this would be to implement, nor about backwards compat. but the  
> > > general idea might be something we want to think about
> > > 
> > > best,  
> > > d
> > > 
> > > On Sunday, 10 April 2016 21:26:22 UTC+1, Matt Bennett wrote:
> > > 
> > > > This behaviour is intentional, but I can understand why it’s confusing.
> > > > 
> > > > As David explained, Nameko can’t tell whether errors are transient or  
> > > > fatal, so it does the “safe” thing and assumes the error is unrecoverable.  
> > > > The only time this isn’t true is during the worker lifecycle (i.e. when  
> > > > executing a service method) because an error there can be attributed to the  
> > > > active worker.
> > > > 
> > > > There is a valid argument that errors specifically in `get_dependency`  
> > > > are also tied to a particular worker. It might be more intuitive to catch  
> > > > them and return an appropriate error to the caller (if there is one). The  
> > > > caller could then retry if appropriate, and the service would stay up in  
> > > > the mean time. This also doesn’t preclude a DependencyProvider from  
> > > > catching exceptions and retrying itself, if it wanted to hide transient  
> > > > errors from the caller.
> > > > 
> > > > However, it wouldn't change the fact that users should “expect failures”  
> > > > and run Nameko under a process manager though. The Nameko process will  
> > > > still exit, for example, if an entrypoint can’t serialise the result of a  
> > > > service method and hasn’t been made appropriately robust. Unlike  
> > > > `get_dependency` there’s nothing we can do in Nameko core to protect  
> > > > against that kind of error.
> > > > 
> > > > What do you think David?
> > > > 
> > > > On Thursday, April 7, 2016 at 1:30:42 PM UTC+1, David Szotten wrote:
> > > > 
> > > > > Hi,
> > > > > 
> > > > > Nameko can't tell the difference between e.g. connection error  
> > > > > (recoverable) and say configuration error (e.g. bad mongo hostname, not  
> > > > > recoverable). What was the nature of the mongo exception in your case? if  
> > > > > you occasionally fail to connect due to temporary errors, you might want to  
> > > > > add retry behaviour inside your dependency.
> > > > > 
> > > > > We tended to find that we get better mileage from running nameko using  
> > > > > a supervisor process (onefinestay uses supervisord), and letting that  
> > > > > restart the whole process on that type of error
> > > > > 
> > > > > D
> > > > > 
> > > > > On Thursday, 7 April 2016 13:17:12 UTC+1, tsachi...@gmail.com wrote:
> > > > > 
> > > > > > Hi David,
> > > > > > 
> > > > > > Thanks for the prompt reply.
> > > > > > 
> > > > > > There is no special use case. It happened to me while my Mongo DB  
> > > > > > threw an exception (the get\_dependency returns a Mongo DB connection  
> > > > > > object).  
> > > > > > I agree that these kind of operations (IO, etc) should be  
> > > > > > try..except'ed, but still, since nameko is a core service which other  
> > > > > > components rely upon, as far as I'm concerned it should "never fail".
> > > > > > 
> > > > > > Imagine that Tornado/Apache/nginx/RabbitMQ/etc simply stop due to an  
> > > > > > internal error. This is unacceptable.
> > > > > > 
> > > > > > IMHO, if a dependency throws an error, the current worker context  
> > > > > > should quit, but the service should still be up and running.  
> > > > > > This error maybe temporary (e.g. timeout, malformed input data to the  
> > > > > > worker-\>injected dependency, etc), so why fail completely?
> > > > > > 
> > > > > > Don't you agree?
> > > > > > 
> > > > > > On Thursday, April 7, 2016 at 3:05:51 PM UTC+3, David Szotten wrote:
> > > > > > 
> > > > > > > Hi,
> > > > > > > 
> > > > > > > `get_dependency` is responsible for creating the dependency to be  
> > > > > > > injected into the service. if this fails, the service won't have its  
> > > > > > > dependency (which it "depends on") and the implication is that the service  
> > > > > > > can't run correctly, so the container terminates.
> > > > > > > 
> > > > > > > what would be your preferred outcome? the service running without  
> > > > > > > that particular dependency? perhaps you could provide some more details  
> > > > > > > around your particular use-case to better help me understand
> > > > > > > 
> > > > > > > Best,  
> > > > > > > David
> > > > > > > 
> > > > > > > On Thursday, 7 April 2016 12:45:53 UTC+1, tsachi...@gmail.com wrote:
> > > > > > > 
> > > > > > > > Hi,
> > > > > > > > 
> > > > > > > > Seems like a runtime error in a dependency stops nameko (you can  
> > > > > > > > "raise" somethind in a get\_dependency() method to test it).
> > > > > > > > 
> > > > > > > > This is a very BAD scenario.  
> > > > > > > > So, other than try..except the code block, is there a way to prevent  
> > > > > > > > nameko from stopping?
> > > > > > > > 
> > > > > > > > Thanks.

---

<div class="post-metadata">

**Author:** ![Tsachi\_Shuval](https://avatars.discourse-cdn.com/v4/letter/t/5e9695/32.png) [@Tsachi\_Shuval](https://discourse.nameko.io/u/Tsachi_Shuval)\
**Post date:** [April 12, 2016, 11:40am UTC](https://discourse.nameko.io/t/making-nameko-not-stop-when-runtime-errors-occur-in-dependencies/128/9 "2016-04-12T11:40:31Z")

</div>

I understand that crashing during a setup phase is unavoidable, and this is  
totally acceptable.  
This is why I was referring specifically to the get\_dependency method which  
is message-bound, and not service-bound.

> **···**
>
> On Tuesday, April 12, 2016 at 2:37:02 PM UTC+3, David Szotten wrote:
> 
> > Just to be clear, the situation isn't quite that dire. Nameko makes a  
> > distinction between service code and "framework/plumbing" (core +  
> > extensions) code. Uncaught exceptions in service code \_are\_ caught by the  
> > framework and result in error responses
> > 
> > This discussion is about the possibility of changing the behaviour of  
> > exceptions in extensions and elsewhere, e.g. from get\_dependency, and other  
> > places that are tied to a worker lifecycle from the former to the latter  
> > behaviour. For certain errors i do think crashing is the correct behaviour,  
> > e.g. during a "setup" phase. E.g. if my app can't resolve the hostname of  
> > my db server, i want it to crash at startup, not fail when serving its  
> > first (and all subsequent) requests
> > 
> > d
> > 
> > On Tuesday, 12 April 2016 08:49:18 UTC+1, tsachi...@gmail.com wrote:
> > 
> > > Hi,
> > > 
> > > I agree with you guys that there is no alternative to good coding  
> > > practices, exception handling, input sanitation and process monitoring.  
> > > Having said that, from the discussion so far, it looks like crashing the  
> > > server is almost by design!? "if there's an error in X then nameko will  
> > > \*crash\*"
> > > 
> > > Of course IO operations in the dependency will generally be  
> > > try..except'd, but imagine a calculation block which happens to throw a  
> > > division by zero. Or maybe I'm relying on an external package (openCV, PIL,  
> > > whatever) that threw an error due to bad arguments passed to one of its  
> > > functions.  
> > > Should the service crash in such cases?
> > > 
> > > IMO, nameko should catch these errors, raise exception or return an  
> > > 'error' response, but never crash.  
> > > Of course I am not involved in this project, so I can't tell if it's  
> > > doable (conceptually, practically, etc) and I don't know the impact of  
> > > implementing it.
> > > 
> > > Anyway, thank you guys for discussing this subject. Would love to hear  
> > > your thoughts and any updates.
> > > 
> > > Best  
> > > T
> > > 
> > > On Monday, April 11, 2016 at 11:51:33 PM UTC+3, David Szotten wrote:
> > > 
> > > > Hi,
> > > > 
> > > > Describing the behaviour and our reasons above, the existing behaviour  
> > > > does feel a bit unsatisfactory.
> > > > 
> > > > Rather than crashing, it might make sense for calls to return some kind  
> > > > of "server error" response (a la 500 errors in http).
> > > > 
> > > > While nameko can't "protect" against dependencies failing or  
> > > > unserialisable responses, we could do something other than crash the  
> > > > process. It doesn't seem unreasonable to me to expect the runner to survive  
> > > > such things and "use a supervisor" (as was my original advice) seems like a  
> > > > bit of a cop out.
> > > > 
> > > > I haven't thought about all implications (e.g. would "extension errors"  
> > > > ack or nack amqp messages? does each entrypoint decide?) nor about how  
> > > > difficult this would be to implement, nor about backwards compat. but the  
> > > > general idea might be something we want to think about
> > > > 
> > > > best,  
> > > > d
> > > > 
> > > > On Sunday, 10 April 2016 21:26:22 UTC+1, Matt Bennett wrote:
> > > > 
> > > > > This behaviour is intentional, but I can understand why it’s confusing.
> > > > > 
> > > > > As David explained, Nameko can’t tell whether errors are transient or  
> > > > > fatal, so it does the “safe” thing and assumes the error is unrecoverable.  
> > > > > The only time this isn’t true is during the worker lifecycle (i.e. when  
> > > > > executing a service method) because an error there can be attributed to the  
> > > > > active worker.
> > > > > 
> > > > > There is a valid argument that errors specifically in `get_dependency`  
> > > > > are also tied to a particular worker. It might be more intuitive to catch  
> > > > > them and return an appropriate error to the caller (if there is one). The  
> > > > > caller could then retry if appropriate, and the service would stay up in  
> > > > > the mean time. This also doesn’t preclude a DependencyProvider from  
> > > > > catching exceptions and retrying itself, if it wanted to hide transient  
> > > > > errors from the caller.
> > > > > 
> > > > > However, it wouldn't change the fact that users should “expect  
> > > > > failures” and run Nameko under a process manager though. The Nameko process  
> > > > > will still exit, for example, if an entrypoint can’t serialise the result  
> > > > > of a service method and hasn’t been made appropriately robust. Unlike  
> > > > > `get_dependency` there’s nothing we can do in Nameko core to protect  
> > > > > against that kind of error.
> > > > > 
> > > > > What do you think David?
> > > > > 
> > > > > On Thursday, April 7, 2016 at 1:30:42 PM UTC+1, David Szotten wrote:
> > > > > 
> > > > > > Hi,
> > > > > > 
> > > > > > Nameko can't tell the difference between e.g. connection error  
> > > > > > (recoverable) and say configuration error (e.g. bad mongo hostname, not  
> > > > > > recoverable). What was the nature of the mongo exception in your case? if  
> > > > > > you occasionally fail to connect due to temporary errors, you might want to  
> > > > > > add retry behaviour inside your dependency.
> > > > > > 
> > > > > > We tended to find that we get better mileage from running nameko using  
> > > > > > a supervisor process (onefinestay uses supervisord), and letting that  
> > > > > > restart the whole process on that type of error
> > > > > > 
> > > > > > D
> > > > > > 
> > > > > > On Thursday, 7 April 2016 13:17:12 UTC+1, tsachi...@gmail.com wrote:
> > > > > > 
> > > > > > > Hi David,
> > > > > > > 
> > > > > > > Thanks for the prompt reply.
> > > > > > > 
> > > > > > > There is no special use case. It happened to me while my Mongo DB  
> > > > > > > threw an exception (the get\_dependency returns a Mongo DB connection  
> > > > > > > object).  
> > > > > > > I agree that these kind of operations (IO, etc) should be  
> > > > > > > try..except'ed, but still, since nameko is a core service which other  
> > > > > > > components rely upon, as far as I'm concerned it should "never fail".
> > > > > > > 
> > > > > > > Imagine that Tornado/Apache/nginx/RabbitMQ/etc simply stop due to an  
> > > > > > > internal error. This is unacceptable.
> > > > > > > 
> > > > > > > IMHO, if a dependency throws an error, the current worker context  
> > > > > > > should quit, but the service should still be up and running.  
> > > > > > > This error maybe temporary (e.g. timeout, malformed input data to the  
> > > > > > > worker-\>injected dependency, etc), so why fail completely?
> > > > > > > 
> > > > > > > Don't you agree?
> > > > > > > 
> > > > > > > On Thursday, April 7, 2016 at 3:05:51 PM UTC+3, David Szotten wrote:
> > > > > > > 
> > > > > > > > Hi,
> > > > > > > > 
> > > > > > > > `get_dependency` is responsible for creating the dependency to be  
> > > > > > > > injected into the service. if this fails, the service won't have its  
> > > > > > > > dependency (which it "depends on") and the implication is that the service  
> > > > > > > > can't run correctly, so the container terminates.
> > > > > > > > 
> > > > > > > > what would be your preferred outcome? the service running without  
> > > > > > > > that particular dependency? perhaps you could provide some more details  
> > > > > > > > around your particular use-case to better help me understand
> > > > > > > > 
> > > > > > > > Best,  
> > > > > > > > David
> > > > > > > > 
> > > > > > > > On Thursday, 7 April 2016 12:45:53 UTC+1, tsachi...@gmail.com wrote:
> > > > > > > > 
> > > > > > > > > Hi,
> > > > > > > > > 
> > > > > > > > > Seems like a runtime error in a dependency stops nameko (you can  
> > > > > > > > > "raise" somethind in a get\_dependency() method to test it).
> > > > > > > > > 
> > > > > > > > > This is a very BAD scenario.  
> > > > > > > > > So, other than try..except the code block, is there a way to  
> > > > > > > > > prevent nameko from stopping?
> > > > > > > > > 
> > > > > > > > > Thanks.