Hi devs
We''re experiencing hanging services in production. Previously I've used
the eventlet backdoor to telnet in and inspect what workers are in flight
and what they are executing, but I cannot remember the steps to do this and
it doesn't seem to be documented - at least not outside of the nameko test
suite.
Can you take us through the steps to do this please? starting from defining
the port in our config, telnetting in and then inspecting the live objects
to find what workers are executing what and with what call args? We're
using the HTTP entrypoints if that is important.
Thanks
Are you running your services with `nameko run`?
nameko run --help
usage: nameko run [-h] [--config CONFIG] [--broker BROKER]
[*--backdoor-port BACKDOOR_PORT*]
module[:service class] [module[:service class] ...]
Run nameko services. Given a python path to a module containing one or more
nameko services, will host and run them. By default this will try to find
classes that look like services (anything with nameko entrypoints), but a
specific service can be specified via ``nameko run module:ServiceClass``.
positional arguments:
module[:service class]
python path to one or more service classes to run
optional arguments:
-h, --help show this help message and exit
--config CONFIG The YAML configuration file
--broker BROKER RabbitMQ broker url
* --backdoor-port BACKDOOR_PORT*
* Specify a port number to host a backdoor, which
can be*
* connected to for an interactive interpreter within
the*
* running service process using `nameko backdoor`.*
if not, have a look at the source for the same
Best,
David
···
On Wednesday, 5 April 2017 10:22:06 UTC+1, simon harrison wrote:
Hi devs
We''re experiencing hanging services in production. Previously I've used
the eventlet backdoor to telnet in and inspect what workers are in flight
and what they are executing, but I cannot remember the steps to do this and
it doesn't seem to be documented - at least not outside of the nameko test
suite.
Can you take us through the steps to do this please? starting from
defining the port in our config, telnetting in and then inspecting the live
objects to find what workers are executing what and with what call args?
We're using the HTTP entrypoints if that is important.
Thanks
Hi David
Yes we are, and we are now specifying a backdoor port.
···
On Wednesday, 5 April 2017 10:26:57 UTC+1, David Szotten wrote:
Are you running your services with `nameko run`?
nameko run --help
usage: nameko run [-h] [--config CONFIG] [--broker BROKER]
[*--backdoor-port BACKDOOR_PORT*]
module[:service class] [module[:service class] ...]
Run nameko services. Given a python path to a module containing one or more
nameko services, will host and run them. By default this will try to find
classes that look like services (anything with nameko entrypoints), but a
specific service can be specified via ``nameko run module:ServiceClass``.
positional arguments:
module[:service class]
python path to one or more service classes to run
optional arguments:
-h, --help show this help message and exit
--config CONFIG The YAML configuration file
--broker BROKER RabbitMQ broker url
* --backdoor-port BACKDOOR_PORT*
* Specify a port number to host a backdoor, which
can be*
* connected to for an interactive interpreter
within the*
* running service process using `nameko backdoor`.*
if not, have a look at the source for the same
Best,
David
On Wednesday, 5 April 2017 10:22:06 UTC+1, simon harrison wrote:
Hi devs
We''re experiencing hanging services in production. Previously I've used
the eventlet backdoor to telnet in and inspect what workers are in flight
and what they are executing, but I cannot remember the steps to do this and
it doesn't seem to be documented - at least not outside of the nameko test
suite.
Can you take us through the steps to do this please? starting from
defining the port in our config, telnetting in and then inspecting the live
objects to find what workers are executing what and with what call args?
We're using the HTTP entrypoints if that is important.
Thanks
Hi,
I'm afraid i can't tell if your problem is solved or if you are looking for
additional info/advice?
Best,
David
···
On Wednesday, 5 April 2017 11:08:39 UTC+1, simon harrison wrote:
Hi David
Yes we are, and we are now specifying a backdoor port.
On Wednesday, 5 April 2017 10:26:57 UTC+1, David Szotten wrote:
Are you running your services with `nameko run`?
nameko run --help
usage: nameko run [-h] [--config CONFIG] [--broker BROKER]
[*--backdoor-port BACKDOOR_PORT*]
module[:service class] [module[:service class] ...]
Run nameko services. Given a python path to a module containing one or
more
nameko services, will host and run them. By default this will try to find
classes that look like services (anything with nameko entrypoints), but a
specific service can be specified via ``nameko run module:ServiceClass``.
positional arguments:
module[:service class]
python path to one or more service classes to run
optional arguments:
-h, --help show this help message and exit
--config CONFIG The YAML configuration file
--broker BROKER RabbitMQ broker url
* --backdoor-port BACKDOOR_PORT*
* Specify a port number to host a backdoor, which
can be*
* connected to for an interactive interpreter
within the*
* running service process using `nameko backdoor`.*
if not, have a look at the source for the same
Best,
David
On Wednesday, 5 April 2017 10:22:06 UTC+1, simon harrison wrote:
Hi devs
We''re experiencing hanging services in production. Previously I've used
the eventlet backdoor to telnet in and inspect what workers are in flight
and what they are executing, but I cannot remember the steps to do this and
it doesn't seem to be documented - at least not outside of the nameko test
suite.
Can you take us through the steps to do this please? starting from
defining the port in our config, telnetting in and then inspecting the live
objects to find what workers are executing what and with what call args?
We're using the HTTP entrypoints if that is important.
Thanks
What's the trick to get the python interpreter to behave?
``apt-get install libreadline`` doesn't help.
Then we need to find the workers from the runner object.
···
On Wednesday, 5 April 2017 11:10:21 UTC+1, David Szotten wrote:
Hi,
I'm afraid i can't tell if your problem is solved or if you are looking
for additional info/advice?
Best,
David
On Wednesday, 5 April 2017 11:08:39 UTC+1, simon harrison wrote:
Hi David
Yes we are, and we are now specifying a backdoor port.
On Wednesday, 5 April 2017 10:26:57 UTC+1, David Szotten wrote:
Are you running your services with `nameko run`?
nameko run --help
usage: nameko run [-h] [--config CONFIG] [--broker BROKER]
[*--backdoor-port BACKDOOR_PORT*]
module[:service class] [module[:service class] ...]
Run nameko services. Given a python path to a module containing one or
more
nameko services, will host and run them. By default this will try to find
classes that look like services (anything with nameko entrypoints), but a
specific service can be specified via ``nameko run module:ServiceClass``.
positional arguments:
module[:service class]
python path to one or more service classes to run
optional arguments:
-h, --help show this help message and exit
--config CONFIG The YAML configuration file
--broker BROKER RabbitMQ broker url
* --backdoor-port BACKDOOR_PORT*
* Specify a port number to host a backdoor, which
can be*
* connected to for an interactive interpreter
within the*
* running service process using `nameko
backdoor`.*
if not, have a look at the source for the same
Best,
David
On Wednesday, 5 April 2017 10:22:06 UTC+1, simon harrison wrote:
Hi devs
We''re experiencing hanging services in production. Previously I've
used the eventlet backdoor to telnet in and inspect what workers are in
flight and what they are executing, but I cannot remember the steps to do
this and it doesn't seem to be documented - at least not outside of the
nameko test suite.
Can you take us through the steps to do this please? starting from
defining the port in our config, telnetting in and then inspecting the live
objects to find what workers are executing what and with what call args?
We're using the HTTP entrypoints if that is important.
Thanks
the runner should be available (injected into
locals): https://github.com/nameko/nameko/blob/master/nameko/cli/run.py#L106
you're a bit limited since going via telnet/netcat, but rlwrap helps a bit
d
···
On Wednesday, 5 April 2017 11:46:37 UTC+1, simon harrison wrote:
What's the trick to get the python interpreter to behave?
``apt-get install libreadline`` doesn't help.
Then we need to find the workers from the runner object.
On Wednesday, 5 April 2017 11:10:21 UTC+1, David Szotten wrote:
Hi,
I'm afraid i can't tell if your problem is solved or if you are looking
for additional info/advice?
Best,
David
On Wednesday, 5 April 2017 11:08:39 UTC+1, simon harrison wrote:
Hi David
Yes we are, and we are now specifying a backdoor port.
On Wednesday, 5 April 2017 10:26:57 UTC+1, David Szotten wrote:
Are you running your services with `nameko run`?
nameko run --help
usage: nameko run [-h] [--config CONFIG] [--broker BROKER]
[*--backdoor-port BACKDOOR_PORT*]
module[:service class] [module[:service class] ...]
Run nameko services. Given a python path to a module containing one or
more
nameko services, will host and run them. By default this will try to
find
classes that look like services (anything with nameko entrypoints), but
a
specific service can be specified via ``nameko run
module:ServiceClass``.
positional arguments:
module[:service class]
python path to one or more service classes to
run
optional arguments:
-h, --help show this help message and exit
--config CONFIG The YAML configuration file
--broker BROKER RabbitMQ broker url
* --backdoor-port BACKDOOR_PORT*
* Specify a port number to host a backdoor,
which can be*
* connected to for an interactive interpreter
within the*
* running service process using `nameko
backdoor`.*
if not, have a look at the source for the same
Best,
David
On Wednesday, 5 April 2017 10:22:06 UTC+1, simon harrison wrote:
Hi devs
We''re experiencing hanging services in production. Previously I've
used the eventlet backdoor to telnet in and inspect what workers are in
flight and what they are executing, but I cannot remember the steps to do
this and it doesn't seem to be documented - at least not outside of the
nameko test suite.
Can you take us through the steps to do this please? starting from
defining the port in our config, telnetting in and then inspecting the live
objects to find what workers are executing what and with what call args?
We're using the HTTP entrypoints if that is important.
Thanks
Cool. Making progress. Thanks for the help here.
We can get the container and then see ``_worker_threads``.
How can we see what entrypoint is being worked on by a worker? Is that even
possible? Because that's what we're after here.
Thanks again!
···
On Wednesday, 5 April 2017 11:48:55 UTC+1, David Szotten wrote:
the runner should be available (injected into locals):
https://github.com/nameko/nameko/blob/master/nameko/cli/run.py#L106
you're a bit limited since going via telnet/netcat, but rlwrap helps a bit
d
On Wednesday, 5 April 2017 11:46:37 UTC+1, simon harrison wrote:
What's the trick to get the python interpreter to behave?
``apt-get install libreadline`` doesn't help.
Then we need to find the workers from the runner object.
On Wednesday, 5 April 2017 11:10:21 UTC+1, David Szotten wrote:
Hi,
I'm afraid i can't tell if your problem is solved or if you are looking
for additional info/advice?
Best,
David
On Wednesday, 5 April 2017 11:08:39 UTC+1, simon harrison wrote:
Hi David
Yes we are, and we are now specifying a backdoor port.
On Wednesday, 5 April 2017 10:26:57 UTC+1, David Szotten wrote:
Are you running your services with `nameko run`?
nameko run --help
usage: nameko run [-h] [--config CONFIG] [--broker BROKER]
[*--backdoor-port BACKDOOR_PORT*]
module[:service class] [module[:service class] ...]
Run nameko services. Given a python path to a module containing one or
more
nameko services, will host and run them. By default this will try to
find
classes that look like services (anything with nameko entrypoints),
but a
specific service can be specified via ``nameko run
module:ServiceClass``.
positional arguments:
module[:service class]
python path to one or more service classes to
run
optional arguments:
-h, --help show this help message and exit
--config CONFIG The YAML configuration file
--broker BROKER RabbitMQ broker url
* --backdoor-port BACKDOOR_PORT*
* Specify a port number to host a backdoor,
which can be*
* connected to for an interactive interpreter
within the*
* running service process using `nameko
backdoor`.*
if not, have a look at the source for the same
Best,
David
On Wednesday, 5 April 2017 10:22:06 UTC+1, simon harrison wrote:
Hi devs
We''re experiencing hanging services in production. Previously I've
used the eventlet backdoor to telnet in and inspect what workers are in
flight and what they are executing, but I cannot remember the steps to do
this and it doesn't seem to be documented - at least not outside of the
nameko test suite.
Can you take us through the steps to do this please? starting from
defining the port in our config, telnetting in and then inspecting the live
objects to find what workers are executing what and with what call args?
We're using the HTTP entrypoints if that is important.
Thanks
_worker_threads is a dict of greenthreads keyed by the worker context for
each call. Inspecting the worker context will tell you everything about
that worker -- the entrypoint that fired, its arguments, context data etc.
Another tip is that you can inspect the greenthread itself to see what it's
currently doing. This is useful if your workers are hanging. Example:
ct = list(runner.containers)[0] # first container
gt = list(ct._worker_threads.values())[0] # first worker greenthread
import traceback
traceback.print_stack(gt.gr_frame) # print current stack
You will find the thread sitting waiting for some I/O (it has to be in
order for your thread to be running). If the worker is hung it'll be the
same stack every time you check.
···
On Wednesday, April 5, 2017 at 12:04:24 PM UTC+1, simon harrison wrote:
Cool. Making progress. Thanks for the help here.
We can get the container and then see ``_worker_threads``.
How can we see what entrypoint is being worked on by a worker? Is that
even possible? Because that's what we're after here.
Thanks again!
On Wednesday, 5 April 2017 11:48:55 UTC+1, David Szotten wrote:
the runner should be available (injected into locals):
https://github.com/nameko/nameko/blob/master/nameko/cli/run.py#L106
you're a bit limited since going via telnet/netcat, but rlwrap helps a bit
d
On Wednesday, 5 April 2017 11:46:37 UTC+1, simon harrison wrote:
What's the trick to get the python interpreter to behave?
``apt-get install libreadline`` doesn't help.
Then we need to find the workers from the runner object.
On Wednesday, 5 April 2017 11:10:21 UTC+1, David Szotten wrote:
Hi,
I'm afraid i can't tell if your problem is solved or if you are looking
for additional info/advice?
Best,
David
On Wednesday, 5 April 2017 11:08:39 UTC+1, simon harrison wrote:
Hi David
Yes we are, and we are now specifying a backdoor port.
On Wednesday, 5 April 2017 10:26:57 UTC+1, David Szotten wrote:
Are you running your services with `nameko run`?
nameko run --help
usage: nameko run [-h] [--config CONFIG] [--broker BROKER]
[*--backdoor-port BACKDOOR_PORT*]
module[:service class] [module[:service class] ...]
Run nameko services. Given a python path to a module containing one
or more
nameko services, will host and run them. By default this will try to
find
classes that look like services (anything with nameko entrypoints),
but a
specific service can be specified via ``nameko run
module:ServiceClass``.
positional arguments:
module[:service class]
python path to one or more service classes to
run
optional arguments:
-h, --help show this help message and exit
--config CONFIG The YAML configuration file
--broker BROKER RabbitMQ broker url
* --backdoor-port BACKDOOR_PORT*
* Specify a port number to host a backdoor,
which can be*
* connected to for an interactive interpreter
within the*
* running service process using `nameko
backdoor`.*
if not, have a look at the source for the same
Best,
David
On Wednesday, 5 April 2017 10:22:06 UTC+1, simon harrison wrote:
Hi devs
We''re experiencing hanging services in production. Previously I've
used the eventlet backdoor to telnet in and inspect what workers are in
flight and what they are executing, but I cannot remember the steps to do
this and it doesn't seem to be documented - at least not outside of the
nameko test suite.
Can you take us through the steps to do this please? starting from
defining the port in our config, telnetting in and then inspecting the live
objects to find what workers are executing what and with what call args?
We're using the HTTP entrypoints if that is important.
Thanks
perfect! thanks chaps.
this sort of debugging know-how could find themselves a place in the docs?
how would you feel about a Troubleshooting section?
···
On Wednesday, 5 April 2017 12:18:38 UTC+1, Matt Yule-Bennett wrote:
_worker_threads is a dict of greenthreads keyed by the worker context for
each call. Inspecting the worker context will tell you everything about
that worker -- the entrypoint that fired, its arguments, context data etc.
Another tip is that you can inspect the greenthread itself to see what
it's currently doing. This is useful if your workers are hanging. Example:
ct = list(runner.containers)[0] # first container
gt = list(ct._worker_threads.values())[0] # first worker greenthread
import traceback
traceback.print_stack(gt.gr_frame) # print current stack
You will find the thread sitting waiting for some I/O (it has to be in
order for your thread to be running). If the worker is hung it'll be the
same stack every time you check.
On Wednesday, April 5, 2017 at 12:04:24 PM UTC+1, simon harrison wrote:
Cool. Making progress. Thanks for the help here.
We can get the container and then see ``_worker_threads``.
How can we see what entrypoint is being worked on by a worker? Is that
even possible? Because that's what we're after here.
Thanks again!
On Wednesday, 5 April 2017 11:48:55 UTC+1, David Szotten wrote:
the runner should be available (injected into locals):
https://github.com/nameko/nameko/blob/master/nameko/cli/run.py#L106
you're a bit limited since going via telnet/netcat, but rlwrap helps a
bit
d
On Wednesday, 5 April 2017 11:46:37 UTC+1, simon harrison wrote:
What's the trick to get the python interpreter to behave?
``apt-get install libreadline`` doesn't help.
Then we need to find the workers from the runner object.
On Wednesday, 5 April 2017 11:10:21 UTC+1, David Szotten wrote:
Hi,
I'm afraid i can't tell if your problem is solved or if you are
looking for additional info/advice?
Best,
David
On Wednesday, 5 April 2017 11:08:39 UTC+1, simon harrison wrote:
Hi David
Yes we are, and we are now specifying a backdoor port.
On Wednesday, 5 April 2017 10:26:57 UTC+1, David Szotten wrote:
Are you running your services with `nameko run`?
nameko run --help
usage: nameko run [-h] [--config CONFIG] [--broker BROKER]
[*--backdoor-port BACKDOOR_PORT*]
module[:service class] [module[:service class] ...]
Run nameko services. Given a python path to a module containing one
or more
nameko services, will host and run them. By default this will try to
find
classes that look like services (anything with nameko entrypoints),
but a
specific service can be specified via ``nameko run
module:ServiceClass``.
positional arguments:
module[:service class]
python path to one or more service classes
to run
optional arguments:
-h, --help show this help message and exit
--config CONFIG The YAML configuration file
--broker BROKER RabbitMQ broker url
* --backdoor-port BACKDOOR_PORT*
* Specify a port number to host a backdoor,
which can be*
* connected to for an interactive interpreter
within the*
* running service process using `nameko
backdoor`.*
if not, have a look at the source for the same
Best,
David
On Wednesday, 5 April 2017 10:22:06 UTC+1, simon harrison wrote:
Hi devs
We''re experiencing hanging services in production. Previously I've
used the eventlet backdoor to telnet in and inspect what workers are in
flight and what they are executing, but I cannot remember the steps to do
this and it doesn't seem to be documented - at least not outside of the
nameko test suite.
Can you take us through the steps to do this please? starting from
defining the port in our config, telnetting in and then inspecting the live
objects to find what workers are executing what and with what call args?
We're using the HTTP entrypoints if that is important.
Thanks
pull requests to the docs are always very welcome!
d
···
On Wednesday, 5 April 2017 12:54:47 UTC+1, simon harrison wrote:
perfect! thanks chaps.
this sort of debugging know-how could find themselves a place in the docs?
how would you feel about a Troubleshooting section?
On Wednesday, 5 April 2017 12:18:38 UTC+1, Matt Yule-Bennett wrote:
_worker_threads is a dict of greenthreads keyed by the worker context for
each call. Inspecting the worker context will tell you everything about
that worker -- the entrypoint that fired, its arguments, context data etc.
Another tip is that you can inspect the greenthread itself to see what
it's currently doing. This is useful if your workers are hanging. Example:
ct = list(runner.containers)[0] # first container
gt = list(ct._worker_threads.values())[0] # first worker greenthread
import traceback
traceback.print_stack(gt.gr_frame) # print current stack
You will find the thread sitting waiting for some I/O (it has to be in
order for your thread to be running). If the worker is hung it'll be the
same stack every time you check.
On Wednesday, April 5, 2017 at 12:04:24 PM UTC+1, simon harrison wrote:
Cool. Making progress. Thanks for the help here.
We can get the container and then see ``_worker_threads``.
How can we see what entrypoint is being worked on by a worker? Is that
even possible? Because that's what we're after here.
Thanks again!
On Wednesday, 5 April 2017 11:48:55 UTC+1, David Szotten wrote:
the runner should be available (injected into locals):
https://github.com/nameko/nameko/blob/master/nameko/cli/run.py#L106
you're a bit limited since going via telnet/netcat, but rlwrap helps a
bit
d
On Wednesday, 5 April 2017 11:46:37 UTC+1, simon harrison wrote:
What's the trick to get the python interpreter to behave?
``apt-get install libreadline`` doesn't help.
Then we need to find the workers from the runner object.
On Wednesday, 5 April 2017 11:10:21 UTC+1, David Szotten wrote:
Hi,
I'm afraid i can't tell if your problem is solved or if you are
looking for additional info/advice?
Best,
David
On Wednesday, 5 April 2017 11:08:39 UTC+1, simon harrison wrote:
Hi David
Yes we are, and we are now specifying a backdoor port.
On Wednesday, 5 April 2017 10:26:57 UTC+1, David Szotten wrote:
Are you running your services with `nameko run`?
nameko run --help
usage: nameko run [-h] [--config CONFIG] [--broker BROKER]
[*--backdoor-port BACKDOOR_PORT*]
module[:service class] [module[:service class]
...]
Run nameko services. Given a python path to a module containing one
or more
nameko services, will host and run them. By default this will try
to find
classes that look like services (anything with nameko entrypoints),
but a
specific service can be specified via ``nameko run
module:ServiceClass``.
positional arguments:
module[:service class]
python path to one or more service classes
to run
optional arguments:
-h, --help show this help message and exit
--config CONFIG The YAML configuration file
--broker BROKER RabbitMQ broker url
* --backdoor-port BACKDOOR_PORT*
* Specify a port number to host a backdoor,
which can be*
* connected to for an interactive
interpreter within the*
* running service process using `nameko
backdoor`.*
if not, have a look at the source for the same
Best,
David
On Wednesday, 5 April 2017 10:22:06 UTC+1, simon harrison wrote:
Hi devs
We''re experiencing hanging services in production. Previously
I've used the eventlet backdoor to telnet in and inspect what workers are
in flight and what they are executing, but I cannot remember the steps to
do this and it doesn't seem to be documented - at least not outside of the
nameko test suite.
Can you take us through the steps to do this please? starting from
defining the port in our config, telnetting in and then inspecting the live
objects to find what workers are executing what and with what call args?
We're using the HTTP entrypoints if that is important.
Thanks