Hi! We're writing tests for our services that test their high level
functionality using *worker_factory* and *container_factory*. When we use
*worker_factory*, all the service's dependencies are replaced with mock
dependencies as specified in the docstring
<https://github.com/onefinestay/nameko/blob/master/nameko/testing/services.py#L141>.
When we use *container_factory*, we are manually mocking all services using
the replace_dependencies helper
<https://github.com/onefinestay/nameko/blob/master/nameko/testing/services.py#L229>,
which replaces the dependencies with mocks. Many of these are *RpcProxy* dependencies
that communicate with other services, but given the specified functionality
of *worker_factory* and *replace_dependencies, *we expect these rpc-level
dependencies to be mocked and for there to be no need for an active
RabbitMQ cluster.
*Unfortunately, even though all of our dependencies are mocked, when we try
to run our tests, the tests error out on container.start() with a failed
connection to RabbitMQ.*
Digging into the code a bit, I believe I've found the reason, but I wanted
to make sure I'm understanding.
I believe that while the *dependencies* are being replaced with mocks, *subextensions
*added by those dependencies are not being replaced. The subextension
addition logic happens in the container *__init__*
<https://github.com/onefinestay/nameko/blob/master/nameko/containers.py#L160>,
which happens before both *worker_factory* and *replace_dependencies* replace
the *RpcProxy* dependencies with mocks. Therefore, *RpcProxy* dependencies
add a QueueConsumer
<https://github.com/onefinestay/nameko/blob/master/nameko/rpc.py#L40> dependencies
to *container.subextensions* which triggers the RabbitMQ connection.
For now, I've been able to work around this by manually overriding the
class dependencies before I use *container_factory, *like this*:*
ExampleService.rpc_dependency_1 = Mock()
ExampleService.rpc_dependency_2 = Mock()
container = container_factory(ExampleService, web_config)
container.start()
However, this seems suboptimal.
Has anyone dealt with this before?
Hi
Could you give some more detail about why/when you are using the
container_factory over the worker_factory? If you want everything mocked
out it sounds like the worker_factory (which won't need rabbitmq) is a
better choice. The container factory is for when you want to simulate more
realistic conditions, and so defaults to hosting real versions of everything
If you need only a few select dependencies, say a db connection, one option
is to still use the worker_factory, and manually inject the db, as
in https://github.com/onefinestay/nameko/blob/master/docs/examples/testing/alternative_dependency_unit_test.py#L52
Best,
David
···
On Wednesday, 10 August 2016 08:21:48 UTC+1, je...@pollak.io wrote:
Hi! We're writing tests for our services that test their high level
functionality using *worker_factory* and *container_factory*. When we use
*worker_factory*, all the service's dependencies are replaced with mock
dependencies as specified in the docstring
<https://github.com/onefinestay/nameko/blob/master/nameko/testing/services.py#L141>\.
When we use *container_factory*, we are manually mocking all services
using the replace_dependencies helper
<https://github.com/onefinestay/nameko/blob/master/nameko/testing/services.py#L229>,
which replaces the dependencies with mocks. Many of these are *RpcProxy* dependencies
that communicate with other services, but given the specified functionality
of *worker_factory* and *replace_dependencies, *we expect these rpc-level
dependencies to be mocked and for there to be no need for an active
RabbitMQ cluster.
*Unfortunately, even though all of our dependencies are mocked, when we
try to run our tests, the tests error out on container.start() with a
failed connection to RabbitMQ.*
Digging into the code a bit, I believe I've found the reason, but I wanted
to make sure I'm understanding.
I believe that while the *dependencies* are being replaced with mocks, *subextensions
*added by those dependencies are not being replaced. The subextension
addition logic happens in the container *__init__*
<https://github.com/onefinestay/nameko/blob/master/nameko/containers.py#L160>,
which happens before both *worker_factory* and *replace_dependencies* replace
the *RpcProxy* dependencies with mocks. Therefore, *RpcProxy* dependencies
add a QueueConsumer
<https://github.com/onefinestay/nameko/blob/master/nameko/rpc.py#L40> dependencies
to *container.subextensions* which triggers the RabbitMQ connection.
For now, I've been able to work around this by manually overriding the
class dependencies before I use *container_factory, *like this*:*
ExampleService.rpc_dependency_1 = Mock()
ExampleService.rpc_dependency_2 = Mock()
container = container_factory(ExampleService, web_config)
container.start()
However, this seems suboptimal.
Has anyone dealt with this before?
Right now, we're primarily using container_factory to test HTTP responses
from HTTP services. In particular, we're creating a customer
HTTPRequestHandler that processes exceptions and transforms them into
standard responses and are testing that this works correctly. Is there a
way to do HTTP-level testing with the worker_factory?
Thanks David!
Jesse
···
On Wednesday, August 10, 2016 at 1:21:13 AM UTC-7, David Szotten wrote:
Hi
Could you give some more detail about why/when you are using the
container_factory over the worker_factory? If you want everything mocked
out it sounds like the worker_factory (which won't need rabbitmq) is a
better choice. The container factory is for when you want to simulate more
realistic conditions, and so defaults to hosting real versions of everything
If you need only a few select dependencies, say a db connection, one
option is to still use the worker_factory, and manually inject the db, as
in
https://github.com/onefinestay/nameko/blob/master/docs/examples/testing/alternative_dependency_unit_test.py#L52
Best,
David
On Wednesday, 10 August 2016 08:21:48 UTC+1, je...@pollak.io wrote:
Hi! We're writing tests for our services that test their high level
functionality using *worker_factory* and *container_factory*. When we
use *worker_factory*, all the service's dependencies are replaced with
mock dependencies as specified in the docstring
<https://github.com/onefinestay/nameko/blob/master/nameko/testing/services.py#L141>\.
When we use *container_factory*, we are manually mocking all services
using the replace_dependencies helper
<https://github.com/onefinestay/nameko/blob/master/nameko/testing/services.py#L229>,
which replaces the dependencies with mocks. Many of these are *RpcProxy* dependencies
that communicate with other services, but given the specified functionality
of *worker_factory* and *replace_dependencies, *we expect these
rpc-level dependencies to be mocked and for there to be no need for an
active RabbitMQ cluster.
*Unfortunately, even though all of our dependencies are mocked, when we
try to run our tests, the tests error out on container.start() with a
failed connection to RabbitMQ.*
Digging into the code a bit, I believe I've found the reason, but I
wanted to make sure I'm understanding.
I believe that while the *dependencies* are being replaced with mocks, *subextensions
*added by those dependencies are not being replaced. The subextension
addition logic happens in the container *__init__*
<https://github.com/onefinestay/nameko/blob/master/nameko/containers.py#L160>,
which happens before both *worker_factory* and *replace_dependencies* replace
the *RpcProxy* dependencies with mocks. Therefore, *RpcProxy* dependencies
add a QueueConsumer
<https://github.com/onefinestay/nameko/blob/master/nameko/rpc.py#L40> dependencies
to *container.subextensions* which triggers the RabbitMQ connection.
For now, I've been able to work around this by manually overriding the
class dependencies before I use *container_factory, *like this*:*
ExampleService.rpc_dependency_1 = Mock()
ExampleService.rpc_dependency_2 = Mock()
container = container_factory(ExampleService, web_config)
container.start()
However, this seems suboptimal.
Has anyone dealt with this before?
Ah, interesting!
Some of the core nameko developers recently met up to discuss roadmaps and
the eco system in general (notes will be posted to the mailing list once
written up). One of the issues we discussed was scaling back the extent to
which nameko assumes amqp. When nameko started out, there was no http
extension, and amqp was so core to everything at onefinestay that we
probably didn't consider the use case you brought up. we definitely need to
though!
i need to have a think about the best way to handle this, but in the mean
time, i threw together a workaround for
you: strip amqp dependencies from nameko class · GitHub
the core idea is a function `strip_rpc(service_class)` which dynamically
creates a subclass of the service with all rpc entrypoints and proxy
dependencies overridden by None.
the gist includes a test which uses the container_factory to host a
stripped version of the example service and successfully makes a http call
without any rabbit broker required
best,
David
···
On Wednesday, 10 August 2016 19:33:08 UTC+1, je...@pollak.io wrote:
Right now, we're primarily using container_factory to test HTTP responses
from HTTP services. In particular, we're creating a customer
HTTPRequestHandler that processes exceptions and transforms them into
standard responses and are testing that this works correctly. Is there a
way to do HTTP-level testing with the worker_factory?
Thanks David!
Jesse
On Wednesday, August 10, 2016 at 1:21:13 AM UTC-7, David Szotten wrote:
Hi
Could you give some more detail about why/when you are using the
container_factory over the worker_factory? If you want everything mocked
out it sounds like the worker_factory (which won't need rabbitmq) is a
better choice. The container factory is for when you want to simulate more
realistic conditions, and so defaults to hosting real versions of everything
If you need only a few select dependencies, say a db connection, one
option is to still use the worker_factory, and manually inject the db, as
in
https://github.com/onefinestay/nameko/blob/master/docs/examples/testing/alternative_dependency_unit_test.py#L52
Best,
David
On Wednesday, 10 August 2016 08:21:48 UTC+1, je...@pollak.io wrote:
Hi! We're writing tests for our services that test their high level
functionality using *worker_factory* and *container_factory*. When we
use *worker_factory*, all the service's dependencies are replaced with
mock dependencies as specified in the docstring
<https://github.com/onefinestay/nameko/blob/master/nameko/testing/services.py#L141>\.
When we use *container_factory*, we are manually mocking all services
using the replace_dependencies helper
<https://github.com/onefinestay/nameko/blob/master/nameko/testing/services.py#L229>,
which replaces the dependencies with mocks. Many of these are *RpcProxy* dependencies
that communicate with other services, but given the specified functionality
of *worker_factory* and *replace_dependencies, *we expect these
rpc-level dependencies to be mocked and for there to be no need for an
active RabbitMQ cluster.
*Unfortunately, even though all of our dependencies are mocked, when we
try to run our tests, the tests error out on container.start() with a
failed connection to RabbitMQ.*
Digging into the code a bit, I believe I've found the reason, but I
wanted to make sure I'm understanding.
I believe that while the *dependencies* are being replaced with mocks, *subextensions
*added by those dependencies are not being replaced. The subextension
addition logic happens in the container *__init__*
<https://github.com/onefinestay/nameko/blob/master/nameko/containers.py#L160>,
which happens before both *worker_factory* and *replace_dependencies* replace
the *RpcProxy* dependencies with mocks. Therefore, *RpcProxy* dependencies
add a QueueConsumer
<https://github.com/onefinestay/nameko/blob/master/nameko/rpc.py#L40> dependencies
to *container.subextensions* which triggers the RabbitMQ connection.
For now, I've been able to work around this by manually overriding the
class dependencies before I use *container_factory, *like this*:*
ExampleService.rpc_dependency_1 = Mock()
ExampleService.rpc_dependency_2 = Mock()
container = container_factory(ExampleService, web_config)
container.start()
However, this seems suboptimal.
Has anyone dealt with this before?
That makes total sense - and thanks for the gist! That's a much more
sophisticated way of doing what I was previously doing manually 
Just a note: we've started using nameko pretty heavily at Clef
<http://getclef.com> and are really enjoying it (Mark, Grace and I -- all
recent mailers -- all work there)! Thanks for all your hard work so far and
I hope we're able to contribute back going forward.
Jesse
···
On Wednesday, August 10, 2016 at 1:50:48 PM UTC-7, David Szotten wrote:
Ah, interesting!
Some of the core nameko developers recently met up to discuss roadmaps and
the eco system in general (notes will be posted to the mailing list once
written up). One of the issues we discussed was scaling back the extent to
which nameko assumes amqp. When nameko started out, there was no http
extension, and amqp was so core to everything at onefinestay that we
probably didn't consider the use case you brought up. we definitely need to
though!
i need to have a think about the best way to handle this, but in the mean
time, i threw together a workaround for you:
strip amqp dependencies from nameko class · GitHub
the core idea is a function `strip_rpc(service_class)` which dynamically
creates a subclass of the service with all rpc entrypoints and proxy
dependencies overridden by None.
the gist includes a test which uses the container_factory to host a
stripped version of the example service and successfully makes a http call
without any rabbit broker required
best,
David
On Wednesday, 10 August 2016 19:33:08 UTC+1, je...@pollak.io wrote:
Right now, we're primarily using container_factory to test HTTP responses
from HTTP services. In particular, we're creating a customer
HTTPRequestHandler that processes exceptions and transforms them into
standard responses and are testing that this works correctly. Is there a
way to do HTTP-level testing with the worker_factory?
Thanks David!
Jesse
On Wednesday, August 10, 2016 at 1:21:13 AM UTC-7, David Szotten wrote:
Hi
Could you give some more detail about why/when you are using the
container_factory over the worker_factory? If you want everything mocked
out it sounds like the worker_factory (which won't need rabbitmq) is a
better choice. The container factory is for when you want to simulate more
realistic conditions, and so defaults to hosting real versions of everything
If you need only a few select dependencies, say a db connection, one
option is to still use the worker_factory, and manually inject the db, as
in
https://github.com/onefinestay/nameko/blob/master/docs/examples/testing/alternative_dependency_unit_test.py#L52
Best,
David
On Wednesday, 10 August 2016 08:21:48 UTC+1, je...@pollak.io wrote:
Hi! We're writing tests for our services that test their high level
functionality using *worker_factory* and *container_factory*. When we
use *worker_factory*, all the service's dependencies are replaced with
mock dependencies as specified in the docstring
<https://github.com/onefinestay/nameko/blob/master/nameko/testing/services.py#L141>\.
When we use *container_factory*, we are manually mocking all services
using the replace_dependencies helper
<https://github.com/onefinestay/nameko/blob/master/nameko/testing/services.py#L229>,
which replaces the dependencies with mocks. Many of these are
*RpcProxy* dependencies that communicate with other services, but
given the specified functionality of *worker_factory* and *replace_dependencies,
*we expect these rpc-level dependencies to be mocked and for there to
be no need for an active RabbitMQ cluster.
*Unfortunately, even though all of our dependencies are mocked, when we
try to run our tests, the tests error out on container.start() with a
failed connection to RabbitMQ.*
Digging into the code a bit, I believe I've found the reason, but I
wanted to make sure I'm understanding.
I believe that while the *dependencies* are being replaced with mocks, *subextensions
*added by those dependencies are not being replaced. The subextension
addition logic happens in the container *__init__*
<https://github.com/onefinestay/nameko/blob/master/nameko/containers.py#L160>,
which happens before both *worker_factory* and *replace_dependencies* replace
the *RpcProxy* dependencies with mocks. Therefore, *RpcProxy* dependencies
add a QueueConsumer
<https://github.com/onefinestay/nameko/blob/master/nameko/rpc.py#L40> dependencies
to *container.subextensions* which triggers the RabbitMQ connection.
For now, I've been able to work around this by manually overriding the
class dependencies before I use *container_factory, *like this*:*
ExampleService.rpc_dependency_1 = Mock()
ExampleService.rpc_dependency_2 = Mock()
container = container_factory(ExampleService, web_config)
container.start()
However, this seems suboptimal.
Has anyone dealt with this before?
glad to help
the usual caveats about using undocumented internals apply. hopefully we'll
find a way to better support this use-case in the future
d
···
On Thursday, 11 August 2016 01:13:56 UTC+1, je...@pollak.io wrote:
That makes total sense - and thanks for the gist! That's a much more
sophisticated way of doing what I was previously doing manually
Just a note: we've started using nameko pretty heavily at Clef
<http://getclef.com> and are really enjoying it (Mark, Grace and I -- all
recent mailers -- all work there)! Thanks for all your hard work so far and
I hope we're able to contribute back going forward.
Jesse
On Wednesday, August 10, 2016 at 1:50:48 PM UTC-7, David Szotten wrote:
Ah, interesting!
Some of the core nameko developers recently met up to discuss roadmaps
and the eco system in general (notes will be posted to the mailing list
once written up). One of the issues we discussed was scaling back the
extent to which nameko assumes amqp. When nameko started out, there was no
http extension, and amqp was so core to everything at onefinestay that we
probably didn't consider the use case you brought up. we definitely need to
though!
i need to have a think about the best way to handle this, but in the mean
time, i threw together a workaround for you:
strip amqp dependencies from nameko class · GitHub
the core idea is a function `strip_rpc(service_class)` which dynamically
creates a subclass of the service with all rpc entrypoints and proxy
dependencies overridden by None.
the gist includes a test which uses the container_factory to host a
stripped version of the example service and successfully makes a http call
without any rabbit broker required
best,
David
On Wednesday, 10 August 2016 19:33:08 UTC+1, je...@pollak.io wrote:
Right now, we're primarily using container_factory to test HTTP
responses from HTTP services. In particular, we're creating a customer
HTTPRequestHandler that processes exceptions and transforms them into
standard responses and are testing that this works correctly. Is there a
way to do HTTP-level testing with the worker_factory?
Thanks David!
Jesse
On Wednesday, August 10, 2016 at 1:21:13 AM UTC-7, David Szotten wrote:
Hi
Could you give some more detail about why/when you are using the
container_factory over the worker_factory? If you want everything mocked
out it sounds like the worker_factory (which won't need rabbitmq) is a
better choice. The container factory is for when you want to simulate more
realistic conditions, and so defaults to hosting real versions of everything
If you need only a few select dependencies, say a db connection, one
option is to still use the worker_factory, and manually inject the db, as
in
https://github.com/onefinestay/nameko/blob/master/docs/examples/testing/alternative_dependency_unit_test.py#L52
Best,
David
On Wednesday, 10 August 2016 08:21:48 UTC+1, je...@pollak.io wrote:
Hi! We're writing tests for our services that test their high level
functionality using *worker_factory* and *container_factory*. When we
use *worker_factory*, all the service's dependencies are replaced
with mock dependencies as specified in the docstring
<https://github.com/onefinestay/nameko/blob/master/nameko/testing/services.py#L141>\.
When we use *container_factory*, we are manually mocking all services
using the replace_dependencies helper
<https://github.com/onefinestay/nameko/blob/master/nameko/testing/services.py#L229>,
which replaces the dependencies with mocks. Many of these are
*RpcProxy* dependencies that communicate with other services, but
given the specified functionality of *worker_factory* and *replace_dependencies,
*we expect these rpc-level dependencies to be mocked and for there to
be no need for an active RabbitMQ cluster.
*Unfortunately, even though all of our dependencies are mocked, when
we try to run our tests, the tests error out on container.start() with a
failed connection to RabbitMQ.*
Digging into the code a bit, I believe I've found the reason, but I
wanted to make sure I'm understanding.
I believe that while the *dependencies* are being replaced with
mocks, *subextensions *added by those dependencies are not being
replaced. The subextension addition logic happens in the container
*__init__*
<https://github.com/onefinestay/nameko/blob/master/nameko/containers.py#L160>,
which happens before both *worker_factory* and *replace_dependencies* replace
the *RpcProxy* dependencies with mocks. Therefore, *RpcProxy* dependencies
add a QueueConsumer
<https://github.com/onefinestay/nameko/blob/master/nameko/rpc.py#L40> dependencies
to *container.subextensions* which triggers the RabbitMQ connection.
For now, I've been able to work around this by manually overriding the
class dependencies before I use *container_factory, *like this*:*
ExampleService.rpc_dependency_1 = Mock()
ExampleService.rpc_dependency_2 = Mock()
container = container_factory(ExampleService, web_config)
container.start()
However, this seems suboptimal.
Has anyone dealt with this before?
Hi Jesse,
Your diagnosis is completely correct. `replace_dependencies` mocks out the
dependency providers only, and isn't clever enough to know about any
subextensions that they may have created.
This is probably an oversight in the way sub-extensions are declared. The
container discovers them by inspecting the "top-level" extensions, but then
treats them all the same. The knowledge of who declared them is lost. I
would rather `start()` and `stop()` only the top-level extensions and make
them responsible for the sub-extensions they declare; then
`replace_dependencies` could work in the way you expected. Unfortunately
this change isn't compatible with current implementation of the
QueueConsumer. The good news is that we also discussed refactoring this at
the recent in-person meetup. As promised I will write up these notes soon.
It's great to hear about nameko being used at Clef! Thanks for sharing and
I'm glad you're enjoying it
Matt.
···
On Thursday, August 11, 2016 at 2:38:32 PM UTC+8, David Szotten wrote:
glad to help
the usual caveats about using undocumented internals apply. hopefully
we'll find a way to better support this use-case in the future
d
On Thursday, 11 August 2016 01:13:56 UTC+1, je...@pollak.io wrote:
That makes total sense - and thanks for the gist! That's a much more
sophisticated way of doing what I was previously doing manually
Just a note: we've started using nameko pretty heavily at Clef
<http://getclef.com> and are really enjoying it (Mark, Grace and I --
all recent mailers -- all work there)! Thanks for all your hard work so far
and I hope we're able to contribute back going forward.
Jesse
On Wednesday, August 10, 2016 at 1:50:48 PM UTC-7, David Szotten wrote:
Ah, interesting!
Some of the core nameko developers recently met up to discuss roadmaps
and the eco system in general (notes will be posted to the mailing list
once written up). One of the issues we discussed was scaling back the
extent to which nameko assumes amqp. When nameko started out, there was no
http extension, and amqp was so core to everything at onefinestay that we
probably didn't consider the use case you brought up. we definitely need to
though!
i need to have a think about the best way to handle this, but in the
mean time, i threw together a workaround for you:
strip amqp dependencies from nameko class · GitHub
the core idea is a function `strip_rpc(service_class)` which dynamically
creates a subclass of the service with all rpc entrypoints and proxy
dependencies overridden by None.
the gist includes a test which uses the container_factory to host a
stripped version of the example service and successfully makes a http call
without any rabbit broker required
best,
David
On Wednesday, 10 August 2016 19:33:08 UTC+1, je...@pollak.io wrote:
Right now, we're primarily using container_factory to test HTTP
responses from HTTP services. In particular, we're creating a customer
HTTPRequestHandler that processes exceptions and transforms them into
standard responses and are testing that this works correctly. Is there a
way to do HTTP-level testing with the worker_factory?
Thanks David!
Jesse
On Wednesday, August 10, 2016 at 1:21:13 AM UTC-7, David Szotten wrote:
Hi
Could you give some more detail about why/when you are using the
container_factory over the worker_factory? If you want everything mocked
out it sounds like the worker_factory (which won't need rabbitmq) is a
better choice. The container factory is for when you want to simulate more
realistic conditions, and so defaults to hosting real versions of everything
If you need only a few select dependencies, say a db connection, one
option is to still use the worker_factory, and manually inject the db, as
in
https://github.com/onefinestay/nameko/blob/master/docs/examples/testing/alternative_dependency_unit_test.py#L52
Best,
David
On Wednesday, 10 August 2016 08:21:48 UTC+1, je...@pollak.io wrote:
Hi! We're writing tests for our services that test their high level
functionality using *worker_factory* and *container_factory*. When
we use *worker_factory*, all the service's dependencies are replaced
with mock dependencies as specified in the docstring
<https://github.com/onefinestay/nameko/blob/master/nameko/testing/services.py#L141>\.
When we use *container_factory*, we are manually mocking all
services using the replace_dependencies helper
<https://github.com/onefinestay/nameko/blob/master/nameko/testing/services.py#L229>,
which replaces the dependencies with mocks. Many of these are
*RpcProxy* dependencies that communicate with other services, but
given the specified functionality of *worker_factory* and *replace_dependencies,
*we expect these rpc-level dependencies to be mocked and for there
to be no need for an active RabbitMQ cluster.
*Unfortunately, even though all of our dependencies are mocked, when
we try to run our tests, the tests error out on container.start() with a
failed connection to RabbitMQ.*
Digging into the code a bit, I believe I've found the reason, but I
wanted to make sure I'm understanding.
I believe that while the *dependencies* are being replaced with
mocks, *subextensions *added by those dependencies are not being
replaced. The subextension addition logic happens in the container
*__init__*
<https://github.com/onefinestay/nameko/blob/master/nameko/containers.py#L160>,
which happens before both *worker_factory* and *replace_dependencies* replace
the *RpcProxy* dependencies with mocks. Therefore, *RpcProxy* dependencies
add a QueueConsumer
<https://github.com/onefinestay/nameko/blob/master/nameko/rpc.py#L40> dependencies
to *container.subextensions* which triggers the RabbitMQ connection.
For now, I've been able to work around this by manually overriding
the class dependencies before I use *container_factory, *like this*:*
ExampleService.rpc_dependency_1 = Mock()
ExampleService.rpc_dependency_2 = Mock()
container = container_factory(ExampleService, web_config)
container.start()
However, this seems suboptimal.
Has anyone dealt with this before?