Recommendations for keeping state between @timer calls

Hello! First of all, thank you for developing nameko; really liking it so
far. I'm quite new to the microservice architecture however and I'd like to
ask for your recommendations on how to approach the following two problems:

1. I have a service based on the timer extension that fires each x seconds,
let's call it TimerService. Its task is to get some information from a
database, which I implemented as a DependencyProvider. However, I also need
to keep state between subsequent calls. I implemented this as a
DependencyProvider as well. Further, on the first call to the TimerService
task, I need to get some additional data from the database to initialize
the state. I was wondering what the best design would be for this use case.
Right now I do something like:

    @timer(interval=1)
    def task(self):
        if not self.state.initialized:
            self.state.update(self.db.get_initial_state())
        else:
            do_work(self.state)

Is there a better approach? I thought it would be cleaner to access the
database's DependencyProvider from the setup() method of the state's
DependencyProvider, but at this point they are not bound to the service
yet, so this seems not possible.

2. In my use case, I don't require multiple TimerServices. What would be
the recommended approach to guarantee that only one TimerService (and one
task call) can be launched at a time?

Hi,

Hello! First of all, thank you for developing nameko; really liking it so
far. I'm quite new to the microservice architecture however and I'd like to
ask for your recommendations on how to approach the following two problems:

Glad you're enjoying Nameko :slight_smile:

1. I have a service based on the timer extension that fires each x
seconds, let's call it TimerService. Its task is to get some information
from a database, which I implemented as a DependencyProvider. However, I
also need to keep state between subsequent calls. I implemented this as a
DependencyProvider as well. Further, on the first call to the TimerService
task, I need to get some additional data from the database to initialize
the state. I was wondering what the best design would be for this use case.
Right now I do something like:

    @timer(interval=1)
    def task(self):
        if not self.state.initialized:
            self.state.update(self.db.get_initial_state())
        else:
            do_work(self.state)

Is there a better approach? I thought it would be cleaner to access the
database's DependencyProvider from the setup() method of the state's
DependencyProvider, but at this point they are not bound to the service
yet, so this seems not possible.

It's 100% correct to be using a DependencyProvider to maintain the state.
Communication between DependencyProviders is only really supported inside
the service methods though, as you have done.

A cleaner option, if you want to abstract the initial state load away,
would be to give your state management DependencyProvider its own way to
access the database rather than trying to "share" the access with the other
dependency provider. There was another thread about this recently:
https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/nameko-dev/5RB4xE5mz2E

2. In my use case, I don't require multiple TimerServices. What would be

the recommended approach to guarantee that only one TimerService (and one
task call) can be launched at a time?

The @timer entrypoint is very simplistic and doesn't really cope with this
case. You could use a distributed lock to make sure only one runs at once
(e.g. using https://github.com/Overseas-Student-Living/platform-lock) but a
much better solution would be a timer implementation that was properly
"cluster-aware". There are many ways to do this, although one relatively
lightweight one would be to use AMQP messages with a TTL, in the same way
that the backoff is implemented
in https://github.com/nameko/nameko-amqp-retry (a message would "wait" in
RabbitMQ until the timer expires, and then be delivered to one of the
consuming services)

···

On Tuesday, December 12, 2017 at 4:45:44 PM UTC, P. wrote:

Hi Matt,

Thanks for taking the time to respond to my question!

It's 100% correct to be using a DependencyProvider to maintain the state.

Communication between DependencyProviders is only really supported inside
the service methods though, as you have done.

A cleaner option, if you want to abstract the initial state load away,
would be to give your state management DependencyProvider its own way to
access the database rather than trying to "share" the access with the other
dependency provider. There was another thread about this recently:
https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/nameko-dev/5RB4xE5mz2E

Yes, I saw that thread and it is indeed a plausible solution, but I guess
then you would need to take the shared functionality (e.g. making the
connection to the database etc.) out of the database DependencyProvider so
that you can import that in the state DependencyProvider as to avoid having
duplicate code. So if I understand correctly it's a tradeoff between having
all your database functionality in one place versus having it spread out
but requiring no initial state load like in my example.

The @timer entrypoint is very simplistic and doesn't really cope with this
case. You could use a distributed lock to make sure only one runs at once
(e.g. using https://github.com/Overseas-Student-Living/platform-lock) but
a much better solution would be a timer implementation that was properly
"cluster-aware". There are many ways to do this, although one relatively
lightweight one would be to use AMQP messages with a TTL, in the same way
that the backoff is implemented in
https://github.com/nameko/nameko-amqp-retry (a message would "wait" in
RabbitMQ until the timer expires, and then be delivered to one of the
consuming services)

Thanks for the info! I thought about the distributed lock but the TTL
messages is something I will look into.

Cheers,

Hi Matt,

Thanks for taking the time to respond to my question!

:slight_smile:

It's 100% correct to be using a DependencyProvider to maintain the state.

Communication between DependencyProviders is only really supported inside
the service methods though, as you have done.

A cleaner option, if you want to abstract the initial state load away,
would be to give your state management DependencyProvider its own way to
access the database rather than trying to "share" the access with the other
dependency provider. There was another thread about this recently:
https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/nameko-dev/5RB4xE5mz2E

Yes, I saw that thread and it is indeed a plausible solution, but I guess
then you would need to take the shared functionality (e.g. making the
connection to the database etc.) out of the database DependencyProvider so
that you can import that in the state DependencyProvider as to avoid having
duplicate code. So if I understand correctly it's a tradeoff between having
all your database functionality in one place versus having it spread out
but requiring no initial state load like in my example.

If your state managing DependencyProvider inherited from
nameko-sqlalchemy's DependencyProvider then I doubt there would be much
code duplication.

The @timer entrypoint is very simplistic and doesn't really cope with this

case. You could use a distributed lock to make sure only one runs at once
(e.g. using https://github.com/Overseas-Student-Living/platform-lock)
but a much better solution would be a timer implementation that was
properly "cluster-aware". There are many ways to do this, although one
relatively lightweight one would be to use AMQP messages with a TTL, in the
same way that the backoff is implemented in
https://github.com/nameko/nameko-amqp-retry (a message would "wait" in
RabbitMQ until the timer expires, and then be delivered to one of the
consuming services)

Thanks for the info! I thought about the distributed lock but the TTL
messages is something I will look into.

No problem. The cluster-aware lock would be a good addition to the Nameko
library if you end up developing it.

···

On Wednesday, December 13, 2017 at 1:00:31 PM UTC, P. wrote:

Cheers,