Causality in Eventstore

EventStoreDB
1

Hey there,

as I am new to event sourcing, I am comparing different tools.

What I did not get from the documentation:

How do I provide the correct causal order of events in consumption and do not have race conditions?

e.g.

I update a field value multiple times over a certain time, create an event for every change and have many consumers within my subscription group.

What if one consumer is very slow and while it is still processing, the last (and final state) event is processed by a faster instance and then my final value is the one of the slow instance or maybe its retry-follower

Is there any possibility (maybe by design?) to avoid that?

Thanks in advance

Laura

2

Hi Laura

This really depends on your integration strategy and what you want to achieve. As a general answer I’d say:

Each consumer could pull events from your event provider. Each consumer would need to know their last processed sequence number. Each event therefore gets processed n times where n ist the number of consumers. You will have race conditions if multiple consumers update the same state. You therefore should either implement a mechanism to detect such a condition and react to it when it occures or combine those consumers into a single one to prevent races alltogether.

If you need to have separate consumers you might want to seperate state so that each can operate without interefing the others.

Based on what you wrote I assume that you have multiple consumers updating the same state and invoking different processes. In this case I’d try yo have single consumer that builds the state and invokes the processes afterwards.

Competing consumers can be struggle so it is worth finding the right solution for your scenario :slight_smile:

3

There are two ids you want to look into which are supported
$correlationId and $causationId will show you including realtime
updating graphs causative paths! See
https://pbs.twimg.com/media/DjCV9jQU4AAvwzg.jpg

"What if one consumer is very slow and while it is still processing,
the last (and final state) event is processed by a faster instance and
then my final value is the one of the slow instance or maybe its
retry-follower"

Can you provide more detail?

1 Like
4

I would suggest that if you require a subscriber to receive events in sequential order you should look at catch up subscriptions.

I have implemented this to keep my read models in sync and find it works very well. I have a service where only a single instance is ever deployed so no competition to process events, events are handled in order and will not proceed until they are successful. The downside is no horizontal scale or geo redundancy but if the service does go down the idea is you manage the position in the stream so nothing is missed when it comes back up.

There are some old blog posts on the topic if you google catch up subscriptions.

5

"I have a service where only a single instance is ever deployed so no
competition to process events, events are handled in order and will
not proceed until they are successful. The downside is no horizontal
scale or geo redundancy"

So there are some pretty trivial ways of getting this :open_mouth:

I am sure you have found that if you write multiple times the same
event (identified by the id) that the write will be idempotent. What
if when outputting your service used deterministic uuids? You could
then run two instances of the service in two geo locations, basically
whoever writes first wins the other will get an idempotent write :wink:
The one tricky thing here is in terms of versioning releases where the
multiple may not be writing the *same thing*.

There are also ways of making the eventstore nodes themselves
georedundant. This is one area I want to push on for a pretty major
feature coming up soon (expect to see over the next say 6 months or
so). I have been working around the idea for a long time. Right now
the clustering works via paxos and gives assured ordering between the
nodes. I have been wanting to get in "asynchronous clustering" which
can work between a group of nodes ... or a group of clusters.

The current model is CP in terms of CAP theorem, this model would be
AP + conflict detection. I will try bring things back up on it this
week. I have built it as an app that runs *next to* eventstore
previously but would like to see built in support for it.

6

Hey,

thanks for your answers.

For more detail:

Imagine a large monolithic system, partly separated into REST microservices. To overcome some issues within the system, i would like to introduce a world-domination-tool usable for event sourcing with answers to every use cases one could have in the future :wink:

Look for a more detailed example at the end of the post

Each consumer could pull events from your event provider. Each consumer would need to know their last processed sequence number. Each event therefore gets processed n times where n ist the number of consumers.
This is fine for different application types, in my case serviceX and serviceY

You will have race conditions if multiple consumers update the same state. You therefore should either implement a mechanism to detect such a condition and react to it when it occures or combine those consumers into a single one to prevent races alltogether.

I would suggest that if you require a subscriber to receive events in sequential order you should look at catch up subscriptions.
I have implemented this to keep my read models in sync and find it works very well. I have a service where only a single instance is ever deployed so no competition to process events, events are handled in order and will not proceed until they are successful. The downside is no horizontal scale or geo redundancy but if the service does go down the idea is you manage the position in the stream so nothing is missed when it comes back up.
There are some old blog posts on the topic if you google catch up subscriptions

I would love to have multiple service instances as consumers serviceX1 etc and my modularity-heart cries at the thought implementing a blocking mechanism for each entity in different use cases to avoid race conditions

I am sure you have found that if you write multiple times the same
event (identified by the id) that the write will be idempotent. What
if when outputting your service used deterministic uuids? You could
then run two instances of the service in two geo locations, basically
whoever writes first wins the other will get an idempotent write :wink:
The one tricky thing here is in terms of versioning releases where the
multiple may not be writing the same thing.
I found that, but I do not want to use only eventstore. The views for frontenddelivery are updatet in some kind of (cached or persisted) viewobjekt https://martinfowler.com/bliki/CQRS.html

There are also ways of making the eventstore nodes themselves
georedundant. This is one area I want to push on for a pretty major
feature coming up soon (expect to see over the next say 6 months or
so). I have been working around the idea for a long time. Right now
the clustering works via paxos and gives assured ordering between the
nodes. I have been wanting to get in “asynchronous clustering” which
can work between a group of nodes … or a group of clusters.
The current model is CP in terms of CAP theorem, this model would be
AP + conflict detection. I will try bring things back up on it this
week. I have built it as an app that runs next to eventstore
previously but would like to see built in support for it.
Sounds interesting, but how das that influence scaling consumers?

Based on what you wrote I assume that you have multiple consumers updating the same state and invoking different processes. In this case I’d try yo have single consumer that builds the state and invokes the processes afterwards.

There are two ids you want to look into which are supported
$correlationId and $causationId will show you including realtime
updating graphs causative paths! See
https://pbs.twimg.com/media/DjCV9jQU4AAvwzg.jpg

Thanks! this solves one of the questions in my head, here is the second

Let me expand you example to a specific use case:

Imagine I determine the current account value of this user by the aggregation of paymentevents (payload contains not the delta, but the new value). The user has 100 at the beginning

He places the order at cost 10. After slow Paymentservice1 has taken “PaymentMade” from paymentstream the user is fast and withdraws all his money. Paymentservice2 is fast, gives him his money and sets value to 0. Paymentservice1 finishes and sets value to 90

If I compare it to Apache Kafka I would say, account is one topic and the partition the user´s accountentity and each instance (within a consumergroup) can only have one partition https://kafka.apache.org/documentation/#intro_consumers

Can I somehow handle this in eventstore?

Thanks for discussion!

7

"Imagine I determine the current account value of this user by the
aggregation of paymentevents (payload contains not the delta, but the
new value). The user has 100 at the beginning
He places the order at cost 10. After slow Paymentservice1 has taken
"PaymentMade" from paymentstream the user is fast and withdraws all
his money. Paymentservice2 is fast, gives him his money and sets value
to 0. Paymentservice1 finishes and sets value to 90"

You just discovered one reason why banks don't "set the value" and use a ledger.

You will also notice that there is optimistic concurrency available
just to work around such cases (the 2nd write would get a version
failure when it tried to write notifying it that further writes had
occurred).

8

Where do you get this nice visualisation? I haven’t got that in my web interface (using the docker image).

9

That view is only available on the commercial version of the server.