thanks for your answers.
For more detail:
Imagine a large monolithic system, partly separated into REST microservices. To overcome some issues within the system, i would like to introduce a world-domination-tool usable for event sourcing with answers to every use cases one could have in the future
Look for a more detailed example at the end of the post
Each consumer could pull events from your event provider. Each consumer would need to know their last processed sequence number. Each event therefore gets processed n times where n ist the number of consumers.
This is fine for different application types, in my case serviceX and serviceY
You will have race conditions if multiple consumers update the same state. You therefore should either implement a mechanism to detect such a condition and react to it when it occures or combine those consumers into a single one to prevent races alltogether.
I would suggest that if you require a subscriber to receive events in sequential order you should look at catch up subscriptions.
I have implemented this to keep my read models in sync and find it works very well. I have a service where only a single instance is ever deployed so no competition to process events, events are handled in order and will not proceed until they are successful. The downside is no horizontal scale or geo redundancy but if the service does go down the idea is you manage the position in the stream so nothing is missed when it comes back up.
There are some old blog posts on the topic if you google catch up subscriptions
I would love to have multiple service instances as consumers serviceX1 etc and my modularity-heart cries at the thought implementing a blocking mechanism for each entity in different use cases to avoid race conditions
I am sure you have found that if you write multiple times the same
event (identified by the id) that the write will be idempotent. What
if when outputting your service used deterministic uuids? You could
then run two instances of the service in two geo locations, basically
whoever writes first wins the other will get an idempotent write
The one tricky thing here is in terms of versioning releases where the
multiple may not be writing the same thing.
I found that, but I do not want to use only eventstore. The views for frontenddelivery are updatet in some kind of (cached or persisted) viewobjekt https://martinfowler.com/bliki/CQRS.html
There are also ways of making the eventstore nodes themselves
georedundant. This is one area I want to push on for a pretty major
feature coming up soon (expect to see over the next say 6 months or
so). I have been working around the idea for a long time. Right now
the clustering works via paxos and gives assured ordering between the
nodes. I have been wanting to get in “asynchronous clustering” which
can work between a group of nodes … or a group of clusters.
The current model is CP in terms of CAP theorem, this model would be
AP + conflict detection. I will try bring things back up on it this
week. I have built it as an app that runs next to eventstore
previously but would like to see built in support for it.
Sounds interesting, but how das that influence scaling consumers?
Based on what you wrote I assume that you have multiple consumers updating the same state and invoking different processes. In this case I’d try yo have single consumer that builds the state and invokes the processes afterwards.
There are two ids you want to look into which are supported
$correlationId and $causationId will show you including realtime
updating graphs causative paths! See
Thanks! this solves one of the questions in my head, here is the second
Let me expand you example to a specific use case:
Imagine I determine the current account value of this user by the aggregation of paymentevents (payload contains not the delta, but the new value). The user has 100 at the beginning
He places the order at cost 10. After slow Paymentservice1 has taken “PaymentMade” from paymentstream the user is fast and withdraws all his money. Paymentservice2 is fast, gives him his money and sets value to 0. Paymentservice1 finishes and sets value to 90
If I compare it to Apache Kafka I would say, account is one topic and the partition the user´s accountentity and each instance (within a consumergroup) can only have one partition https://kafka.apache.org/documentation/#intro_consumers
Can I somehow handle this in eventstore?
Thanks for discussion!