Persistent Subscription: preserve order

EventStoreDB V20.X
#1

Hi - given a following setup:

  • EventStore (what a surprise!) with a persistent subscription on a certain stream
  • MongoDb for read models
  • Kubernetes hosting three pods with a listener on the persistent subscription
  • Three events available within the persistent subscription: event 1 updates a property A, event 2 updates a property B and event 3 again updates a property A

I assume that it might result in Pod 1 receiving event 1, Pod 2 receiving event 2 and Pod 3 receiving event 3. How can I ensure, that my read model is updated in the correct order? Of course I could skip event 1 as event 3 updates the same property again. But I cannot rely on some version number alone, as event 2 must not get lost.
So I am thinking on defering executing of event 2 and event 3 in their pods until the current version of my read model is “[event version] - 1”. Can I rely on the presence of such an version number that always increments by 1?

Is there any other way I could ensure that my read model represents the correct state?

#2

Persistent Subscriptions do not guarantee ordering. If ordering is a requirement then you should take a look at catch-up subscriptions. It does mean you need to maintain the checkpoint yourself. MongoDb supports multi document transactions so this should be doable.

#3

And, I’d argue, you don’t need to wrap the document and checkpoint updates in a transaction as Mongo projections can be easily made idempotent.

Keeping the checkpoint in Mongo, in this case, would be preferred not only because of the order but as the read model is something transient. By keeping the checkpoint in the same database, if the database is gone, the whole thing will just rebuild.

#4

In our production, we use a similar setup as you mentioned for a few of our read copies. We like to use persistent subscriptions to create competing consumers and share the load between multiple instances of the same read copy populator service.

First of all, I assume you don’t need event ordering guarantee across multiple aggregate instances, and you only really care about event order inside single streams and not whole projections. If you need event ordering across multiple streams I would also suggest using catch-up subscriptions.

If your read copy documents match one-to-one to your aggregate instances the solution is conditional updates to MongoDB. Your read copy should also have a “version” field attached to it and when performing the update for event number X you explicitly filter using {version: X-1} and check updated count to make sure your update went through, and if not just nack the event and requeue for future processing to allow re-ordering.

Pinned persistent subscriptions help a lot in this case. They do most of the work by distributing your aggregate events to same service instances, but hiccups occur when service instances drop or service scale changes. We usually only see events being requeued for reordering purposes during these scale changes.

If your read copy documents do not match one-to-one to your aggregates (e.g. N read copy documents created out of 1 aggregate instance), then you need to also create a document collection that has one-to-one relation with your aggregates to still hold the version field. You basically make sure the updates to your read copy are idempotent operations and you update the aggregate document just before you ack the event. If anything goes south during handling an event, you retry the whole thing.

#5

Nothing stops you from detecting that you haven’t processed the preceding events when you hit the read model and going back and asking for them directly from the event store and applying them all at once (making the preceding events no-ops when they turn up in a consumer)