Greg's observation re. projections that handle multiple concurrent events

EventStoreDB
1

Greg recently tweeted an interesting observation…

“I am curious. To this day I am the only person I know who has written projections that handle multiple concurrent events. I can think of many reasons for why this is. Are people not running into IO issues due to vast hardware improvements or are people just not working on high throughput systems?”

I have reproduced it here because Twitter is not ideal for technical discussion (for someone as wordy as me).

This isn’t necessarily about EventStore but I think it is relevant to this community, and I hope Greg doesn’t mind it being put here.

2

So what does this observation actually mean?

1) Before I can absorb this I need to know more about these concurrent events:

  • Are these events in the same stream?

  • Are they concurrent by design/intention, or is it accidental and undesirable?

  • Is this a “Projection” in the EventStore sense, or something different?

2) If I don’t explicitly handle these events in my projection then what negative outcome might occur:

  • If in the same stream then surely they now have an order, even if it was decided by chance.

  • If in different streams then I see why this could blow up.

  • If the projection was reset then might it produce a different result?

3) Why are others seemingly not handling this scenario:

  • They haven’t been burned by it (yet)?

  • They are being burned but haven’t noticed!

4) Does the mention of IO issues and hardware suggest that this problem could become more prevalent?

  • What kind of bottleneck would cause events to be recorded as concurrent (even if they weren’t intended to be)?

  • How could it be avoided?

5) Finally, the big question, assuming there really is a problem that is not being adequately handled…

  • How has Greg handle these events?

  • Is it a pattern that can be generalised?

3

Are there any new links or resources on how to handle concurrency in event sourced systems or is it the same as handling concurrency in an HTTP API?

4

Well you need a database

  • that support optimistic concurrency ( i.e lock free at the db level)
  • linearization , i.e global ( at the log level ) & local (at the stream level) guaranteed ordering and gapless sequence number (revision) at all stream levels.
    • which make sure all reactive processing ( read models, reactions) can checkpoint and do gap / missing / out of order detection
1 Like