Handling guaranteed only-once processing of certain events

EventStoreDB
#1

I’ve done a lot of searching and haven’t found a good example or explanation that answers my question. I’m curious how people out there are handling the processing/dispatching of events that result in the sending simple notifications via email, SMS, etc. These notifications should be sent once and only once with no chance of event replay triggering a resending of the notification and should potentially retry on failure to send. I’ve thought about using a live subscription to accomplish. My concern is that if my live subscription goes down for a time, for whatever reason, I’d be losing out on any notification triggering events that occur during the downtime. This leads me to using a catch-up subscription. In this case, I’d be keeping track of my checkpoints and running through anything that was missed while the subscription was down. I’m already doing this with my read model projections in a single relational database, and, as we already know, the nice part about event sourcing is that I can completely blow away my read model store and feel confident that I can rebuild it all. If I were to keep track of my notification checkpoints in the same read model store, how could I still retain the benefit of being able to wipe out my read model store and yet guarantee only-once delivery of my email notifications? Is the read model store the wrong place to be keeping track of my already successfully sent notifications? Do I need to implement some sort of state machine (saga) to coordinate the sending of notifications? Perhaps I am completely over-thinking (wrongly-thinking) the whole concept. Any suggestions or comments will be greatly appreciated.

Thanks,
Jordan

#2

I think you’re just overthinking things here. You seem to already have the answer figured out.

  • Use catch-up/live subscription.
  • Persist the event number / position when email/sms is successfully sent
  • Determining whether send is successful may require a state machine, saga, or simpler retry logic
  • Make sure you keep this information around and/or don’t replay events with this handler. (This is not the same thing as your typical read model)
    Brian
#3

Or use a persistent subscription (competing consumers) if you want multiple handlers

#4

Well, I’m glad that I’m on the right track then. In relation to this, is it more common to have a single subscriber that dispatches to the appropriate event handlers or to have multiple subscriptions, one for each handler/type of handler? I guess in the case of competing consumers that Greg is talking about each subscription would be running as a separate process, which would be fine for the case of sending notifications. However, I’m not sure that this would be appropriate for updating my read model.

Thanks,

Jordan

#5

It depends on the read model and whether or not ordering is required.

Also as a side note here with something like sending a notification
(say a SMS) there is no such thing as "only-once processing" there
will always be some probability of a duplicate unless you work with a
very interesting telecom :slight_smile:

#6

Greg, I agree with you there. I just want to minimize the chance of duplicates being sent out on my end.

Let me be a little more specific about my case. Right now, I have a single EventStore node and I am projecting my events to a relational read model in a single SQL Server database. I have a table in my read model which keeps track of the current checkpoint for each type of projection. Right now I have no partitioning of the read model across multiple nodes. I have a single catch-up subscriber that is dispatching to my projection builders based on the lowest checkpoint reported by each projection builder at application startup. In the case of adding notification handlers, I am wondering whether or not it would be better to add another subscriber just for this purpose or to stick with my single read model catch-up subscriber. Maybe it doesn’t really make a difference either way. Thoughts?

Thanks,

Jordan

#7

If you write your checkpoints atomically with the read model update
you basically simulate only-once messaging. I wouldn't worry too much
about things beyond that until I ran into (or saw obviously coming)
performance problems