.NET Client API catch-up subscription callback concurrency?

EventStoreDB
#1

Last week I was seeing some odd behavior in my ES client during debugging. I make a simple call to SubscribeToAllFrom, which I expect to give me a sequential stream of events, but when I set a breakpoint inside the eventAppeared handler I occasionally will see a weird concurrency situation. The same event seems to be sent to the callback twice. Since I wasn’t expecting that, my (non-thread-safe) event processing code goes kaboom. :stuck_out_tongue:

So I have two couple of questions. (Assuming no bugs in ES/ES.Client)

  1. Is SubscribeToAllFrom expected to provide an exactly-once deliver guarantee for each event?
  2. Is SubscribeToAllFrom’s invocation of its various callbacks (eventAppeared, liveProcessingStarted, subscriptionDropped) totally sequential (meaning only one of those callbacks will ever be invoked at a time, and no other callback will be invoked until that invocation has finished), or is it totally concurrent (it doesn’t seem to be - I normally don’t get bombarded with a dozen simultaneous tasks for processing the next event), or is it partially concurrent in some way?
    If the ES/ES.Client behavior isn’t what I’m expecting, I’m going to need some kind of in-memory construct to deduplicate and help with event ordering (something like a HashSet of received event IDs and a queue), but even that isn’t going to absolutely guarantee ordered delivery (though it will guarantee at-most-once delivery).

On the other hand, if the ES/ES.Client behavior is what I’m expecting, then I can try to reproduce the behavior in some sample code and hopefully you can tell me what I’m doing wrong.

Thanks!

#2

I would love to see some sample code without a debugger connected or
restarting a subscription exhibite the behavior.

By default the assurance is at least once delivery (its log +
checkpoint) on things like reconnect/restart it is possible to get
duplicated messages (especially when coming back in and saying start
from X where is the last checkpoint you know of).

The concurrency model for events is synchronous as can be seen here:

https://github.com/EventStore/EventStore/blob/dev/src/EventStore.ClientAPI/EventStoreCatchUpSubscription.cs#L513

Only a single thread should ever be in that code at a time. Some of
the others things e.g. dropped etc it might be possible to happen it
would take me a bit longer of looking through to determine.

Cheers,

Greg

#3

Sorry wrong code place:)
http://github.com/EventStore/EventStore/blob/dev/src/EventStore.ClientAPI/EventStoreCatchUpSubscription.cs#L289

#4

Thanks so much Greg!

It was only ever with the debugger attached and hitting a breakpoint in the event handler, so based on what you’ve said I’m guessing it was a reconnection-related issue. I haven’t been able to reproduce the behavior so far today, but when I can I will update this post with sample code.

And given what you’ve said about reconnection behavior, assuming I want exactly-once delivery, I would at least need to put in a greater-than comparison with the previously processed event’s CommitPosition, right? And that’s assuming that there’s no concurrent behavior, so if there is concurrency I would need some kind of stronger guarantee in my own code?

#5

There is no concurrency on a single subscriber. The at least once can
be seen quite easily. You start a connection by giving it a position
to start from (say 1 for a stream subscriptions). It then gives you
event #2 you process and mark 2 complete (checkpoint). It then gives
you number 3, while you are processing it the power is pulled. When
you start up again you will re-read 2 and the processing of 3 is
unknown. If you store your checkpoint atomically with your write then
this mostly saves you but there are some other circumstances around
things like reconnects that may cause you to receive one twice.

In the competing consumer model it is also at least once.

Cheers,

Greg

#6

Well, with the possible exception of concurrency when there’s a dropped subscription, right? Because there was something causing my issues last week, I had the processing results to prove it. :slight_smile:

Thankfully in my case, my ES client is actually designed to read from the beginning every time it starts up, so the only issue is when the subscription is dropped - that’s when I might want to jump ahead as an optimization. I use the liveProcessingStarted callback to inform my code of when to start processing.

Lars

#7

"Well, with the possible exception of concurrency when there's a
dropped subscription, right? Because there was something causing my
issues last week, I had the processing results to prove it. :)"

If you get a dropped connection while you are debugging something (say
because of a heartbeat timeout) what event should we start by asking
for? :slight_smile:

#8

in case that wasn't clear, you are still processing the current event,
how do we know you finished it?

#9

Right. Ideally, a reconnect will pick up from the last handler that was known to have completed. I assume you do this? (In my case I don’t really care if you resubscribe from the first event in the store, since I’ll just ignore everything up until my own checkpoint, but that’s beside the point.) The thing is that when the reconnect happens it does seem like it happens on a separate thread, so while my handler is processing, say, event 10 in thread A (in step mode in the debugger), your ES.Client thread B determines a heartbeat timeout between two successive debugger steps and causes a reconnect to occur which then starts by sending me a duplicate of event 10 which I’m still processing in thread A. That’s the behavior I observed, at any rate.

As long as there’s no way that the ES.Client will ever send me event 11 on one thread, followed by event 10 on another thread (i.e., from a different connection attempt), I’m quite happy with the design. :slight_smile: But I suppose it could happen, right? At least with the debugger attached?

#10

"As long as there's no way that the ES.Client will ever send me event
11 on one thread, followed by event 10 on another thread (i.e., from a
different connection attempt), I'm quite happy with the design. :slight_smile: But
I suppose it could happen, right? At least with the debugger
attached?"

This should not happen. If you can make it happen I would be happy to look at it

#11

Alright, thank you! If I can make it happen I’ll post it. :slight_smile:

#12

btw to help you there is also a setting to increase heartbeat timeouts
for exactly this reason (both in client and in server)