Problems with stable subscriptions when subscriptions drop or connection is reconnected

EventStoreDB
1

Hello,
I have several components inside my system that establish subscriptions with the EventStore.

It works well as long as no subscription or connection has to be re-established, but if that happens, I get either none, one or multiple concurrent subscriptions.

When SubscriptionDropped is invoked, is it my job to establish a new subscription? I’d guess so, but it seems like that in some cases, either the old subscription survives (even though it said it dropped) or the ClientAPI is automatically establishing a new subscription, resulting in two/multiple subscriptions to the same stream and multiple invocations of EventAppeared for the same event.

Example:

Everything starts up, everything connects,

App has active Subscription (ID #1) to EventStore

event occurs, Subscription ID #1 invokes EventAppeared

I kill event store manually

ClientAPI invokes SubscriptionDropped for Subscription ID #1 with DropReason ConnectionClosed

I restart event store manually

App subscribes again because previous Subscription was supposedly dropped, so App has now Subscription ID #2

event occurs, Subscription ID #1 AND #2 invoke EventAppeared.

I could try to deduplicate etc. on the client-side but that would lead to multi-threading and possibly message reordering etc. and I’d guess that it shouldn’t be that hard.

Am I doing something wrong? How can I be sure a subscription is really “gone” if not based on SubscriptionDropped?

Should I only reconnect based on specific DropReasons? Although ConnectionClosed sounds severe enough to warrant a new subscription…

Versions:

Server+Client: Current dev branch

Thanks in advance,

regards,

Nicolas

2

Without looking at the code (I will later this evening) my guess is it is automatically reconnecting your subscription for you.

This may not be the best default behaviour on a subscription but instead to always force you to reconnect yourself.

@James what do you think about this?

Cheers,

Greg

3

If that’s true, then either - like you said - let the user decide if subscriptions should auto-reconnect or make a flag available inside SubscriptionDropped (additional parameter perhaps) which states which side should handle the reconnect.

Thanks,

Nicolas

4

We have also run into problems with disconnection issues - https://groups.google.com/forum/#!msg/event-store/M4I84pfB2lc/PpxDRRoBqZAJ

5

Is there any update on this? :slight_smile:

6

What does your connection code look like? What options do you have from the following page?

https://github.com/eventstore/eventstore/wiki/Overview-(.NET-API)

We managed to overcome our issues by changing our connection options - there is a KeepRetrying flag that we removed which then allowed disconnects to bubble up

Tom

7

Sorry I meant KeepReconnecting

8

We do use KeepReconnecting and KeepRetrying. If I understand you correctly, the ClientAPI would automatically reconnect subscriptions as long as Disconnected isn’t called?
So If I were to remove KeepReconnecting, I would have to manually reconnect in OnDisconnected but could be sure that I don’t need to re-create subscriptions as long as OnDisconnected was NOT called?

That is my greatest problem at the moment, I find it quite hard to understand what and when the ClientAPI does something (like Reconnect Subscriptions).

Some kind of demonstration/picture like from the guys from SignalR ( http://www.asp.net/signalr/overview/signalr-20/hubs-api/handling-connection-lifetime-events ) would be extremely helpful.

Otherwise I have no other choice than via Trial&Error and that isn’t very efficient.

Nicolas

9

What are you trying to achieve when the connection is not available?

I have different settings for different applications, but they involve a combination of .SetReconnectionDelayTo() and .LimitReconnectionsTo() to influence the retry strategy. For example, on a catch-up subscription (lets say for example I am populating a read-model) that is reading from a stream, I typically want it to try to reconnect for longer (or forever) than when running a live application that is writing to a stream. I set a strategy using the options above that lets me control the reconnection strategy driven by business requirements.

It sounds like you are having issues because both ES and your application code are trying to re-establish connections? I guess you need to decide if you want to manually control the re-connection strategy or are happy for ES to do it for you.

Tom

p.s. I’m just a user of event store, not one of the authors :slight_smile:

10

The part of my application that is experiencing these problems is for updating my read models, so like you said, it should reconnect for forever. I would be very happy if ES could just handle all of the reconnect and resubscribing activities, but I don’t think it does this at all times and isn’t documented enough (at least for the connection lifecycle topics) to be trustworthy enough.

@Greg, What would be the best way to have a rock-stable subscription that survives all the usual cases of fast reconnect, longer reconnect, disconnect periods, etc. ? Is this achievable with “only” ES/ClientAPI or would one need a wrapper that handles some or all of the (Re-)Connection logic?

Thanks,

Nicolas

11

Better question might be: When should one re-establish subscriptions? Since these seem to run even after SubscriptionDropped was executed (maybe automatically re-established), I need a point in the lifecycle when I can be sure that the first subscription is disabled and I should re-create it.

12

Can you give us a case where subscribefrom does not reconnect etc properly with say keep reconnecting?

13

There seems to be a bit of confusion here with connections re-establishing and how this relates to subscriptions. i.e. if you subscribe using Connection.SubscribeToStreamFrom() and the connection gets disconnected then auto-reconnected, does the subscription itself automatically pick up where it left off? I’m under the impression this is all taken care of internally given the right options on the connection. Is this correct?

cheers

Tom

14

I just confirmed this is happening on my system. :slight_smile:

Subscription settings are:

var settings = ConnectionSettings.Create()

.SetDefaultUserCredentials(new UserCredentials(Settings.Default.EventStoreUsername, Settings.Default.EventStorePassword))

.SetReconnectionDelayTo(TimeSpan.FromSeconds(Settings.Default.EventStoreReconnectionDelay))

.LimitReconnectionsTo(Settings.Default.EventStoreReconnectionAttempts)

.WithConnectionTimeoutOf(TimeSpan.FromSeconds(5))

.SetOperationTimeoutTo(TimeSpan.FromSeconds(5));

If I run the program with ES running, the subscribers are successfully subscribed. If I manually stop the service, the SubscriptionDropped event fires. If I restart the service, the subscribers reconnect. All this is done automatically by ES, I’m not manually handling anything…

So… Nicolas, can you try removing your manual re-subscription code and try just letting ES handle it for you?

tom

15

That sounded good and I am currently changing all components to not handle reconnect at all, with EventStoreConnection at KeepReconnecting and KeepRetrying. Initial results seem good, but will test further.
Am I right that in this case I can “more or less” ignore SubscriptionDropped invocations? My initial guess was no (hence the manual reconnects and new subscriptions), but I gladly ignore it if it is indeed possible. If that’s the case, that might be a good thing to add to the docs.

I’ll keep you all informed.

16

The event is there to let you know a subscription was dropped (it may or may not be reconnected depending on your settings). You can tell the connection to handle reconnection logic itself or you can decide that you want to handle it on your own (eg dont reconnect and you handle your logic as you see fit).

There is going to be an effort announced in the next week or so to get a LOT more documentation on the client api and on these kinds of settings on the wiki (there is a lot in the code as xml docs but has not been exported to the wiki)

Cheers,

Greg

17

Ah ok I understand.

Is it possible that the ClientAPI will not re-establish/continue subscriptions if one or many specific Reason’s are set in SubscriptionDropped? Connectivitiy issues seem to be handled just fine, but if an exception is thrown during EventAppeared, the subscription is dropped with Reason: CatchUpError and won’t process further events.

Thanks,

Nicolas

18

Yes because continuing (calling your handler again will throw an exception again) if you want a retrying strategy to your handler compose it in the event appeared.

19

Hello old thread!

Is there a definitive list of which SubscriptionDropReasons require manual re-subscription? Either in code or docs?

SubscriptionDropReason.UserInitiated:

SubscriptionDropReason.NotAuthenticated:

SubscriptionDropReason.AccessDenied:

SubscriptionDropReason.SubscribingError:

SubscriptionDropReason.ServerError:

SubscriptionDropReason.ConnectionClosed:

SubscriptionDropReason.CatchUpError:

SubscriptionDropReason.ProcessingQueueOverflow:

SubscriptionDropReason.EventHandlerException:

SubscriptionDropReason.Unknown:

Thanks

Tom

20

I would have to look through the code but AccessDenied and
SubscribingError I know for a fact shouldn't be.

If you have reconnects etc done then all normal operations should
automatically reconnect. Likely things that are not reconnecting are
not reconnect able.

Greg