Server managed durable subscribers

EventStoreDB
#1

I’ve been reading the documentation for Competing Consumers and noted that it recommends catch up subscriptions for cases like projecting into a read model as it will be sensitive to the order of events. It also says that one of the benefits of competing consumers is to support failover, which is hard to do when subscriptions are client managed. Is it possible to have both, i.e. a server tracked, single consumer, fully ordered subscription with the same ack/checkpoint semantics as the competing consumers have today?

Specifically, what happens if I create a persistent subscription with MaxSubscriberCount = 1? Will one subscriber claim an implicit lock and other subscribers receive an error when trying to subscribe that goes away when the first subscriber does? Would it be possible to implement a single consumer subscription with HA using this method?

I’ve also just noticed the discussion about consumer strategies at http://docs.geteventstore.com/dotnet-api/3.5.0/competing-consumers/. The “pinned” strategy appears to be very close to what I’d ideally want, but mentions that ordering is not guaranteed

  • Under what circumstances may I receive out of order messages when using this strategy?

  • Would it be possible in a future release to provide something like this strategy, but with a fixed number of partitions (for example 1) to avoid buckets being re-assigned as subscribers come and go?

Cheers,

Kristian

#2

- Under what circumstances may I receive out of order messages when
using this strategy?

Retries for any reason (timeouts etc).

- Would it be possible in a future release to provide something like
this strategy, but with a fixed number of partitions (for example 1)
to avoid buckets being re-assigned as subscribers come and go?

You can already do this with round robin and a size of 1.

#3
  • Under what circumstances may I receive out of order messages when

using this strategy?

Retries for any reason (timeouts etc).

  • Would it be possible in a future release to provide something like

this strategy, but with a fixed number of partitions (for example 1)

to avoid buckets being re-assigned as subscribers come and go?

You can already do this with round robin and a size of 1.

So, if the number of attempted subscribers is greater than the configured max number of subscribers, will the additional ones receive an error when subscribing, or simply receive no events until the first subscription goes away?

Additionally, even if this does cover the case of 1 subscriber (i.e. no errors - and additional subscribers are parked until the first one goes away), it would be nice to generalize this to n subscribers, n being configured on the persistent subscription. While, as I understand it, the pinned strategy today would potentially mean that events are processed out of order as new subscribers come in and buckets therefore get reassigned to subscriptions, an approach with a fixed number of subscribers would at worst have events replayed to subscriber 2 when they were already processed by subscriber 1 - something that subscribers already need to cope with due to checkpointing and retries.

#4

"Additionally, even if this does cover the case of 1 subscriber (i.e.
no errors - and additional subscribers are parked until the first one
goes away), it would be nice to generalize this to n subscribers, n
being configured on the persistent subscription. While, as I
understand it, the pinned strategy today would potentially mean that
events are processed out of order as new subscribers come in and
buckets therefore get reassigned to subscriptions, an approach with a
fixed number of subscribers would at worst have events replayed to
subscriber 2 when they were already processed by subscriber 1 -
something that subscribers already need to cope with due to
checkpointing and retries."

Pinned also has out of order for retries of any kind. Both (and any at
least once messaging system) will have this order.

Normally in such systems you assume 99.9% of messages are in order and
assume a cost when they are not.

#5

I probably didn’t express my idea very well, let me try again and please ignore previous context.

Would it make sense to add a new strategy, PinnedFixed, that divided up the stream in buckets much like the current Pinned, but in a fixed number of partitions? The semantics would be that, each new subscriber that connects to the persistent subscription would be assigned a partition. New subscribers attempting to connect when the current number of subscribers exceeds the configured number of partitions would be paused, and not receive any messages until any of the currently active subscribers went away.

The benefit to this scheme would be less reshuffling of buckets to subscribers, and therefore less risk of out of order delivery due to retries in combination with subscribers joining/leaving.