EventStore use case

EventStoreDB
1

Hi,

Just wanted an advice on what would be the best solution for using Event Store since its new for me.

I started to work on solution that is a part of bigger Ecommerce system that have already implemented customer streams.

I’ve done some initial testing on it using Persistent subscriptions.

The plan was to use Persistent Subscription (or maybe sth else will be more relevant) on @ce-Customer and react on the each customer change.

Each of the customer will have quite a huge payload of data (mostly customer specyfic prices). Processing one call take some time even more then few seconds.

In situation of hundred or thousands Customer events that are populated during periodic exports to ES… it takes time. Persistent Subscriptions with single Subscriber handling at one message seems not to be very efficient.

It seems its not a ES issue after all but is any try of parallel consumption need to be designed and done completely on consumer end or is there any support for that? Therefor is using persistent subscription is actually a case here.

In general what would be your advice on it.

Cheers!!

2

So a two things based on your description…

  1. the number of events etc all seem fine. Not much of an issue overall…

  2. "Persistent Subscriptions with single Subscriber handling at one message seems not to be very efficient. "

You can hook up multiple subscribers to your persistent subscription, its designed to support N. Usually N is relatively small say < 10 but this should give you an order of magnitude better performance and ability to scale it up/down no?

3

Just as to add a bit.

  1. with persistent subscriptions you can still do things like use async and process more than one message concurrently.
4

Hi Greg.
Thank you for your fast answer. I appreciate it.

Done quick test using .net Api and subscribe with many subscribers with different group names and auto ack. Yet all subscribers got all messages. WIll probably need some configuration on those to limit message handling to one.

Will dig in to documentation then.

Thanks a lot!

W dniu sobota, 12 października 2019 20:57:37 UTC+2 użytkownik Greg Young napisał:

5

I think the issue here is your group names …

They should be on the SAME group name.

What the group name allows you to do is say:

On machine 1 I have subscriber A to stream1 group 1 (stream1:group1)

On machine 2 I have subscriber B to stream1 group 2 (stream1:group2)

These two subscribers will receive all messages in stream1 (they are two groups… another word here might be “subscription”)

On machine 1 I have subscriber A to stream1 group 1 (stream1:group1)

On machine 2 I have subscriber B to stream1 group 2 (stream1:group1)

These machines are BOTH working in the same group on two machines! Messages will be managed between the multiple subscribers (think competing consumers)

On machine 1 I have subscriber A to stream1 group 1 (stream1:group1)

On machine 1 I have subscriber A to stream1 group 1 (stream1:group1)

On machine 2 I have subscriber B to stream1 group 2 (stream1:group1)

This is also totally fine! its not one per machine etc. There are N subscribers on the same stream. Now in terms of how the messages between them get dispatched look at:

https://eventstore.org/docs/dotnet-api/code/eventstore.clientapi.common.systemconsumerstrategies

There are three choices here. DispatchToSingle, RoundRobin, and Pinned.

DisptachToSingle will prefer to always give messages to the first subscriber.

RoundRobin will prefer to move between the subscribers in a round robin fashion

Pinned will prefer to take a hash of the stream name and assign the message to the equivalent subscriber (eg messages from stream AStreamName will go to consumer 2).

Cheers,

Greg

6

Sorry copy and paste fail in second example I mean

On machine 2 I have subscriber B to stream1 group 1 (stream1:group1)

7

Ok got it now. The example you bring explain it completely. Thanks again! Will do some testing then.

Cheers!!!

sob., 12 paź 2019, 22:08 użytkownik Greg Young [email protected] napisał:

8

Just as a side note. Where I say “prefer” there by default it will go to the next one if the buffer for the first one is full. This is however IIRC configurable for whether this happens or whether it must go down the assigned one!

9

Hi Greg,

This is however IIRC configurable for whether this happens or whether it must go down the assigned one!

I’m wondering, where can I find such setting to configure? Competing consumers docs ( https://eventstore.org/docs/dotnet-api/competing-consumers/index.html?q=pinned#pinned ) warns us that “The main aim of this strategy is to decrease the likelihood of concurrency and ordering issues while maintaining load balancing. This is not a guarantee, and you should handle the usual ordering and concurrency issues.” So if there is a chance of events reaching subscribers out-of-order, would someone share any strategies which could help in this case – to avoid out of order application of events?

  • Karolis
10

You can never assume ordering with competing consumers as a message could get retried etc. You can tweak settings I believe to make it so it can’t happen but you likely don’t want to. You can also write code above to make it so it can’t happen but you likely don’t want to. If you need ordering use a catch up subscription.

Things like projections which require perfect should be using catch up subscriptions. Where competing consumers are useful are things like “I need to charge a credit card and I want to make it highly available with three processors”. Remember competing consumers handles things like retries which by nearly any definition throws out ordering even if not in the normal case…

11

Thank you for answer! It helped a lot.

I want to make sure that I understand correctly: in order to have highly available projection with (almost?) perfect ordering, we should use catchup subscriptions with some sort of leader election (or locking) on our side and probably add idempotency?

With competing consumers we get monitoring for free, with catchup we would need to build it ourselves, as there are no stats of subscriptions builtin into ES, right?

Thanks again.

  • Karolis
12

OK…

The catchupsubscription gives you the metrics its just pushing them to whatever (eg: there is information available in the client side) …

You don’t generally have leader elections etc there is no need … just use N. Example: you use 3 copies of a read model each with its own catchup subscription.

Idempotency will need to be handled for all, persistent subscriptions do not handle this for you. Its actually easier with a catchup subscription :open_mouth: Just remember atomically your last processed checkpoint and whatever change you made because of it.