Persistent Subscription Limits

EventStoreDB
#1

I’m new to EventStore and have been playing around with some features, mainly persistent subscriptions.

I’ve been experimenting with creating large numbers of persistent subscriptions - I have a figure of 20,000 in mind. I’m experimenting with this feature to explore server-side check-pointing for stream consumers (removing the need for clients to do this themselves). I am mainly experimenting with creation of subscription groups, rather than using them at this stage. What I’ve found so far is that maintenance of the persistentSubscriptionConfig stream is a limiting performance factor. Every time a persistent subscription is created a snapshot of the configurations for all subscriptions is saved to the stream (I believe this stream is read from the tail when ES starts up to get the system configuration for persistent subscriptions). As large numbers are created this process gets slower and slower.

Just wanted to know if I’m barking up the wrong tree, and trying to create far more subscriptions than is sensible. Have been looking at ways of optimizing things.

Thanks, Tim.

#2

It was never intended to have 20,000 concurrent subscriptions. Maybe
you can discuss your use case a bit?

#3

Hi Greg,

This is probably a cruel and unusual use of Event Store, but experimenting with different technologies and designs.

The use case I’m playing with is command distribution to processing nodes (aggregates). The basic idea is to create streams per aggregate and then select a node and give it the stream details so it can start processing it. The persistent stream comes into play as a way of check-pointing where a stream has been processed up until so that the stream can be moved to a different process - a rebalancing and failover algorithm. The large number represents a large number of aggregates. The message rate per aggregate would be low, and they would not all be processing concurrently.

Just playing with persistent subscriptions for now.

Thanks.

#4

Maybe I am missing something but I can't come up with any reason why I
would want a persistent subscription (e.g. a queue in rabbitmq) per
aggregate. This just sounds like a bad idea. What would I gain by
having this?

"The persistent stream comes into play as a way of check-pointing
where a stream has been processed up until so that the stream can be
moved to a different process"

I can kind of get a queue per process .. Just don't see any advantage
to queue/aggregate

#5

The queue per aggregate does result in a large number problem, and probably makes it untenable.

The reason for experimenting with this over a queue per process, is that it makes moving processing of individual aggregates to another process easier. The aggregate stream is check-pointed and in a known state so it’s easy for something else to pick it up. A queue per process (which may have commands for many aggregates) seems much more difficult to deal with, the large number problem probably rules out the simpler per aggregate queue.

The real intent is to maintain many checkpoints, rather than many concurrent subscriptions.

#6

Actually I guess with a check-pointed queue per process, it’s not too difficult to shred the queue on node failure and copy the commands to other queues as I’d know what had been processed.

One remaining thing the aggregate queue gives is that they are independent and not susceptible to the situation where a process level queue has a high number of commands in series for one aggregate instance and blocks command processing for the other aggregates (I am making assumptions here though on how per aggregate queues may work out in reality).

#7

I want to follow-up on this old thread since we are planning to use persistent subscription a fair bit and want to know its limitations.

In short, we have aggregates that emits events and we often create/update read models in SQL Server using “event handlers”. These “event handlers” are just .net core processes that creates (during runtime) and listens to a persistent subscription.

Having 1 persistent subscription per event handler means that each “event handler” can have their own separate error handling that doesn’t affect other event handler. For example, if one of the event handler had trouble with an invalid message (for itself only), then we can stop the subscription until this is fixed.

So far we have 3 systems using this pattern with around 20 or so of these “event handlers” so naturally 20 or so persistent subscription all on the same ESDB instance. If we replicate this pattern moving forward, I can see our ESDB to have potentially 100s or maybe even a 1000 or so of these kinds of persistent subscription.

I’m not sure whether this is a good use of persistent subscription as some on the forum has suggested that it is not meant to have a high volume. I’ve seen people suggested only having 1 process to read the $all stream but I’m not sure what the pros/cons are.

Does our use case make sense?

#8

Hey @stephen_tung,

As rule I don’t use persistent subscriptions for updating read models. I just subscribe to the $all stream and handle checkpointing myself outside of EventStoreDB. I do this by storing the position of the last event processed.

These kind of subscriptions are a lot more lightweight and will have no problem scaling

#9

mat,

Something caught my eye with your reply.
My understanding is, if you use a persistent Subscription over multiple processes, Eventstore will make sure duplicate events are not being delivered to the subscribers.
If your own process subscribed to $all (maintaining last processed event yourself) what what would happen if you needed to added another subscriber in (to try and power through a load of data for example).
Surely that would start from the same position as currently stored, and you would have two subscribers essentially reading the same events?

Most likely, I have missed something here, but would be good to understand this.

#10

Each projection should be maintaning its own checkpoint. This should be orthogonal to how many physical subscriptions you have.

#11

Projection maintaining it’s own checkpoint I get, but it’s the subscription part of it I can’t understand how it would scale.
Just trying to understand If I am missing a trick here (our systems all heavily rely on persistent subscriptions, for the most part, scalability (at least in my head)

#12

There are lots of different ways to solve this.

The easiest thing to do is one subscription per projection. If you have a small number of projections this can work. If you have hundreds then maybe not.

The next thing to try is to group multiple projections together under a single subscription. You can of course do all projections in one group - but then your throughput is limited by the slowest projection. A happy medium would be to partition them in some way. Maybe you group related projections together (e.g. blog posts and blog comments), or maybe you put slow projections (e.g., full text search) in their own group. You will have to measure and figure out what is best for your system.

#13

Maybe something more for me tot think about.
How you described it, it sounds like exactly what Persistent Subscription do perfectly.

#14

Not quite. There is no ordering guarantee with persistent subscriptions. Dealing with out of order messages in projections is not fun… you would need to stick a resequencer in front.

#15

For our core system, ordering (for now at least) is not a problem.