Is there a limit to the number of client subscriptions?

Clients
#1

We are using the .NET Core GRPC client and have some subscriptions set up using the Subscribe() method.

Is there a limit to the number of subscriptions that can/should be created? If I have hundreds of event types, can a single application have individual subscriptions to each of those event types?

#2

There’s no hard limit, but your database server will choke as it won’t be able to serve so many subscribers at some point. To go beyond the question itself, I’ve never seen a use case of having tons of subscriptions, where each of them is handling a single event type. Maybe you can explain a bit more about your design?

#3

@alexey.zimarev, thanks for responding. Here’s the scenario we are working through using the C# GRPC client:

Today: The service for Bounded Context A listens for 5 event types in a subscription. The service for Bounded Context B listens for 3 event types in a subscription. Both bounded contexts store the last position read so that when services restart we don’t reprocess those events.

Next month: Bounded Context A needs to listen to 2 additional event types and needs to start the subscription at position 0. Right now we are creating additional subscriptions in each service when new event types need to be listened to.

Questions:

  1. Is it best to do some type of catch-up and allow each service (bounded context) to have a single subscription to the event store?
  2. I’ve seen some references to “catch-up” subscriptions but I haven’t yet found any good reference implementations. If catch-up is the right approach, is there any documentation describing how to do a catch-up and then how to transition from the catch-up back into the normal subscription?

Thanks!

#4

Could you explain what do you mean by a “normal subscription”?

#5

Sorry for the confusion. By “normal subscription”, I just meant that the new event would be included as part of the single subscription running within the service for the bounded context after the catch-up process has completed.

#6

It is the same subscription. It catches up on history and then becomes realtime. It’s described in the docs:

When the subscription starts for the first time and the stream it subscribes to already has events, the subscription will get into a catch-up state and receive historical events. When the subscriber eventually catches up and processes all the historical events, it will switch to real-time mode and will get events as they are appended to the stream. If the stream gets more events that the subscriber can process in real time, the subscriber will lag behind and switch to the catch-up mode again until it manages to process all the pending events and then switches to real-time mode again.

The gRPC subscription documentation is available.

Here you can find my implementation of a subscription to $all.

#7

Thanks @alexey.zimarev for your patience while I try to describe my questions.

We have a very similar implementation based on your book. The problem I’m trying to resolve now is how to handle events that the subscription wasn’t previously handling. Since the subscription stores the checkpoint, once the subscription has run at least once, it will only pay attention to future events.

Our system will have hundreds of different event types across 20+ different bounded contexts, each potentially running in a different microservice. I’d like to come up with a solution that allows us to catch up with event types that a bounded context wasn’t previously handling while still minimizing the number of subscriptions to EventStore that we have.

#8

I don’t bother that much about the number of subscriptions. Do you have performance issues due to the number of subscriptions, or is it a precaution? If you have a projection that needs to handle events, which it wasn’t handling before, it needs to run from scratch. The way to do it is to make a new version of the projection, then run it for an empty database, where you also store the checkpoint, then switch the old one to the new one when they come on par. I rarely (never) have seen a complex system with one subscription only. Also, never felt a need to really limit the number of subscriptions, if we are talking about tens of them. Subscriptions can use follower nodes or even read-only replicas, so there’s no additional load for the leader.

#9

Thanks @alexey.zimarev! We’ll go down the path of switching from the old to the new subscription once they are on par. The advice is much appreciated.

1 Like