Persistent subscription to $all events

EventStoreDB
#1

Hi,

We are planning to create read side using persistent projections (v5.0.2.0).

I am trying to create persistent subscription to all events using “$all” stream, and getting error Subscription group testGroup1 on stream $all failed 'Bad stream name.’ with following code:

var connection = EventStoreConnection.Create(new Uri("tcp://admin:changeit@localhost:1113"), "testingsubscriptions");
await connection.ConnectAsync();

var settings = PersistentSubscriptionSettings.Create()
    .StartFromBeginning()
    .WithMessageTimeoutOf(TimeSpan.FromMinutes(2))
    .WithMaxRetriesOf(3)
    .Build();

await connection.CreatePersistentSubscriptionAsync("$all", "testGroup1", settings, new UserCredentials("admin", "changeit"));

The same issue happens while trying to create a persistent subscription using admin web GUI.

My assumption is that “$all” is not actually a stream or not a stream we could subscribe to, if that is the case I have a question. Can we subscribe (with persistent subscriptions) to all events in another way? If not, would that mean that subscriptions are possible only to continuous projections and single streams?

#2

You generally would not want to use a persistent subscription for a read model. There are cases where messages can be delivered out of order. Generally you want to use a catchupsubscription to $all for your projections to a read model. Order tends to be very important for projections :slight_smile:

As of now I believe persistent subscriptions are only stream based though it would not be a huge thing to add and seems worthwhile. It was discussed being added some time ago …

1 Like
#3

Hi Greg,

Thank you for the prompt response.

We want to use multiple instances of read service for performance and redundancy reasons. Messages out of order aren’t problematic in this case, we can handle out of order (and duplicate) messages. As I understand only persistent subscriptions can support multiple consumers (with catch up subscriptions each instance needs to store position for itself).

We will investigate persistent subscription to projection stream. Another option could be to negotiate between read instances and determine which instance is “primary” and the only one subscribed to catchup subscription, but that solution that could be messy.

#4

If you want multiple instances of your read model, the best thing to do is have multiple subscriptions to $all. PersistentSubscriptions are really meant for external services that have side effects - e.g. talking to a payment processor or sending emails.

#5

Hi João,

Thank you for the response.

At the moment we have two instances of same read service that are subscribed via MassTransit/ASB to certain event types. MassTransit/ASB is balancing message delivery to one or the other instance which processes the message and writes the result to SQL db (both instances are writing to the same db).

We want to remove MassTransit/ASB dependency for this service. If we add catchupsubscription to all instances it means that every instance will process all messages, so every message will be processed at least twice (2 instances of the same service), one instance will write to DB, and the other will skip the message because of idempotency check.

That’s the whole reason why we wanted to use PersistentSubscriptions for read service, so we can laod balance the messages and have redundancy in case one instance stops working. If we decide to use catchupsubscriptions, we would probably have to run only one instance of the service.

#6

Hi,

Rather than having two instances, what you could do is use a catch up subscription and when you process an event successfully, write
a payload with the last processed event number to some other data store (I’ve used both Redis and a second EventStore stream for this in the past).

Doing this will allow you to pick up from the last processed event should you get any issues.

The other thing to consider is whether you really need all events.

If you don’t and can categorise them in some way, you may be able to look at:

https://eventstore.org/blog/subscription-service/

I hope this helps.

Sean.

#7

Thank you Sean for your input,

We don’t really need all events to be processed by read service, just some types of events (about 20% of all events), that’s why we are considering to create a continuous projection that will write to separate stream and then subscribe to that stream.

Regarding the third party service (from the link) it looks interesting, however, we would like to avoid HTTP requests, during peaks we could have hundreds of thousands of messages in the bus waiting to be processed.

Catch up subscription in an option we are considering, Zimarev demonstrated an example of catch up subs I saw earlier today on YT.

#8

In the general case, an individual read model will only care about a small percentage of events emitted out of your system. If you are concerned about having way too many subscriptions to $all
you have a couple of options:

  1. use the atom pub endpoint and put varnish / nginx / squid / etc in front

  2. if you have low latency requirements here, you can opt for a single physical subscription to $all (per machine) than fans out to your read model handlers on the inside.

#9

Only part of the system is using ES, and we will have only two read models, we will go for two subscriptions to $all for now.

Thank you João