Pub/Sub best practices

EventStoreDB
1

Hi,

I am trying to figure out the best approach for using ES internal pub/sub capabilities.

I am using CQRS so I would like to update my read model out of process triggered on an event being written to a stream (write model). I have initially been looking into ways of subscribing to sort or a wildcard stream but this seems to be incorrect - I was looking for something like (listen where stream name = entity-*).

I have now started to look at:

  1. creating a persistent subscription when a new stream is created (before appending event) using CreatePersistentSubscriptionAsync(…)

  2. connect to the new subscription with ConnectToPersistentSubscription(…)

Then when my application is restarted I:

  1. read in all persistent subscriptions using List(UserCredentials userCredentials = null)

  2. register each via ConnectToPersistentSubscription(…)

This seems to be a much neater solution, but would like to confirm if this is inline with best practice. I do have one concern with this and that is, let’s say I have 1,000,000 streams, how do i go about managing all of these subscriptions? Surely I cannot register all of these at startup.

If anyone can point me to an online post/example that would be great, I have failed numerous times to find anything.

Thanks in advance.

2

You shouldn’t use that many persistent subscriptions. They are “heavy weight”.

This is the exact use case the category system projection is designed for. It generates an indexing stream of e.g. $ce-entity which contains all events in all streams matching entity-*. Have a single persistent subscription on that stream.

3

That’s exactly what I’m looking for. Docs look simple enough so will look into this when I’m back in the office first thing.

Thanks for the pointer.

4

I have been giving this model some more thought this morning which has led me to another question around application architecture.

Currently I have my read models being updated via a mediator therefore my updates are in process within the same api microservice. If I move to pub/sub I will need another service to subscribe to the stream and handle updates. This leaves me in a situation where I have two services connected to a single database, as I’ll be serving get requests from my read models. This is obviously bad practice as I need to take both services down if I needed to update a schema.

How have others handled this?

5

"This is obviously bad practice as I need to take both services down if I needed to update a schema. "

Why would you need to do that?

6

You’re completely right. Now that I’ve posted this I’ve realised a simple listener initialised on startup that registers callbacks is all that’s needed. Very simple.

Ive over thought it.

7

You can also update schemas in specific ways to avoid versioning issues of the internal events between versions of software. Free to read online: https://leanpub.com/esversioning but this topic is discussed in it.

8

Thanks Greg.