Persistent Subscriptions & $all stream

EventStoreDB
1

I need to read all events chronologically

But I could not attach to $all stream

How can I read all events in all streams chronologically?

2

AFAIK you can’t. Next best thing is to create a projection that does a linkTo for every event.

3

You can't setup competing consumer on $all. As Joao states you can do
a link to. If someone really wants this we are happy to discuss doing
it but its just an optimization for the use case.

4

Our use case is the following.
I try to create event dispatcher for

  • projections

  • process managers

by projections I mean read model views which are built by processing events.

by process managers I mean special case of aggregates consuming events and producing commands or calling domain services and then proceeding to new state (they are state machines as well)

I used simple es-projection to create linked stream

fromAll().whenAny(function(s, e) { linkTo(‘all’, e);})

but sometimes projection fails, I already found another discussions on this group and the recommendation was to reset subscription

but when resetting subscription - all events are pushed to consumers again

I thought it will be more robust solution if there is native support for competitive consumers and $all stream

in this case we never will need to reset a es-projection and never will have pushed same event twice

(I am aware that some events still are pushed twice because how checkpoints are written)

Do you think I still need to use catchup subscription instead?

Is it a valid use case to use competitive consumers to scale-out PMs and projections processing?

5

"
I thought it will be more robust solution if there is native support
for competitive consumers and $all stream
in this case we never will need to reset a es-projection and never
will have pushed same event twice
(I am aware that some events still are pushed twice because how
checkpoints are written)"

This has been left off with it only on individual streams. If you
would really like it on all and are willing to pay for it this can be
done (internally stream vs all is totally different e.g.
checkpointing, read methods, etc)

"I used simple es-projection to create linked stream
fromAll().whenAny(function(s, e) { linkTo('all', e);})

but sometimes projection fails"

This is being resolved.

More in general. For your "processes" a competing consumer is probably
better. For projections would most likely want a catch up
subscription.

Cheers,

Greg

6

Thanks for quick responses
As far as projections issues are fixed, I will keep using linked stream

More in general. For your “processes” a competing consumer is probably

better. For projections would most likely want a catch up

subscription.

This makes sense - I am handling this by limiting max consumers on a subscription to 1 right now

7

"This makes sense - I am handling this by limiting max consumers on a
subscription to 1 right now"

I would just use a catch up then. Is the logic of writing/reading your
check point to a stream your issue or?

8

Just did not want to use another persistence service for storing checkpoint, but this is not much problem
I have chosen this subscription type because of parked messages/ checkpoint but can implement all of these myself

Thanks

9

Most projections shouldn't park messages.

10

So if a sequential event cannot be processed because of a bug in a projection, whole read model builder should stop?

11

In most cases... yes.

If building something generic.. yes.

Otherwise you are into specific state relation dependent logic.

12

Makes sense, will consider this
Thanks