OK, classic ‘it depends’ question, but would be good to hear others reasoning:
We’ve implemented our processmanagers as competing consumers. A processmanager handles multiple eventtypes, an eventtype is handled by multiple processmanagers. Atm the moment they all subscribe to a single stream (a projection with all domain events). This is clearly not the most efficient way to do it (lots of checkpoints written for events not applicable to a certain subscription)
I can see a couple of different solutions to this:
1. Create projection per pm with only the events it is interested in (not sure I want to maintain this)
2. Create a persistent subscription per pm/eventtype against $et-event. (lots of subscriptions, and no guarantees on ordering within each pm.)
3. ?
/Peter
“We’ve implemented our processmanagers as competing consumers. A processmanager handles multiple eventtypes, an eventtype is handled by multiple processmanagers. Atm the moment they all subscribe to a single stream (a projection with all domain events). This is clearly not the most efficient way to do it (lots of checkpoints written for events not applicable to a certain subscription)”
So two major questions here.
-
Why have a projection with “all events” why not just read from all? it will be much cheaper 
-
Why checkpoint events that are not applicable? The worst thing that would happen on a restart is you decide those same events are not applicable. Going through 10,000 such events might take a second or two, is this an issue?
“We’ve implemented our processmanagers as competing consumers. A processmanager handles multiple eventtypes, an eventtype is handled by multiple processmanagers. Atm the moment they all subscribe to a single stream (a projection with all domain events). This is clearly not the most efficient way to do it (lots of checkpoints written for events not applicable to a certain subscription)”
So two major questions here.
- Why have a projection with “all events” why not just read from all? it will be much cheaper
Tried that, creating persistent subscription to $all, but failed with “Invalid stream name” (or similar) so I figured it was not supported. Is there another way?
- Why checkpoint events that are not applicable? The worst thing that would happen on a restart is you decide those same events are not applicable. Going through 10,000 such events might take a second or two, is this an issue?
I don’t see a way to differentiate between handled and ’skipped’ events, Acknowledge is the only option? Or are you referring to changing the minimum checkpoints setting? (Running this as default now)
/Peter
Anyone?
Seems like this should be a common usecase?
(I’d be happy to ask these questions using the commercial support, but it seems they could benefit the product more being public)
/Peter
“Tried that, creating persistent subscription to $all, but failed with “Invalid stream name” (or similar) so I figured it was not supported. Is there another way?”
try replacing $all with %24all in the URL you’re requesting.
“Tried that, creating persistent subscription to $all, but failed with “Invalid stream name” (or similar) so I figured it was not supported. Is there another way?”
try replacing $all with %24all in the URL you’re requesting.
Using .Net client, does this still apply?
(pretty happy with being rid of $all anyway to be honest, EventPosition is clunky to handle in projections, having a streamposition makes it a lot easier to show progress etc)
/Peter
Last time I checked, you can’t have competing consumers on $all stream.
This has been something to be added for a while I can take a look at it it currently works on streams. Its actually not a large thing to add but has not had many use cases associated with it.
“I don’t see a way to differentiate between handled and ’skipped’ events, Acknowledge is the only option? Or are you referring to changing the minimum checkpoints setting? (Running this as default now)”
The ack is just that you are finished with an event, whether it was handled or skipped is not something being tracked by a subscription. An ACK here means “this event is completed” whether this means “nothing to be done” is not something tracked (we are only concerned with whether or not something has been delivered).
“(pretty happy with being rid of $all anyway to be honest, EventPosition is clunky to handle in projections, having a streamposition makes it a lot easier to show progress etc)”
$all can show progress as well. It just doesn’t know the number of events to come, it knows the size of data it still needs to process.
This has been something to be added for a while I can take a look at it it currently works on streams. Its actually not a large thing to add but has not had many use cases associated with it.
“I don’t see a way to differentiate between handled and ’skipped’ events, Acknowledge is the only option? Or are you referring to changing the minimum checkpoints setting? (Running this as default now)”
The ack is just that you are finished with an event, whether it was handled or skipped is not something being tracked by a subscription. An ACK here means “this event is completed” whether this means “nothing to be done” is not something tracked (we are only concerned with whether or not something has been delivered).
I get that, but surely it will increase load on EventStore acking a lot of events that are not relevant? (network, disk, io). So I’m curious about how others have dealt with this. We’re about to go live with EventStore in a multitenant environment, want to have some options in case optimisation is required.
“(pretty happy with being rid of $all anyway to be honest, EventPosition is clunky to handle in projections, having a streamposition makes it a lot easier to show progress etc)”
$all can show progress as well. It just doesn’t know the number of events to come, it knows the size of data it still needs to process.
Yep, but it’s not very accurate considering stats-events, scavenge etc….
/Peter