Competing consumer subscription to a category

EventStoreDB
1

Hi,

We are considering using EventStore in production, and I am pretty impressed, but I do have a concern that I can’t seem to find an answer for (sorry if this is somewhere I have spent about an hour searching before posting!).

We have an event driven system today (using NServiceBus over MSMQ) and make use of the ability to have multiple workers processing events in parallel, at first glance it seems that a competing consumer subscription would be useful for this as that would allow us to have N processors each responding to an event that requires additional processing.

Is it possible to create a competing consumer subscription to a specific event within a category (i.e. all “OrderAccepted” events within the “Order” category so that we could start an asynchronous billing process?

My understanding is that this would require the use of a projection where each of the events from the individual streams are linked?

If so, would this be subject to the stream limit of Int.MaxValue?

Would there be any way of recovering if our system had sufficient events to raise this projection to near that level of events?

If we were to remove older, no longer necessary events would these still count against the limit?

Given the volume we are expecting a total of Int.MaxValue events is not unreasonable across all event types and streams in a category - is this an issue?

Regards,

Peter Stephenson

2

I can help answer a couple questions -

There is a system project $by_category that you will probably want to use. Basically if you save all your order-related events under the stream “order-[Guid]” or customer events under “customer-{Guid}” the category projection will create a stream of all streams starting with “order” and “customer” which you can subscribe to. If you want more fine-grained control you’ll have to write your own projection.

Something that might be of interest to you since you are using NSB is some code I wrote to pull all the event types an endpoint is listening for and create a projection for just those types. Allowing the consumer to receive just the events from ES that he cares about. Here’s the code https://github.com/volak/Aggregates.NET/blob/master/src/Aggregates.NET.Consumer/Internal/EventSubscriber.cs#L109

Im not sure but I think projections are limited by Int.MaxValue - and I dont think deleting previous events will help with that either.

3

Thanks :slight_smile:

Would the $by_category projections also be limited to Int.MaxValue? That’s a slight concern to me, to be honest! :S A single stream would get no where near that but we will have a lot of streams in the same category :s

Peter