How expensive is it to have million(s) of stateful projections?

EventStoreDB
1

Hi guys!

I’m not even quite sure that I need event sourcing in my situation (I also looked at Sheduler-Agent-Supervisor here), but now I’m wondering if my usage of projections is acceptable.

So, I’ve got Trips, Groups, Users, and Places in my system. Each trip has a route of Places and belongs to some groups. The trip is created or updated rarely, queries like ‘give me all Trips for some Place|Group filter by Trip parameters order by Trip date’ are very frequent. That’s why I’m going to keep Places and Groups in a document database and keep a list of Trips (only tripID and filterable parameters) for each Place|Group.

Now about projections. There will be a stream for each Trip. And a projection for each stream from category ‘trip’. The projection will keep track of current state of the Trip, at least most important parameters and list of linked Place and Group ID’s. When new Place is added to Trip, an event is emitted by this projection to ‘places’ stream containing current parameters of the Trip. When some parameter changes the event is linked to streams of all interested Places|Groups (the list of interested Places|Groups is contained in projections state). When a Place is removed from Trip, the event is linked to that Place’s streem. Then separate services subscribe to streams of places|groups and update their read models (in MongoDB).

The question is how expensive is it to keep million(s) of projections, where each projection holds (partially) the current state of the aggregate and can emit dozens of events upon each received event (which though happens not often)?

2

Hi Artem,
It doesn’t seem like you’ll have millions of projections, but rather possibly that many streams flowing through a single projection.

You can think of a projection as being a set of filters and then once a stream, event falls into a particular bucket, you can perform an operation on that said event.

So for instance

fromCategory(‘trip’)

.foreachStream()

.when({

$init: function(s,e){

   return {

       count: 0

       //other parameters

   }

},

placeAdded: function(s,e){

   s.count += 1;
   linkTo('places',e);

}

})

``

You can then get the state for each projection via

curl -i http://localhost:2113/projection/tripState/state?partition=trip-913826c6-84a5-4558-8b93-352ba8e929dd -u admin:changeit

``

Rather than trying to explain how you will be using projections is to explain the problem you are trying to solve and then we could possibly guide/provide you with a solution be it using projections or something else?