How do you guys deal with situations where your read model is only interested in some events from the stream, yet you want to enforce that they are applied in order. For example if status is changed multiple times you want to get the last one for example. Do you subscribe to every event from stream and just bump version when there is nothing to do?
The first question is if this is a problem that you experience and need to solve, or if you are just speculating that it would be.
The second question is then, how would you determine if you are interested in an event without looking at it?
The answer to your initial question is that our event handlers read all events and just discards the uninteresting ones.
Regards,
Martin
Yes this is a problem I am currently facing. I have 2 streams, and for this projection I am only interested in some of the events, not all of them. If I understand you correctly you create empty event handlers that just bump up version, for the events that are not interesting for particular projection? So I guess, you have to keep track of version of events for each stream that is part of that projection?
All event handlers read all events from a combined stream. Depending on your actual architecture, number of events etc this could be expensive, but until it is to expensive, that’s a good solution.
I’m really new to all this so I may be wrong, but couldn’t you create a new stream using a projection which links the events you care about?
I guess this is unnecessary in a small stream and would add overhead but if there were a lot of events in the stream.
For example, I need to look at the $all stream but want to filter out system events, so I found this on the ES Github -
fromAll().
when({
$any : function(s,e) {
if (e.eventType && !e.eventType.startsWith(’$’))
linkTo(‘allNonSystemEvents’, e);
}});
You could tweak that to get a dedicated stream of the things you care about with its own index.
Maybe we have out terminology mixed up. I consider “event handler” a piece of code that handles specific event type, a method. Are we talking about the same thing? I am asking because we have about 30 mil. events right now and about 150 distinct event types. I may need only 5 event types to respond too. I cannot believe that you are suggesting that I have to chew through all events.
And how would that event help me with ordering anyway? Each stream has its own event numbers within a stream. I am obviously confused. I know I can create a projection from $all, and filter out ES events “$”.
Right now we have stream per aggregate, such as $ce-order, $ce-customer etc. For my projection I am interested in only these two streams, not all other aggregates that we have.
Yes, that is doable, but I read somewhere (Greg said it) that you should not have many projections in ES. And then I am not sure that that would even work, because if you have slightly different requirements, where you need one more events, or maybe don’t need, you would have to create another projection and so on, as not to have gaps.
Yeah, I mean you either filter some out or you look at them all, you can’t have it both ways can you? You could partition based on an attribute of the event rather than the event type couldn’t you? I am just thinking to the chat room example Greg gives where he partitions the streams based on user, across the chat aggregates (so you can see all chat by Person A, from every stream)
Or, considering you can use the built in Category projection to see all events for an aggregate type, can you not make a projection that draws from two categories ($ce-order, $ce-customer in your case)?
I may be off the mark, as I said I am very new to this stuff but that would make sense to me.
That might work. If I had a persistent subscription to that stream, then for the events that I am not interested in, I just acknowledge them as processed, I don’t have to have dedicated event handler. Then I can just enforce that next event is after the previous. I guess ES will not send next event until previous is acknowledged ?
Or maybe it would still be better to have handlers that just bump up version?
The easiest thing to do here is subscribe to all and ignore messages you don’t care about. If scaling out is a problem for you, use the atom feed instead.
Ok, so let’s say I subscribe to “$all” stream, or create allNonSystemEvents projection. Events are being delivered in the order, now on one of the updates fails, and message ends up in parked stream. What now?. I have option to stop applying or continue and replay later. But just because update for one customer failed, it does not mean I stop it for all customers. Now I need to keep track if event related to particular customer has failed, and nak all related?
Don’t use persistent subscriptions for read models. They are meant for a different use case (competing consumers).
Typically in this scenario my personal preference is to suspend the projection on a non transient error e.g. NRE. Fix the bug and redeploy. If you are persisting the checkpoint transactionally with your projection it’ll just pick up where it left off.