I’m trying to implement some buffering and batching for catch-up subscriptions so that I can hopefully improve my processing performance.
My plans are to set up subscriptions to the all events stream, depending on the positions of the read side projections (cqrs/es) and utilise what TPL Dataflow offers to implement buffering and batching.
The areas I’m trying to improve are determining if an event is used in my projection and the actual deserialisation - I estimate to get a performance increase if I can transform multiple events in parallel (which I will test to verify).
How can I determine if an event in a catch-up subscription is currently the last event in the stream? Are there some properties or other information available?
I could not toggle this depending on liveProcessingStarted, because I might be able to buffer and batch during live-processing as well - if my projections are slower than the speed at which new events are generated.
Slow projections might be things like generating data for a DWH, where eventual consistency could be expected to be more relaxed.