We currently have projections which filter events of interest for different denormalisers to process. We record the position in the stream of the last event processed by the denormliser in a DB. This allows us to work out how far behind the head of the stream the denormaliser is, by querying eventstore for the stream and seeing the index at the head and comparing it to the index in the DB, and we can use this to get notified if a consumer is slow, or stopped processing for some reason.
To avoid write amplification of the projections we are moving away from this model and instead having the catch-up subscriptions subscribe to the $all stream instead and filter on the client. This requires us to not store the event index, but the commit position and prepare position in the $all stream instead.
We are now not sure how we can workout how far behind the head of the $all stream we are, as the commit position and prepare positions are not sequential.
Any ideas on how we can solve this