built-in $by_event_type projection not getting all events

we have many streams where the same event type is written (i.e. some feeder streams). We use the $by_event_type projection to consolidate this in another stream of ($et-ItemEventsPublished).

Last night/early this morning we noticed that several events were not there, and we have a ‘gap’ in the consolidated stream.

So we see an event written 2/8/2016 @19:01 (event 12) in the $et-ItemEventsPublished stream

Next event written to this stream (event 13) is on 2/9/2016 @ 04:01, and then we have another ~11k after that.

However, it’s at least missing ~1000k events that arrived on 2/9/2016 @ 00:01 that arrived in one of these 'feeder" streams.

The projections all look good (i.e. all caught up and not in a faulted state).

I shut down GES and backed up the database directory, but I’m do not know how to proceed, both to recover and understand why this happened.

This is using GES 3.0.3.0 on a windows 2012 r2 vm.

Thanks,

Karl

added an issue in github github.com/EventStore/EventStore/issues/803 to help track.

Thank you, it’s better to keep track of issues in github anyway.