Projections appear to be skipping some events


  • Event Store 3.9.3

  • on a VM running Ubuntu Server 14.04 LTS

  • Projections enabled


We have many Widget-{UUID} streams and a couple projections for creating derivative streams:

One creates a single “Widget.all-changes” stream from certain high-level events for all Widgets:




WidgetCreated: function(s,e) { linkTo(“Widget.all-changes”, e); },

WidgetAltered: function(s,e) { linkTo(“Widget.all-changes”, e); },

WidgetDeleted: function(s,e) { linkTo(“Widget.all-changes”, e); },

WidgetVersionCompleted: function(s,e) { linkTo(“Widget.all-changes”, e); }



Another creates a derivative “header” stream for each Widget, capturing same high-level events:

function headerStream(e) {

var id =;

var headerStream = “Widget.header-” + id;

return headerStream;





WidgetCreated: function(s,e) { linkTo(headerStream(e), e); },

WidgetAltered: function(s,e) { linkTo(headerStream(e), e); },

WidgetDeleted: function(s,e) { linkTo(headerStream(e), e); },

WidgetVersionCompleted: function(s,e) { linkTo(headerStream(e), e); }




Started from brand-new, blank Event Store. Populated Widget streams by running a mass migration script from an older SQL Server database into Event Store. But we first enabled all projections, including the above two. So projections were running when the migration occurred, writing events into all the individual Widget-*** streams.


For many of the Widgets, the initial WidgetCreated event in their stream is straight-up missing from the “header” stream. For example, in stream Widget-e6a4de9d-cfa6-4ab1-97dc-5a3573ef638d, event 0 is (as it ought to be) a WidgetCreated event. In the stream Widget.header-e6a4de9d-cfa6-4ab1-97dc-5a3573ef638d, that WidgetCreated event is just not there. The first event in that projection-created stream is 10853@Widget-e6a4de9d-cfa6-4ab1-97dc-5a3573ef638d, which is a WidgetVersionCompleted event. Based on the projection code, I would have expected that 0@Widget-e6a4de9d-cfa6-4ab1-97dc-5a3573ef638d : WidgetCreated would have/should have been the first event.

Furthermore, this crucial WidgetCreated event is also missing from the unified projection stream Widget.all-changes. It’s just not in the stream; I checked.


Am I not understanding something about how projections work? Is there an eventual-consistency guarantee - i.e. will those missing events show up later? Or are we looking at a bug? Or are projections not guaranteed to include every event they’re supposed to include?

Obviously the answer to these questions will inform whether we are able to use projections at all in our solution.

  • How is the migration writing these events?

  • Have you checked the source streams that these events exists?

Update - I ran the whole migration a second time, but this time I did not enable projections (system ones nor our two custom ones) until the migration was complete.

Doing things this way, we do appear to have the missing events where they’re supposed to be. Not that this is really any comfort, because when our solution is running live, we need these projections continuously running and not skipping any events. But I include the information in case it helps.

  • The migration is using the .NET client; it’s a .NET Framework console application. Therefore I think that means it’s using a TCP connection? The entire program is single threaded.

  • As I wrote above, in the single case I was highlighting I started by checking the source stream and confirming that the WidgetCreated event was there, but was not in either projection-derived stream.

Why are you using foreachStream when you don't have any state? Are
these the actual projections or just something similar that you put

These are the actual projections. I was using foreachStream() because I built these projections by following code samples I found scattered across blog posts on your site.

Can you explain what foreachStream() is for and why I wouldn’t need to use it in my case? What should my code look like instead?

.foreachSteam partitions state so you have a state per stream
(projections can accumulate state is the s in your function(s,e)). If
you don't have state you don't need it.It is described here:

Oh, good! I hadn’t realized that the 4.0.0 projection docs were online - will enjoy reading them…

For now, do you think that my unnecessary call to foreachStream() is causing this problem?

It shouldn't though its curious if it makes any difference. I know
Pieter has quite a few fixes in 4.0 as well for projections

Ok, I am going to re-run the migration using method 1 (start projections first, then run migration) with the “foreachStream()” calls removed from the projection code. Will report back shortly.

Confirmed that, even without the “foreachStreams()” call in both projections, it fails exactly as above.

That one example widget I was using above: the same widget has its projection stream data malformed in the same way.

Another perhaps-useful metric: in the “correct” run (where I enabled projections after data was already in the streams), the Widget.all-changes stream had over 3000 events in it. Current total for this run is 2,671. Same data set in all cases, same migration logic.

Would you be willing to share your logs and database with us?
Also, does resetting the user defined projections fix the issue?

Getting the exact same failures seems really unlikely as most the
issues I have seen are usually timing senstive etc. Pieter likely has
more to say.

Reply to Greg: I don’t mean to imply I think it’s the exact same failures; it’s the widget that our solution falls down on first (when the service attempts to read the malformed projection streams). The solution died on the same widget both times. That’s all I know for sure.

Reply to Pieter: I would be willing to share logs and data, yes. Would you PM me so we can work out details?

Further reply to Pieter: - what did you mean “resetting the user-defined projections”? In between test runs, I am fully wiping the Event Store and starting from an empty data directory in /var/lib/eventstore.

On Pieter’s advice we re-ran our migration without using a transaction (i.e. calling IEventStoreConnection.AppendToStreamAsync instead of EventStoreTransaction.WriteAsync).

This appears to have fixed the issue.

Pieter can you provide more information about why using transactions caused this issue?

there is already a PR for it