Fixing up System Projections

Hey Guys, I’ve made a bit of a mess of our System projections - I was attempting to upgrade to use a local build of the dev branch, and didn’t upgrade the projections correctly. They all got into a Faulted state because of ‘Constructor not found’. (Only now did I read that I must first disable all projections before upgrading).

Anyway I though I could just delete them all and then they would get recreated on a restart, which didn’t work, so after looking at the code, I manually deleted all the $projections-* and $projections-$all streams. This successfully recreated them but they all are in a Faulted state again with the error:

[PID:10432:015 2014.07.02 09:51:35.850 ERROR QueuedHandlerMRES ] Error while processing message WRITE COMPLETED: Corr

elationId: 480f8eaa-ec94-40da-bdc4-f52c8d60852a, Result: Success, Message: , FirstEventNumber: 5, LastEventNumber: 5 in

queued handler ‘Projections Master’.

System.Exception: Projection version and event number mismatch

at EventStore.Projections.Core.Services.Management.ManagedProjection.WriteCompleted(WriteEventsCompleted message, St

ring eventStreamId) in c:\code\EventStore\src\EventStore\EventStore.Projections.Core\Services\Management\ManagedProject

ion.cs:line 718

at EventStore.Core.Messaging.RequestResponseDispatcher`2.Handle(TResponse message) in c:\code\EventStore\src\EventSt

ore\EventStore.Core\Messaging\RequestResponseDispatcher.cs:line 64

at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message) in c:\code\EventStore\src\EventStore\EventStore.C

ore\Bus\MessageHandler.cs:line 33

at EventStore.Core.Bus.InMemoryBus.Publish(Message message) in c:\code\EventStore\src\EventStore\EventStore.Core\Bus

\InMemoryBus.cs:line 324

at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o) in c:\code\EventStore\src\EventStore\EventStore.Cor

My fault they all got screwed up, but anyway is there an easy way I can just reset all the projections to start from the beginning again (without having to clear all of my data?).

Oh actually I just looked at the newly recreated System Projections in the Web UI again and they have the same error I originally had:

“Constructor on type ‘EventStore.Projections.Core.Standard.IndexEventsByEventType’ not found.”

I’ll try going back to 3.0.0rc2, see what happens.

I’ve managed to fix this by changing the constructors of the system projections to accept Action<string,object[]> as the logger:

Looks like the issue was introduced in this commit:

Created a pull request for that above fix, but my streams are Faulted again when I start back up:

[PID:07904:017 2014.07.02 11:22:55.935 ERROR ProjectionManager ] The ‘$streams’ projection faulted due to 'Multiple p

rojections emitting to the same stream detected. Stream: ‘$streams’. Last event projection: ‘2’. Emitting projection:


I’ll probably just clear down the database and start from scratch if I can’t sort out these projections. It’ll be a pain to restore all that data but at least will get me moving again!

We were seeing a similar behaviour on a projection restarting the other day when playing with on in London. @yuriy have you seen this before?

This actually has to do with the cleanup of the logging interface by the sound of it… Guess reflective magic doesn’t get picked up by Resharper…

The multiple writing to a stream when it wasn’t? We saw the same on Szymon’s laptop in London on a restart though there was only one projection writing to the stream

I guess it’s something to do with me having deleted those system projections (or at least attempted to) and the old checkpoint still hanging around.

if (_projectionVersion.ProjectionId != parsed.Version.ProjectionId)




“Multiple projections emitting to the same stream detected. Stream: ‘{0}’. Last event projection: ‘{1}’. Emitting projection: ‘{2}’”,

_streamId, parsed.Version.ProjectionId, _projectionVersion.ProjectionId));



No, the reflective creation of the system projections - not sure on the multiple projections.


@James I’ve fixed that issue with the logger now (see pull request)

Now more errors! (See above)

I basically just want to delete all traces of the old system projections and have them recreated from scratch

Thats a bit more difficult than it sounds because they write to many streams (possibly millions)

OK cheers Greg, I’ll probably just start from a clean database, then I can just ‘manually’ copy across the streams I need from the old instance. It’s not all data I need anyway.

Anyway the reason I was mucking about with the dev branch in the first place was to see if I could figure out that other issue on github, since it’s blocking my current work. We do some secondary indexing that subscribes to a category event stream, and that is not working because of that issue.

Ah so I know where the problem is with that one. The code in the atomcontroller and I presume in TCP is not handling properly the case where the ResolvedEvent on ReadEventsBackwards etc is null but it is a link. The fix there isn’t too tough mostly need to write tests around it and implement (prob 1-2 days worth of work). I will likely hit that up next week (in Italy this week and home next)