Fixing up System Projections

EventStoreDB
1

Hey Guys, I’ve made a bit of a mess of our System projections - I was attempting to upgrade to use a local build of the dev branch, and didn’t upgrade the projections correctly. They all got into a Faulted state because of ‘Constructor not found’. (Only now did I read that I must first disable all projections before upgrading).

Anyway I though I could just delete them all and then they would get recreated on a restart, which didn’t work, so after looking at the code, I manually deleted all the $projections-* and $projections-$all streams. This successfully recreated them but they all are in a Faulted state again with the error:

[PID:10432:015 2014.07.02 09:51:35.850 ERROR QueuedHandlerMRES ] Error while processing message WRITE COMPLETED: Corr

elationId: 480f8eaa-ec94-40da-bdc4-f52c8d60852a, Result: Success, Message: , FirstEventNumber: 5, LastEventNumber: 5 in

queued handler ‘Projections Master’.

System.Exception: Projection version and event number mismatch

at EventStore.Projections.Core.Services.Management.ManagedProjection.WriteCompleted(WriteEventsCompleted message, St

ring eventStreamId) in c:\code\EventStore\src\EventStore\EventStore.Projections.Core\Services\Management\ManagedProject

ion.cs:line 718

at EventStore.Core.Messaging.RequestResponseDispatcher`2.Handle(TResponse message) in c:\code\EventStore\src\EventSt

ore\EventStore.Core\Messaging\RequestResponseDispatcher.cs:line 64

at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message) in c:\code\EventStore\src\EventStore\EventStore.C

ore\Bus\MessageHandler.cs:line 33

at EventStore.Core.Bus.InMemoryBus.Publish(Message message) in c:\code\EventStore\src\EventStore\EventStore.Core\Bus

\InMemoryBus.cs:line 324

at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o) in c:\code\EventStore\src\EventStore\EventStore.Cor

My fault they all got screwed up, but anyway is there an easy way I can just reset all the projections to start from the beginning again (without having to clear all of my data?).

2

Oh actually I just looked at the newly recreated System Projections in the Web UI again and they have the same error I originally had:

“Constructor on type ‘EventStore.Projections.Core.Standard.IndexEventsByEventType’ not found.”

I’ll try going back to 3.0.0rc2, see what happens.

3

I’ve managed to fix this by changing the constructors of the system projections to accept Action<string,object[]> as the logger:

https://github.com/betgenius/EventStore/commit/e70a5966108a0a71a0e12815f8292b3d8af075cd

Looks like the issue was introduced in this commit: https://github.com/betgenius/EventStore/commit/d500282468584a631a01bf33103497eab69e905b

4

Created a pull request for that above fix, but my streams are Faulted again when I start back up:

[PID:07904:017 2014.07.02 11:22:55.935 ERROR ProjectionManager ] The ‘$streams’ projection faulted due to 'Multiple p

rojections emitting to the same stream detected. Stream: ‘$streams’. Last event projection: ‘2’. Emitting projection:

‘9’’

I’ll probably just clear down the database and start from scratch if I can’t sort out these projections. It’ll be a pain to restore all that data but at least will get me moving again!

5

We were seeing a similar behaviour on a projection restarting the other day when playing with on in London. @yuriy have you seen this before?

6

This actually has to do with the cleanup of the logging interface by the sound of it… Guess reflective magic doesn’t get picked up by Resharper…

7

The multiple writing to a stream when it wasn’t? We saw the same on Szymon’s laptop in London on a restart though there was only one projection writing to the stream

8

I guess it’s something to do with me having deleted those system projections (or at least attempted to) and the old checkpoint still hanging around.

if (_projectionVersion.ProjectionId != parsed.Version.ProjectionId)

{

Failed(

string.Format(

“Multiple projections emitting to the same stream detected. Stream: ‘{0}’. Last event projection: ‘{1}’. Emitting projection: ‘{2}’”,

_streamId, parsed.Version.ProjectionId, _projectionVersion.ProjectionId));

return;

}

9

No, the reflective creation of the system projections - not sure on the multiple projections.

James

10

@James I’ve fixed that issue with the logger now (see pull request)

Now more errors! (See above)

11

I basically just want to delete all traces of the old system projections and have them recreated from scratch

12

Thats a bit more difficult than it sounds because they write to many streams (possibly millions)

13

OK cheers Greg, I’ll probably just start from a clean database, then I can just ‘manually’ copy across the streams I need from the old instance. It’s not all data I need anyway.

Anyway the reason I was mucking about with the dev branch in the first place was to see if I could figure out that other issue on github, since it’s blocking my current work. We do some secondary indexing that subscribes to a category event stream, and that is not working because of that issue.

14

Ah so I know where the problem is with that one. The code in the atomcontroller and I presume in TCP is not handling properly the case where the ResolvedEvent on ReadEventsBackwards etc is null but it is a link. The fix there isn’t too tough mostly need to write tests around it and implement (prob 1-2 days worth of work). I will likely hit that up next week (in Italy this week and home next)