Projections in Faulted state

EventStoreDB
1

I have a 3 node cluster on windows servers.

The cluster status using /gossip is all ok and I can write events to streams

However all projections (which were working) are now in a Faulted state

{

“coreProcessingTime”: 0,

“version”: 0,

“epoch”: -1,

“effectiveName”: “$users”,

“writesInProgress”: 0,

“readsInProgress”: 0,

“partitionsCached”: 1,

“status”: “Faulted/Running (Enabled)”,

“stateReason”: “Unexpected ‘STARTED’ message in FaultedState”,

“name”: “$users”,

“mode”: “Continuous”,

“position”: "C:33683526/P:33683526; $UserCreated: -1; ",

“progress”: 100.0,

“lastCheckpoint”: "C:33683526/P:33683526; $UserCreated: -1; ",

“eventsProcessedAfterRestart”: 0,

“statusUrl”: “http://192.168.147.52:2113/projection/$users”,

“stateUrl”: “http://192.168.147.52:2113/projection/$users/state”,

“resultUrl”: “http://192.168.147.52:2113/projection/$users/result”,

“queryUrl”: “http://192.168.147.52:2113/projection/$users/query%3Fconfig=yes”,

“enableCommandUrl”: “http://192.168.147.52:2113/projection/$users/command/enable”,

“disableCommandUrl”: “http://192.168.147.52:2113/projection/$users/command/disable”,

“checkpointStatus”: “”,

“bufferedEvents”: 0,

“writePendingEventsBeforeCheckpoint”: 0,

“writePendingEventsAfterCheckpoint”: 0

},

The error log is filling up with messages such as

[PID:05020:016 2015.02.19 08:38:57.207 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Projections.Core.Messages.CoreProjectionManagementMessage+Ge

tState in queued handler ‘Projection Core #2’.

System.Exception: Current state is Initial. Expected states are: Running, Stopping, Stopped, FaultedStopping, Faulted, CompletingPhase, PhaseCompleted

at EventStore.Projections.Core.Services.Processing.CoreProjection.EnsureState(State expectedStates) in c:\EventStore\src\EventStore.Projections.Core\Services\Processi

ng\CoreProjection.cs:line 616

at EventStore.Projections.Core.Services.Processing.CoreProjection.Handle(GetState message) in c:\EventStore\src\EventStore.Projections.Core\Services\Processing\CorePr

ojection.cs:line 263

at EventStore.Projections.Core.Services.Processing.ProjectionCoreService.Handle(GetState message) in c:\EventStore\src\EventStore.Projections.Core\Services\Processing

\ProjectionCoreService.cs:line 270

at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message) in c:\EventStore\src\EventStore.Core\Bus\MessageHandler.cs:line 33

at EventStore.Core.Bus.InMemoryBus.Publish(Message message) in c:\EventStore\src\EventStore.Core\Bus\InMemoryBus.cs:line 324

at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o) in c:\EventStore\src\EventStore.Core\Bus\QueuedHandlerMRES.cs:line 121

Stopping and restarting individual projections doesnt help

If I stop the master node projections just enter a preparing state and never get any further

When I restart the whole cluster all projections are still there and working

Can anyone suggest

a) what may cause the situation to occur in the first place

b) is there any less drastic method to recover the projections if they re-enter this state

Thanks

Ian

I

2

I am using eventstore 3.0.1

3

​This is a known issue with projections at the moment - the easiest solution is to kill the master which results in a takeover.

4

Thanks for letting me know James - I’ll try killing the master when it hapens again.

On the back of that can I ask few further questions?

I appreciate that projections are still in Beta but wondered if you have any idea when the issue is likely to be fixed

Would I be worth using the the 3.1 release (even as a pre-release) - any target dates for the official release?

And finally does the cluster monitoring with the comercial support help in any way with this issue ?

Ian

5

We just had this as well on 3.0.2, and James’ suggestion solved it.

We recently purchased a commercial support contract, but still haven’t upgraded :smiley: I gather that these are the type of hiccups that the commercial tools handle anyway.