All projections hanging is status Preparing

Denis_Mikhaylov · February 9, 2016, 9:05am

$by_category
Preparing
Continuous
0.0%
0 / 0
0 / 0
0
0.0
0
0
$by_event_type
Preparing
Continuous
0.0%
0 / 0
0 / 0
0
0.0
0
0
$stream_by_category
Preparing
Continuous
0.0%
0 / 0
0 / 0
0
0.0
0
0
$streams
Preparing
Continuous
0.0%
0 / 0
0 / 0
0
0.0
0
0
All my projections are in preparing status.

I can neither stop nor delete them.

What could be the problem?

pieter.germishuys · February 9, 2016, 9:08am

Are you running a cluster of nodes?

Denis_Mikhaylov · February 9, 2016, 9:11am

Yes. 5 nodes

pieter.germishuys · February 9, 2016, 9:22am

Do you have logs for the nodes that you are able to share?
Interim fix would be to restart the master node. This is by no means a real solution, however as we move projections from beta into production, which is the current focus, this issue will be addressed.

Denis_Mikhaylov · February 9, 2016, 9:30am

Regarding logs, It’s gonna take a while. Like one hour.

What are the conditions when this may happen?
We are in the process of moving to production, and I need to know how to react, and what not to do.
Projections aren’t going to change often. So I need them up and running.

pieter.germishuys · February 9, 2016, 9:44am

Projections are currently in beta and not supported in production.
Unfortunately there are a couple of scenarios were projections can end up being stuck in preparing like this.

For now, until we release a fix for this specific issue you can write a service that just checks the status of projections and curl -i http://localhost:2113/projections/any and if they are stuck in preparing, you can kill the master node, which will resolve the issue.

Unfortunately as of now I cannot give you a better answer.

Have you considered getting a support contract?

Denis_Mikhaylov · February 9, 2016, 1:00pm

Here are the logs from master node (rebooting actually helped)
[PID:00850:034 2016.02.08 15:37:51.158 ERROR QueuedHandlerAutoRes] Error while processing message EventStore.Projections.Core.Messages.ReaderSubscriptionManagement+Unsubscribe in queued handler ‘Projection Core #2’.

System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.

at System.Collections.Generic.Dictionary`2[System.Guid,System.Guid].get_Item (Guid key) [0x00000] in :0

at EventStore.Projections.Core.Services.Processing.EventReaderCoreService.Handle (EventStore.Projections.Core.Messages.Unsubscribe message) [0x00000] in :0

at EventStore.Core.Bus.MessageHandler`1[EventStore.Projections.Core.Messages.ReaderSubscriptionManagement+Unsubscribe].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0

at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0

at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0

at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0

[PID:00850:032 2016.02.08 15:37:51.158 ERROR QueuedHandlerAutoRes] Error while processing message EventStore.Projections.Core.Messages.ReaderSubscriptionManagement+Unsubscribe in queued handler ‘Projection Core #0’.

System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.

at System.Collections.Generic.Dictionary`2[System.Guid,System.Guid].get_Item (Guid key) [0x00000] in :0

at EventStore.Projections.Core.Services.Processing.EventReaderCoreService.Handle (EventStore.Projections.Core.Messages.Unsubscribe message) [0x00000] in :0

at EventStore.Core.Bus.MessageHandler`1[EventStore.Projections.Core.Messages.ReaderSubscriptionManagement+Unsubscribe].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0

at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0

at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0

at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0

[PID:00850:034 2016.02.08 15:37:51.161 ERROR QueuedHandlerAutoRes] Error while processing message EventStore.Projections.Core.Messages.ReaderSubscriptionManagement+Unsubscribe in queued handler ‘Projection Core #2’.

System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.

at System.Collections.Generic.Dictionary`2[System.Guid,System.Guid].get_Item (Guid key) [0x00000] in :0

at EventStore.Projections.Core.Services.Processing.EventReaderCoreService.Handle (EventStore.Projections.Core.Messages.Unsubscribe message) [0x00000] in :0

at EventStore.Core.Bus.MessageHandler`1[EventStore.Projections.Core.Messages.ReaderSubscriptionManagement+Unsubscribe].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0

at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0

at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0

at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0

[PID:00850:032 2016.02.08 15:37:51.161 ERROR QueuedHandlerAutoRes] Error while processing message EventStore.Projections.Core.Messages.ReaderSubscriptionManagement+Unsubscribe in queued handler ‘Projection Core #0’.