EventStore Projection in faulted or other states....what to do.

EventStoreDB
21

Hi John,

When you try to start your projection, do you have one of those errors reported:

  • Cannot stop a projection in the Creating state

  • Projection version and event number mismatch

  • Unsupported error code received

?

22

I meant stop not start.

23

Just going by memory, but I believe the error was cannot stop projection at this time.

24

I am guessing this is the UI error? What do logs say?

25

Last night, the client stopped all services and eventstore, rebooted the server, and restarted everything up again.

Project is now running and events seem to be processing again. No idea why, but for now client is ok. There was some discussion about the server’s cpu being pegged for some time. The only thing I can think of is that the server was in an unstable state and the client tried to stop the projection, and eventstore could not process the request keeping it in a stopping state.

I have the latest logs and I am going to see if I can find why it could not stop the projection.

Stay tuned.

26

Projection not project

27

https://drive.google.com/file/d/14vkXDG6Z5cdVMDSKjftTL0mCNby_ohix/view?usp=sharing

https://drive.google.com/file/d/1Ks4LiwH7e_Nqk8sz3iB7g3mAkDCNZErX/view?usp=sharing

Here are links to the logs. Sorry I don’t see anything in them regarding stopping a projection.

28

As an aside just noticed while reading through … will read through more … You likely want to up your checkpointing on your persistent subscription here it looks like you are checkpointing about once every 1000-1500ms…

[PID:02560:012 2019.11.19 00:42:29.024 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint
[PID:02560:006 2019.11.19 00:42:31.947 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint: 25166447
[PID:02560:012 2019.11.19 00:42:32.040 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint
[PID:02560:014 2019.11.19 00:42:34.947 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint: 25166460
[PID:02560:012 2019.11.19 00:42:34.994 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint
[PID:02560:009 2019.11.19 00:42:37.947 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint: 25166472
[PID:02560:012 2019.11.19 00:42:37.963 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint
[PID:02560:009 2019.11.19 00:42:39.432 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-textExtractionQueue::textExtraction-checkpoint: 206365
[PID:02560:012 2019.11.19 00:42:39.510 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-textExtractionQueue::textExtraction-checkpoint
[PID:02560:014 2019.11.19 00:42:40.950 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint: 25166485
[PID:02560:012 2019.11.19 00:42:41.029 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint
[PID:02560:031 2019.11.19 00:42:43.951 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint: 25166499
[PID:02560:012 2019.11.19 00:42:44.013 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint
[PID:02560:023 2019.11.19 00:42:45.952 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint: 25166509
[PID:02560:012 2019.11.19 00:42:45.999 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint
[PID:02560:006 2019.11.19 00:42:47.952 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint: 25166519
[PID:02560:012 2019.11.19 00:42:48.015 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint
[PID:02560:004 2019.11.19 00:42:50.953 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint: 25166532
[PID:02560:012 2019.11.19 00:42:51.015 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint
[PID:02560:014 2019.11.19 00:42:53.953 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint: 25166546
[PID:02560:012 2019.11.19 00:42:54.015 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint
29

Greg:

Thanks we have a developer looking at the client’s environment now. I will pass it on.

30

Also, another developer here looked at the EventStore and seems to think there may be no explicit code to change from a stopping state to stopped state. So if the server was unstable and failed to stop a projection, leaving it in a stopping state, subsequent attempts would fail to stop the projection.

31

There is a feature in es6 that will allow just the projection subsystem to be restarted. While not ideal it gives a workaround for projections not being in a state where they can be started while we finish root cause analysis of the issue.