EventStore Projection in faulted or other states....what to do.

John_Ginzo · November 14, 2019, 1:09pm

Hopefully someone here has an explanation. I cannot find any help on EventStore support or documentation.

We have a client using EventStore. They have a projection created by us. This projection is in a “Stopping/Initial” state. We also see that the system projection $by_event_type is in a “Preparing/Faulted” state. At this point, it does not seem like out user defined projection is in a working state. The apps that are subscribing to this projection are no longer receiving events.

We cannot see any errors in logs.

Does anybody here know what the various projection states mean and how to recover from this?

Any help is appreciated.

John G.

yorick.laupa · November 14, 2019, 3:12pm

Hello John,

It will be challenging to help you out, considering you say it doesn’t work with no information we can exploit.

Which version of eventstore do you use?

What is the eventstore’s server configuration (OS, is it a cluster of eventstore nodes)?

Can you share your user-defined projection’s code?

When you go to the projection tabs and click on your projection, what are the evens processed, position, last checkpoint values?

What is the configuration of your user-defined projection?

Considering how ‘visible’ your problem is, I’m pretty sure having access to the log will narrow down significantly the causes.

Yorick

John_Ginzo · November 14, 2019, 4:13pm

Version is 4.1.4
single cluster, Windows Server 2012

Here is the latest error log info. We do have some errors.

I guess the problem is we don’t know what those states actually mean. what does Stopping/Initial or Preparing/Faulted mean? If we knew what they meant or even how to find an error related to it in the logs we may be able to figure it out.

[PID:02560:024 2019.11.04 02:18:52.790 DEBUG TcpConnectionSsl ] Exception during EndRead.

System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)

at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)

— End of inner exception stack trace —

at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)

at EventStore.Transport.Tcp.TcpConnectionSsl.OnEndRead(IAsyncResult ar)

System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host

at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)

at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)

[PID:02560:036 2019.11.04 15:52:03.427 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Projections.Core.Messages.ProjectionManagementMessage+Command+Disable in queued handler ‘Projections Master’.

System.NotSupportedException: Specified method is not supported.

at EventStore.Projections.Core.Services.Management.ManagedProjection.StopUnlessPreparedOrLoaded()

at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message)

at EventStore.Core.Bus.InMemoryBus.Publish(Message message)

at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o)

[PID:02560:036 2019.11.04 15:52:07.182 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Projections.Core.Messages.ProjectionManagementMessage+Command+Disable in queued handler ‘Projections Master’.

System.NotSupportedException: Specified method is not supported.

at EventStore.Projections.Core.Services.Management.ManagedProjection.StopUnlessPreparedOrLoaded()

at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message)

at EventStore.Core.Bus.InMemoryBus.Publish(Message message)

at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o)

[PID:02560:037 2019.11.04 16:02:38.171 DEBUG TcpConnectionSsl ] Exception during EndRead.

System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)

at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)

— End of inner exception stack trace —

at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)

at EventStore.Transport.Tcp.TcpConnectionSsl.OnEndRead(IAsyncResult ar)

System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host

at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)

at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)

[PID:02560:028 2019.11.04 16:27:02.507 DEBUG TcpConnectionSsl ] Exception during EndRead.

System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)

at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)

— End of inner exception stack trace —

at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)

at EventStore.Transport.Tcp.TcpConnectionSsl.OnEndRead(IAsyncResult ar)

System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host

at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)

at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)

[PID:02560:036 2019.11.04 16:30:11.179 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Projections.Core.Messages.ProjectionManagementMessage+Command+Disable in queued handler ‘Projections Master’.

System.NotSupportedException: Specified method is not supported.

at EventStore.Projections.Core.Services.Management.ManagedProjection.StopUnlessPreparedOrLoaded()

at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message)

at EventStore.Core.Bus.InMemoryBus.Publish(Message message)

at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o)

yorick.laupa · November 14, 2019, 4:22pm

Do you know what SSL method is used because it isn’t supported by your server (TLS, SSLv23, …etc) ?

John_Ginzo · November 14, 2019, 4:27pm

We are using TLS 1.2 with a certificate using port 1115. We don’t see any issues with services connecting to Event store. But we do see these projections in the state we mentioned.

John_Ginzo · November 14, 2019, 4:31pm

We are also seeing the following. See the highlighted section:

[PID:02560:012 2019.11.04 15:51:42.522 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint

[PID:02560:019 2019.11.04 15:51:53.586 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint: 20105606

[PID:02560:012 2019.11.04 15:51:53.617 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint

[PID:02560:015 2019.11.04 15:51:59.634 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint: 20105620

[PID:02560:012 2019.11.04 15:51:59.697 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint

[PID:02560:025 2019.11.04 15:52:01.656 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint: 20105630

[PID:02560:012 2019.11.04 15:52:01.687 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint

[PID:02560:026 2019.11.04 15:52:02.949 TRACE CoreProjectionCheckp] Writing checkpoint for $streams at C:341471938761/P:341471938761 with expected version number 44940

[PID:02560:033 2019.11.04 15:52:02.949 TRACE CoreProjectionCheckp] Writing checkpoint for $stream_by_category at C:341471938761/P:341471938761 with expected version number 44940

[PID:02560:032 2019.11.04 15:52:02.996 TRACE CoreProjectionCheckp] Writing checkpoint for $by_category at C:341471938761/P:341471938761 with expected version number 44940

[PID:02560:026 2019.11.04 15:52:03.043 TRACE CoreProjectionCheckp] Checkpoint has been written for projection $streams at sequence number 44941 (current)

[PID:02560:033 2019.11.04 15:52:03.074 TRACE CoreProjectionCheckp] Checkpoint has been written for projection $stream_by_category at sequence number 44941 (current)

[PID:02560:032 2019.11.04 15:52:03.090 TRACE CoreProjectionCheckp] Checkpoint has been written for projection $by_category at sequence number 44941 (current)

[PID:02560:036 2019.11.04 15:52:03.427 INFO ProjectionManager ] Disabling ‘$by_event_type’ projection

[PID:02560:036 2019.11.04 15:52:03.427 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Projections.Core.Messages.ProjectionManagementMessage+Command+Disable in queued handler ‘Projections Master’.

System.NotSupportedException: Specified method is not supported.

at EventStore.Projections.Core.Services.Management.ManagedProjection.StopUnlessPreparedOrLoaded()

at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message)

at EventStore.Core.Bus.InMemoryBus.Publish(Message message)

at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o)

[PID:02560:009 2019.11.04 15:52:04.681 DEBUG PersistentSubscripti] Publishing checkpoint for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint: 20105645

[PID:02560:012 2019.11.04 15:52:04.744 DEBUG PersistentSubscripti] Checkpoint write successful for $persistentsubscription-PageIndexingQueue::PageIndexing-checkpoint

[PID:02560:036 2019.11.04 15:52:07.182 INFO ProjectionManager ] Disabling ‘$by_event_type’ projection

yorick.laupa · November 14, 2019, 4:37pm

It seems a user deliberately disabled $by_event_type stream. I’m not working for EventStore LTD, but when looking at the code for version 4.1.4, I don’t see the Disable command being sent by the database itself. Is it possible that someone disabled some of your projections by mistake?

John_Ginzo · November 14, 2019, 7:41pm

How would someone “disable” a projection?

I can see a user being able to Stop a projection but that sets the state to “stopped” not the states I am seeing. You can use the “Disable All” item but again that sets them to stopped. We are seeing the state I mentioned above.

See Image:

John_Ginzo · November 14, 2019, 7:43pm

The image I sent btw is what happens when you disable all or stop a projection. This is not what we see with the client

yorick.laupa · November 14, 2019, 8:03pm

It’s hard to exactly know what happened considering we can’t tell if the log you gave correspond to when the issue occurred. You only gave bits of the log. I suggested you to gave more information about your projection in my first post, to know if your projection got the chance to start or at least how far it got before failing. Having also the code of your projection could help, especially if you use a system projection that got disabled in the meantime. The more valuable information you give, the best we can help.

John_Ginzo · November 15, 2019, 12:43pm

I appreciate your attempts to help. Trying to get the rest of the logs from the client. Waiting on that.

Below is the projection code. Sorry it may be hard to read:

John_Ginzo · November 15, 2019, 3:19pm

Here are the log files we have for the client. I have not seen anything there pointing to this issue.

https://drive.google.com/open?id=1Q8sIHoxw9dwRpf3I1_1vgbXvNSOge9DT

https://drive.google.com/open?id=15uUecwc6o8LY8vwHb9r-iVtgpXHSSxyf

John_Ginzo · November 15, 2019, 3:44pm

Here is what we see in the UI

John_Ginzo · November 15, 2019, 3:45pm

See attached

yorick.laupa · November 17, 2019, 1:38pm

Hi John,

I looked into the log you provided. There are two occurrences, it starts on 4th, Nov 15:47:55 and 11th, Nov 22:19:53. When happening, those incidents are not spread across the day.

2019-11-04/2019-11-04/10.70.1.112-2113-cluster-node.log:38774:[PID:02560:036 2019.11.04 15:47:55.784 INFO ProjectionManager ] Disabling ‘$by_event_type’ projection

2019-11-04/2019-11-04/10.70.1.112-2113-cluster-node.log:38814:[PID:02560:036 2019.11.04 15:48:40.775 INFO ProjectionManager ] Disabling ‘backExtractionProjection’ projection

2019-11-04/2019-11-04/10.70.1.112-2113-cluster-node.log:38823:[PID:02560:036 2019.11.04 15:48:47.937 INFO ProjectionManager ] Disabling ‘backExtractionProjection’ projection

2019-11-04/2019-11-04/10.70.1.112-2113-cluster-node.log:38832:[PID:02560:036 2019.11.04 15:48:57.943 INFO ProjectionManager ] Disabling ‘backExtractionProjection’ projection

2019-11-04/2019-11-04/10.70.1.112-2113-cluster-node.log:38975:[PID:02560:036 2019.11.04 15:52:03.427 INFO ProjectionManager ] Disabling ‘$by_event_type’ projection

2019-11-04/2019-11-04/10.70.1.112-2113-cluster-node.log:38984:[PID:02560:036 2019.11.04 15:52:07.182 INFO ProjectionManager ] Disabling ‘$by_event_type’ projection

2019-11-04/2019-11-04/10.70.1.112-2113-cluster-node.log:39001:[PID:02560:036 2019.11.04 15:52:17.465 INFO ProjectionManager ] Disabling ‘$by_event_type’ projection

2019-11-04/2019-11-04/10.70.1.112-2113-cluster-node.log:40601:[PID:02560:036 2019.11.04 16:30:11.179 INFO ProjectionManager ] Disabling ‘$by_event_type’ projection

2019-11-04/2019-11-04/10.70.1.112-2113-cluster-node.log:40616:[PID:02560:036 2019.11.04 16:30:21.860 INFO ProjectionManager ] Disabling ‘$by_event_type’ projection

``

In this case, almost every 10secs ‘disabling’ happens.

2019-11-10/2019-11-11/10.70.1.112-2113-cluster-node.log:61315:[PID:02560:036 2019.11.11 22:19:53.104 INFO ProjectionManager ] Disabling ‘backExtractionProjection’ projection

2019-11-10/2019-11-11/10.70.1.112-2113-cluster-node.log:61320:[PID:02560:036 2019.11.11 22:19:58.388 INFO ProjectionManager ] Disabling ‘backExtractionProjection’ projection

2019-11-10/2019-11-11/10.70.1.112-2113-cluster-node.log:61327:[PID:02560:036 2019.11.11 22:20:08.504 INFO ProjectionManager ] Disabling ‘backExtractionProjection’ projection

2019-11-10/2019-11-11/10.70.1.112-2113-cluster-node.log:61334:[PID:02560:036 2019.11.11 22:20:15.558 INFO ProjectionManager ] Disabling ‘$by_event_type’ projection

2019-11-10/2019-11-11/10.70.1.112-2113-cluster-node.log:61353:[PID:02560:036 2019.11.11 22:20:25.827 INFO ProjectionManager ] Disabling ‘$by_event_type’ projection

2019-11-10/2019-11-11/10.70.1.112-2113-cluster-node.log:61529:[PID:02560:036 2019.11.11 22:22:47.043 INFO ProjectionManager ] Disabling ‘$by_event_type’ projection

2019-11-10/2019-11-11/10.70.1.112-2113-cluster-node.log:61548:[PID:02560:036 2019.11.11 22:22:57.160 INFO ProjectionManager ] Disabling ‘$by_event_type’ projection

2019-11-10/2019-11-11/10.70.1.112-2113-cluster-node.log:61585:[PID:02560:036 2019.11.11 22:23:28.257 INFO ProjectionManager ] Disabling ‘$by_event_type’ projection

2019-11-10/2019-11-11/10.70.1.112-2113-cluster-node.log:61606:[PID:02560:036 2019.11.11 22:23:38.303 INFO ProjectionManager ] Disabling ‘$by_event_type’ projection

2019-11-10/2019-11-11/10.70.1.112-2113-cluster-node.log:61627:[PID:02560:036 2019.11.11 22:23:48.307 INFO ProjectionManager ] Disabling ‘$by_event_type’ projection

2019-11-10/2019-11-11/10.70.1.112-2113-cluster-node.log:61648:[PID:02560:036 2019.11.11 22:23:58.384 INFO ProjectionManager ] Disabling ‘$by_event_type’ projection

2019-11-10/2019-11-11/10.70.1.112-2113-cluster-node.log:61673:[PID:02560:036 2019.11.11 22:24:08.449 INFO ProjectionManager ] Disabling ‘$by_event_type’ projection

2019-11-10/2019-11-11/10.70.1.112-2113-cluster-node.log:61696:[PID:02560:036 2019.11.11 22:24:18.518 INFO ProjectionManager ] Disabling ‘$by_event_type’ projection

2019-11-10/2019-11-11/10.70.1.112-2113-cluster-node.log:61717:[PID:02560:036 2019.11.11 22:24:28.579 INFO ProjectionManager ] Disabling ‘$by_event_type’ projection

``

I looked (again) at the 4.1.4 server code, and I saw nowhere the server responsible for disabling a projection by itself (I’m not working for EventStore LTD, so it’s possible I missed something). I bet a full round of beers, those projections got disabled by a human or some code written by a human that uses the client interface. All those exceptions only happen when projections start to get disabled.

[PID:02560:036 2019.11.04 15:52:03.427 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Projections.Core.Messages.ProjectionManagementMessage+Command+Disable in queued handler ‘Projections Master’.

System.NotSupportedException: Specified method is not supported.

at EventStore.Projections.Core.Services.Management.ManagedProjection.StopUnlessPreparedOrLoaded()

at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message)

at EventStore.Core.Bus.InMemoryBus.Publish(Message message)

at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o)

[PID:02560:036 2019.11.04 15:52:07.182 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Projections.Core.Messages.ProjectionManagementMessage+Command+Disable in queued handler ‘Projections Master’.

System.NotSupportedException: Specified method is not supported.

at EventStore.Projections.Core.Services.Management.ManagedProjection.StopUnlessPreparedOrLoaded()

at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message)

at EventStore.Core.Bus.InMemoryBus.Publish(Message message)

at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o)

``

[PID:02560:036 2019.11.11 22:20:15.558 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Projections.Core.Messages.ProjectionManagementMessage+Command+Disable in queued handler ‘Projections Master’.

System.NotSupportedException: Specified method is not supported.

at EventStore.Projections.Core.Services.Management.ManagedProjection.StopUnlessPreparedOrLoaded()

at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message)

at EventStore.Core.Bus.InMemoryBus.Publish(Message message)

at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o)

[PID:02560:036 2019.11.11 22:22:47.043 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Projections.Core.Messages.ProjectionManagementMessage+Command+Disable in queued handler ‘Projections Master’.

System.NotSupportedException: Specified method is not supported.

at EventStore.Projections.Core.Services.Management.ManagedProjection.StopUnlessPreparedOrLoaded()

at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message)

at EventStore.Core.Bus.InMemoryBus.Publish(Message message)

at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o)

[PID:02560:036 2019.11.11 22:23:28.257 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Projections.Core.Messages.ProjectionManagementMessage+Command+Disable in queued handler ‘Projections Master’.

System.NotSupportedException: Specified method is not supported.

at EventStore.Projections.Core.Services.Management.ManagedProjection.StopUnlessPreparedOrLoaded()

at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message)

at EventStore.Core.Bus.InMemoryBus.Publish(Message message)

at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o)

[PID:02560:036 2019.11.11 22:23:38.303 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Projections.Core.Messages.ProjectionManagementMessage+Command+Disable in queued handler ‘Projections Master’.

System.NotSupportedException: Specified method is not supported.

at EventStore.Projections.Core.Services.Management.ManagedProjection.StopUnlessPreparedOrLoaded()

at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message)

at EventStore.Core.Bus.InMemoryBus.Publish(Message message)

at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o)

[PID:02560:036 2019.11.11 22:23:48.307 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Projections.Core.Messages.ProjectionManagementMessage+Command+Disable in queued handler ‘Projections Master’.

System.NotSupportedException: Specified method is not supported.

at EventStore.Projections.Core.Services.Management.ManagedProjection.StopUnlessPreparedOrLoaded()

at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message)

at EventStore.Core.Bus.InMemoryBus.Publish(Message message)

at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o)

[PID:02560:036 2019.11.11 22:23:58.384 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Projections.Core.Messages.ProjectionManagementMessage+Command+Disable in queued handler ‘Projections Master’.

System.NotSupportedException: Specified method is not supported.

at EventStore.Projections.Core.Services.Management.ManagedProjection.StopUnlessPreparedOrLoaded()

at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message)

at EventStore.Core.Bus.InMemoryBus.Publish(Message message)

at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o)

[PID:02560:036 2019.11.11 22:24:08.449 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Projections.Core.Messages.ProjectionManagementMessage+Command+Disable in queued handler ‘Projections Master’.

System.NotSupportedException: Specified method is not supported.

at EventStore.Projections.Core.Services.Management.ManagedProjection.StopUnlessPreparedOrLoaded()

at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message)

at EventStore.Core.Bus.InMemoryBus.Publish(Message message)

at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o)

[PID:02560:036 2019.11.11 22:24:18.518 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Projections.Core.Messages.ProjectionManagementMessage+Command+Disable in queued handler ‘Projections Master’.

System.NotSupportedException: Specified method is not supported.

at EventStore.Projections.Core.Services.Management.ManagedProjection.StopUnlessPreparedOrLoaded()

at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message)

at EventStore.Core.Bus.InMemoryBus.Publish(Message message)

at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o)

[PID:02560:036 2019.11.11 22:24:28.579 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Projections.Core.Messages.ProjectionManagementMessage+Command+Disable in queued handler ‘Projections Master’.

System.NotSupportedException: Specified method is not supported.

at EventStore.Projections.Core.Services.Management.ManagedProjection.StopUnlessPreparedOrLoaded()

at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message)

at EventStore.Core.Bus.InMemoryBus.Publish(Message message)

at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o)

``

It worth asking again, are you sure there is no other party able to disable those projections?

John_Ginzo · November 17, 2019, 3:31pm

Thanks for this information. The clients usually do not deal with EventStore directly. But I guess anything is possible. The issue I see is that even if some code or a human disabled them, they are stuck in a faulted or stopping state and we are unable to recover from this state.

yorick.laupa · November 17, 2019, 4:08pm

Can’t you just stop and start those projections again (without resetting of course)?

John_Ginzo · November 18, 2019, 4:12pm

We have tried that…but has not resolved the issue.

John_Ginzo · November 18, 2019, 6:36pm

Just to be clear. the stop button is disabled for these projections. It is not possible to stop and start.

Out support is working with the client to stop event store and all services connected to it and restart everything. Waiting to hear back.

It would be great if someone from EventStore staff could comment on this issue. It would be also great if there was some explanation from EventStore as to what these states mean and how to recover from this.

John_Ginzo · November 19, 2019, 6:45pm

After speaking with the client, the latest information is as follows:

We tried to start and stop the projection. This produced an error in the UI that the projection cannot be stopped. We don’t know yet what caused the projection to be in this state, nor how to recover. The client has already tried stopping all services and Eventstore and restarting, which does not fix the issue.

I also spoke to Daniel Warren from EventStore. He admits that EventStore is lacking documentation on the projection states and how to manage them, and a ticket was created to add this. However, this does not really help us since we cannot recover from this state. At this point, it seems like there is some type of defect in EventStore. We should be able to recover from a particular state. Daniel and EventStore will only assist if we have a support contract with them or pay per incident.

However, this does seem to be a defect that EventStore should look into. In the meantime, we will continue to research this on our side in the hopes we find a solution.

Any other help is much appreciated.