Projection stuck in RunningRequested state

EventStoreDB
1

Hello,

We have a single cluster node GES 3.9.3 running in a Window 2012 r2 server.

It has small events, in 5 chunk files with a total size of 1.5 GB.

The problem is a projection that suddenly change to “RunningRequested” state, with 183 events buffered. I’ve tried to stop and reset the projection, but it stays in ‘StoppingRequested’ state.

The console displays the following messages:

I’ve found an error log with several messages like this one:

[PID:08880:040 2017.01.26 17:57:59.532 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Core.Messages.ClientMessage+ReadStreamEventsBackwardCompleted in queued handler ‘Projection Core #0’.

System.Exception: Exception of type ‘System.Exception’ was thrown.

at EventStore.Projections.Core.Services.Processing.TransactionFilePositionTagger.AdjustTag(CheckpointTag tag) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\TransactionFilePositionTagger.cs:line 23

at EventStore.Projections.Core.Services.Processing.CheckpointTagVersion.AdjustBy(PositionTagger tagger, ProjectionVersion version) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\CheckpointTagExtensions.cs:line 24

at EventStore.Projections.Core.Services.Processing.EmittedStream.ReadStreamEventsBackwardCompleted(ReadStreamEventsBackwardCompleted message, CheckpointTag lastCheckpointPosition) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\EmittedStream.cs:line 305

at EventStore.Core.Messaging.RequestResponseDispatcher`2.Handle(TResponse message) in c:\projects\eventstore\src\EventStore.Core\Messaging\RequestResponseDispatcher.cs:line 75

at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message) in c:\projects\eventstore\src\EventStore.Core\Bus\MessageHandler.cs:line 33

at EventStore.Core.Bus.InMemoryBus.Publish(Message message) in c:\projects\eventstore\src\EventStore.Core\Bus\InMemoryBus.cs:line 324

at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o) in c:\projects\eventstore\src\EventStore.Core\Bus\QueuedHandlerMRES.cs:line 122

If I try to restart the ES, it throws the following errors:

In order to start it I used the --skip-db-verify, and then I’m where I started.

It has happened before, last time I reset the database, but I’m concerned because it seems that the root cause still remains.

regards,

Gustavo

2

Hi Gustav,
Would it be possible for you to give me a copy of the db to diagnose the issue, that’s if you are happy to share it.

You can email a link to a downloadable copy to pieter at geteventstore dot com

3

Just a couple of follow on questions…

What is the status of all the projections?

4

Hello Pieter,

Thank you for your response.

I sent you a link with the zipped db at your email.

The other projections are stopped,… but it was working before in that state.

regards,

Gustavo

5

Thank you for the database and the other info, will reply within the next hour.

6

Gustav, I have replied to you directly, but to others that might run into this issue it’s because there was a stream that was written to by the user originally and each event had metadata which incorrectly got parsed as projection metadata. When a projection attempts to emit to a stream it will read the metadata from the events in the stream it attempts to write to and this is where the issue above comes in. The metadata, although parsed (incorrectly), then fails to be adjusted because the projection expects that metadata to represent a checkpoint.

There can be work done around this area to ensure that the user is better informed of such issues, as it’s not obvious what’s going on here.

7

Pieter, yes, you’re right.

I was writing directly to a stream the projection was writing too.

thank you very much!

Gustavo