Event Store stopped running with this Exception

steven.blair · July 14, 2016, 3:16pm

Just upgraded to 3.8 and it has been running for the last hour or so, then all of sudden I get an error window and then the Event Store process is terminated.
In the log file, I have the following:

[PID:07468:010 2016.07.14 13:33:14.381 ERROR StorageScavenger ] Failed to write the $scavengeIndexInitialized event to the $scavenges stream. Reason: WrongExpectedVersion

[PID:07468:019 2016.07.14 13:35:11.299 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Projections.Core.Messages.ReaderSubscriptionMessage+CommittedEventDistributed in queued handler ‘Projection Core #0’.

System.ArgumentException: complete TF position required

Parameter name: committedEvent

at EventStore.Projections.Core.Services.Processing.EventByTypeIndexPositionTagger.IsMessageAfterCheckpointTag(CheckpointTag previous, CommittedEventDistributed committedEvent) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\EventByTypeIndexPositionTagger.cs:line 36

at EventStore.Projections.Core.Services.Processing.ReaderSubscriptionBase.ProcessOne(CommittedEventDistributed message) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\ReaderSubscriptionBase.cs:line 95

at EventStore.Projections.Core.Services.Processing.HeadingEventReader.DistributeMessage(CommittedEventDistributed message) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\HeadingEventReader.cs:line 204

at EventStore.Projections.Core.Services.Processing.HeadingEventReader.Handle(CommittedEventDistributed message) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\HeadingEventReader.cs:line 97

at EventStore.Projections.Core.Services.Processing.EventReaderCoreService.Handle(CommittedEventDistributed message) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\EventReaderCoreService.cs:line 157

at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message) in c:\projects\eventstore\src\EventStore.Core\Bus\MessageHandler.cs:line 33

at EventStore.Core.Bus.InMemoryBus.Publish(Message message) in c:\projects\eventstore\src\EventStore.Core\Bus\InMemoryBus.cs:line 324

at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o) in c:\projects\eventstore\src\EventStore.Core\Bus\QueuedHandlerMRES.cs:line 122

Any idea what could have caused this?

pieter.germishuys · July 14, 2016, 3:23pm

That should not be causing the Event Store process to terminate.
Is this a fairly large database?

Does it happen consistently on your database?

What other projections other than the system ones are you running?

If you have sensitive information that you cannot share, you are more than welcome to contact me on pieter at geteventstore dot com

steven.blair · July 14, 2016, 3:33pm

Peter,

Thanks for the quick reply

I am trying to restart the Event Store just now to get that information for you, but it’s taking a long time.

messages similar to this:

[02468,04,15:32:55.863] Verifying hash for TFChunk ‘C:\Users\Administrator\Desktop\Event Store\db\chunk-000000.000002’…

Anyway, I would say we have somewhere around 500,000 - 1,000,000 events (a bit vague I know)

This is the first time we have seen this issue.

We run one other projection, which checks specific category and writes them to a new stream for today’s date.

We have the by category system projection on as well.

Greg_Young1 · July 14, 2016, 3:36pm

That is verifying chunks. You can stop that with --skip-db-verify but
with only 1m events it should be quick.

pieter.germishuys · July 14, 2016, 3:36pm

If you just want to gather data you can skip verification which would decrease the startup time. Run Event Store with --skip-db-verify

steven.blair · July 14, 2016, 3:46pm

I will try the skip, but it’s still doing these checks.
That’s 30 minutes now.

Greg_Young1 · July 14, 2016, 3:47pm

for a few million events what is this running on?

steven.blair · July 14, 2016, 3:57pm

A very low end server in the cloud:

RAM: 1.7GB

CPU: E5-2650 @ 2.00GHz

Greg_Young1 · July 14, 2016, 3:57pm

Ah yes you definitely want --skip-db-verify then

steven.blair · July 14, 2016, 4:01pm

It’s on now, and starts within a few seconds!

Greg_Young1 · July 14, 2016, 4:01pm

When you have db verify it will hash all your data that you have. The
slow part is likely more so your storage on a-cheap-cloud-server.

steven.blair · July 15, 2016, 2:48pm

Hi,

Unfortunately this has happened another two times today:

The last error:

[PID:08348:022 2016.07.15 12:48:23.963 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Projections.Core.Messages.ReaderSubscriptionMessage+CommittedEventDistributed in queued handler ‘Projection Core #0’.