How to recover from out of disk space error?

EventStoreDB
1

Hi

Ran out of disk space while running eventstore, and now seem to have some corruption.

Starting up, i get tons of:

[PID:03964:007 2014.12.04 10:39:21.232 ERROR StorageReaderWorker ] Error during processing ReadStreamEventsBackward request.

System.NullReferenceException: Object reference not set to an instance of an object.

at EventStore.Core.Index.TableIndex.TryGetLatestEntryInternal(UInt32 stream, IndexEntry& entry) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Core\Index\TableIndex.cs:line 459

at EventStore.Core.Index.TableIndex.TryGetLatestEntry(UInt32 stream, IndexEntry& entry) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Core\Index\TableIndex.cs:line 440

at EventStore.Core.Services.Storage.ReaderIndex.ReadIndex.GetLastStreamEventNumberUncached(ITransactionFileReader reader, String streamId) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Core\Services\Storage\ReaderIndex\ReadIndex.cs:line 705

at EventStore.Core.Services.Storage.ReaderIndex.ReadIndex.GetLastStreamEventNumberCached(ITransactionFileReader reader, String streamId) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Core\Services\Storage\ReaderIndex\ReadIndex.cs:line 686

at EventStore.Core.Services.Storage.ReaderIndex.ReadIndex.GetLastStreamEventNumberCached(ITransactionFileReader reader, String streamId) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Core\Services\Storage\ReaderIndex\ReadIndex.cs:line 675

at EventStore.Core.Services.Storage.ReaderIndex.ReadIndex.GetStreamMetadataUncached(ITransactionFileReader reader, String streamId) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Core\Services\Storage\ReaderIndex\ReadIndex.cs:line 1110

at EventStore.Core.Services.Storage.ReaderIndex.ReadIndex.GetStreamMetadataCached(ITransactionFileReader reader, String streamId) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Core\Services\Storage\ReaderIndex\ReadIndex.cs:line 1097

at EventStore.Core.Services.Storage.ReaderIndex.ReadIndex.CheckStreamAccessInternal(ITransactionFileReader reader, String streamId, StreamAccessType streamAccessType, IPrincipal user) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Core\Services\Storage\ReaderIndex\ReadIndex.cs:line 631

at EventStore.Core.Services.Storage.ReaderIndex.ReadIndex.EventStore.Core.Services.Storage.ReaderIndex.IReadIndex.CheckStreamAccess(String streamId, StreamAccessType streamAccessType, IPrincipal user) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Core\Services\Storage\ReaderIndex\ReadIndex.cs:line 598

at EventStore.Core.Services.Storage.StorageReaderWorker.ReadStreamEventsBackward(ReadStreamEventsBackward msg) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Core\Services\Storage\StorageReaderWorker.cs:line 154

I shutdown the server, and restarted it - I get the same errors.

Now I’ve got 3 projections that are in a “starting” state that I cant get out of. I assume that its due to errors like this:

[PID:05012:013 2014.12.04 12:48:30.889 ERROR ProjectionManager ] The ‘SuccessRate’ projection faulted due to 'Cannot create a projection state handler.

Handler type: JS

Query: < snip />

Message:

Failed to compile script. Terminated?’

EventStore.Projections.Core.v8.Js1Exception: Failed to compile script. Terminated?

at EventStore.Projections.Core.v8.CompiledScript.CheckResult(IntPtr scriptHandle, Boolean terminated, Boolean disposeScriptOnException) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Projections.Core\v8\CompiledScript.cs:line 78

at EventStore.Projections.Core.v8.PreludeScript…ctor(String script, String fileName, Func2 getModuleSourceAndFileName, Action2 cancelCallbackFactory, Action`1 logger) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Projections.Core\v8\PreludeScript.cs:line 69

at EventStore.Projections.Core.Services.v8.V8ProjectionStateHandler…ctor(String preludeName, String querySource, Func2 getModuleSource, Action1 logger, Action`2 cancelCallbackFactory) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Projections.Core\Services\v8\V8ProjectionStateHandler.cs:line 54

at EventStore.Projections.Core.Services.Management.ProjectionStateHandlerFactory.Create(String factoryType, String source, Action2 cancelCallbackFactory, Action1 logger) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Projections.Core\Services\Management\ProjectionStateHandlerFactory.cs:line 75

at EventStore.Projections.Core.Services.Management.ManagedProjection.<>c__DisplayClass42.b__41() in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Projections.Core\Services\Management\ManagedProjection.cs:line 741

So, the questions are:

  • how do I recover from the corrupt index

  • how do I get my projections to run?

This is using 2.0.1.

thanks,

Matt

2

Corrupt index is simple. Delete it, it will rebuild. Will look at
projections in a bit.

Greg

3

Thanks - Just ran into James at NDC who gave me the same advice.
Rebuilding now.

Cheers,
Matt

4

Rebuild complete - errors gone and projections running okay now.

Thanks,
Matt

5

great :slight_smile: