Corrupt eventstore, anything that can be done?

EventStoreDB
1

Gettings this a startup:

[PID:18900:014 2015.05.28 14:49:11.928 FATAL StorageWriterService] Unexpected error in StorageWriterService. Terminating the process…

System.ArgumentException: Log record at actual pos 210293440 has too large length: 18741015 bytes, while limit is 16777216 bytes. Something is seriously wrong in chunk 19-19 (C:\Temp\ConfidenceTicketEventStore\chunk-000019.000000).

at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.TFChunkReadSide.TryReadForwardInternal(ReaderWorkItem workItem, Int64 actualPosition, Int32& length, LogRecord& record) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Core\TransactionLog\Chunks\TFChunk\TFChunkReadSide.cs:line 484

at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.TFChunkReadSideUnscavenged.TryReadClosestForward(Int64 logicalPosition) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Core\TransactionLog\Chunks\TFChunk\TFChunkReadSide.cs:line 103

at EventStore.Core.TransactionLog.Chunks.TFChunkReader.TryReadNextInternal(Int32 retries) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Core\TransactionLog\Chunks\TFChunkReader.cs:line 84

at EventStore.Core.Services.Storage.ReaderIndex.ReadIndex.Init(Int64 writerCheckpoint, Int64 buildToPosition) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Core\Services\Storage\ReaderIndex\ReadIndex.cs:line 141

at EventStore.Core.Services.Storage.StorageWriterService.EventStore.Core.Bus.IHandle<EventStore.Core.Messages.SystemMessage.SystemInit>.Handle(SystemInit message) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Core\Services\Storage\StorageWriterService.cs:line 197

at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Core\Bus\MessageHandler.cs:line 60

at EventStore.Core.Bus.InMemoryBus.Publish(Message message) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Core\Bus\InMemoryBus.cs:line 362

at EventStore.Core.Services.Storage.StorageWriterService.CommonHandle(Message message) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Core\Services\Storage\StorageWriterService.cs:line 178

Background:

Faulty inputvalidation caused a huge commit (100000+ events). After that es will not start.

This is from version 2, also tried loading it in version 3.03, same error.

/Peter

2

Hi.

This might not be appropriate for the mailing list, but I’d be interested in in knowing how others run and operate eventstore.

We would really like commercial support, but the current options doesn’t work for us. We have many small customers running their own servers. The per-cluster cost is more than our yearly support cost in most instances.

Is everyone else in the cloud doing multitennant saas nowadays? We’re not really there yet...

/Peter

3

Just been told this is on a production server without backup… Suffice to say, I’m not happy.

What are my options?

Thinking about trying running from source, disabling stuff that throws and just export as much as I can…

4

We have many small customers running their own servers. The per-cluster cost is more than our yearly support cost in most instances.

There is a specific OEM option for this case - get in touch with [email protected] and we’ll see what we can figure out. It mostly sounds like a margins problem rather than anything else though…

5

This was a known bug in version 2 fixed in version 3.

However, now the data is written it will need the chunk file rewriting to fix the issue, all versions will throw at the moment. Is this the ongoing chunk or a previously completed one?

6

It’s the ongoing. The highest numbered one at least, so I’m assuming.

/Peter

7

Email me off list, james at geteventstore.com and we’ll see what we can figure out.