This looks like out of file handles.
I can’t imagine why that would be though, it wasn’t doing anything
It was at least collecting system stats and pushing them to $stats stream, when the total number of events summed up to whole million, MemTable was dumped into PTable on disk. That’s when file handles finished,
There is almost never such a thing as “doing nothing” 
I would still appreciate logs (at least today ones) to see what caused the issue of chaser not catching up.
It was merging indexes. How long prior had your Projection finished?
Andrii - ping me on skype rob_ashton and I’ll give you access to the box
Probably finished for an hour or two at the rate it was crawling through
I’m getting a similar error everytime my store gets to 50GB. I’ve upped the ulimit of open files to 10000, but still get the error (see below).
I’m running on a build from the dev branch (9740c6) on CentOS 6.4 with mono 3.0.5.
Any ideas?
[PID:29449:007 2013.05.23 15:08:05.572 ERROR StorageWriterService] Exception in writer.
System.IO.IOException: Too many open files
at System.IO.FileStream…ctor (System.String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean anonymous, FileOptions options) [0x00000] in :0
at System.IO.FileStream…ctor (System.String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options) [0x00000] in :0
at (wrapper remoting-invoke-with-check) System.IO.FileStream:.ctor (string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare,int,System.IO.FileOptions)
at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.CreateInternalReaderWorkItem () [0x00000] in :0
at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.CreateReaderStreams () [0x00000] in :0
at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.InitNew (EventStore.Core.TransactionLog.Chunks.ChunkHeader chunkHeader, Int32 fileSize) [0x00000] in :0
at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.CreateWithHeader (System.String filename, EventStore.Core.TransactionLog.Chunks.ChunkHeader header, Int32 fileSize) [0x00000] in :0
[PID:29449:007 2013.05.23 15:08:05.577 ERROR QueuedHandlerAutoRes] Error while processing message EventStore.Core.Messages.StorageMessage+WritePrepares in queued handler ‘StorageWriterQueue’.
System.IO.IOException: Too many open files
at System.IO.FileStream…ctor (System.String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean anonymous, FileOptions options) [0x00000] in :0
at System.IO.FileStream…ctor (System.String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options) [0x00000] in :0
at (wrapper remoting-invoke-with-check) System.IO.FileStream:.ctor (string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare,int,System.IO.FileOptions)
at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.CreateInternalReaderWorkItem () [0x00000] in :0
at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.CreateReaderStreams () [0x00000] in :0
at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.InitNew (EventStore.Core.TransactionLog.Chunks.ChunkHeader chunkHeader, Int32 fileSize) [0x00000] in :0
at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.CreateWithHeader (System.String filename, EventStore.Core.TransactionLog.Chunks.ChunkHeader header, Int32 fileSize) [0x00000] in :0
[PID:29449:007 2013.05.23 15:08:05.580 ERROR StorageWriterService] Exception in writer.
System.NullReferenceException: Object reference not set to an instance of an object
at EventStore.Core.TransactionLog.Chunks.TFChunkWriter.Write (EventStore.Core.TransactionLog.LogRecords.LogRecord record, System.Int64& newPos) [0x00000] in :0
at EventStore.Core.Services.Storage.StorageWriterService.WritePrepareWithRetry (EventStore.Core.TransactionLog.LogRecords.PrepareLogRecord prepare) [0x00000] in :0
at EventStore.Core.Services.Storage.StorageWriterService.EventStore.Core.Bus.IHandle<EventStore.Core.Messages.StorageMessage.WritePrepares>.Handle (EventStore.Core.Messages.WritePrepares message) [0x00000] in :0
[PID:29449:007 2013.05.23 15:08:05.580 ERROR QueuedHandlerAutoRes] Error while processing message EventStore.Core.Messages.StorageMessage+WritePrepares in queued handler ‘StorageWriterQueue’.
System.NullReferenceException: Object reference not set to an instance of an object
at EventStore.Core.TransactionLog.Chunks.TFChunkWriter.Write (EventStore.Core.TransactionLog.LogRecords.LogRecord record, System.Int64& newPos) [0x00000] in :0
at EventStore.Core.Services.Storage.StorageWriterService.WritePrepareWithRetry (EventStore.Core.TransactionLog.LogRecords.PrepareLogRecord prepare) [0x00000] in :0
at EventStore.Core.Services.Storage.StorageWriterService.EventStore.Core.Bus.IHandle<EventStore.Core.Messages.StorageMessage.WritePrepares>.Handle (EventStore.Core.Messages.WritePrepares message) [0x00000] in :0
[PID:29449:007 2013.05.23 15:08:05.580 ERROR StorageWriterService] Exception in writer.
As we near this size, this is scary!
Dumb question as I made the same mistake, did you do it for the account the event store is actually running under? (10,000 for 50gb was fine for me iirc)
I think so, it’s running under supervisord as root…actually come to think as I type - I didn’t restart supervisor after I updated the file handle limit so I’ll try doing that!
50GB is a lot, though it managed just fine with more than 10 times that on Windows…
You have to force a relogin to get the new limits iirc