EventStore keeps restarting ReadIndex Rebuilding

EventStoreDB
1

I have increased open files to 50000 but it seems not to be enough: Event Store 4.0.1.4
Do i have to keep on adding or increasing the open file?

Logs:

[PID:18126:023 2018.01.15 23:15:53.977 FATAL GLOBAL-LOGGER ] Global Unhandled Exception occurred.

System.IO.IOException: Too many open files

at System.IO.FileStream…ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOpti

ons options) [0x0025f] in <8f2c484307284b51944a1a13a14c0266>:0

at System.IO.FileStream…ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.IO.FileOptions options) [0x00000] in

<8f2c484307284b51944a1a13a14c0266>:0

at (wrapper remoting-invoke-with-check) System.IO.FileStream:.ctor (string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare,int,System.IO.FileOptions)

at EventStore.Core.Index.PTable+WorkItem…ctor (System.String filename, System.Int32 bufferSize) [0x00006] in :0

at EventStore.Core.Index.PTable+c__AnonStorey1.<>m__0 () [0x00000] in :0

at EventStore.Core.DataStructures.ObjectPool1[T]..ctor (System.String objectPoolName, System.Int32 initialCount, System.Int32 maxCount, System.Func1[TResult] factory, System.Action`1[T] dispose, Syst

em.Action`1[T] onPoolDisposed) [0x000d1] in :0

at EventStore.Core.Index.PTable…ctor (System.String filename, System.Guid id, System.Int32 initialReaders, System.Int32 maxReaders, System.Int32 depth) [0x00102] in :

0

at EventStore.Core.Index.PTable.MergeTo2 (System.Collections.Generic.IList1[T] tables, System.Int64 fileSize, System.Int32 indexEntrySize, System.String outputFile, System.Func3[T1,T2,TResult] upgrad

eHash, System.Func2[T,TResult] existsAt, System.Func2[T,TResult] readRecord, System.Byte version, System.Int32 cacheDepth) [0x002d6] in :0

at EventStore.Core.Index.PTable.MergeTo (System.Collections.Generic.IList1[T] tables, System.String outputFile, System.Func3[T1,T2,TResult] upgradeHash, System.Func`2[T,TResult] existsAt, System.Func

`2[T,TResult] readRecord, System.Byte version, System.Int32 cacheDepth) [0x00066] in :0

at EventStore.Core.Index.IndexMap.AddPTable (EventStore.Core.Index.PTable tableToAdd, System.Int64 prepareCheckpoint, System.Int64 commitCheckpoint, System.Func`3[T1,T2,TResult] upgradeHash, System.Fun

c2[T,TResult] existsAt, System.Func2[T,TResult] recordExistsAt, EventStore.Core.Index.IIndexFilenameProvider filenameProvider, System.Byte version, System.Int32 indexCacheDepth) [0x00069] in <caba33f27

9964903a4792ae6112d9f47>:0

at EventStore.Core.Index.TableIndex.ReadOffQueue () [0x0011b] in :0

[PID:00991:023 2018.01.15 23:55:39.378 DEBUG MonitoringService ] Could not get free mem on linux, received memory info raw string: []

System.IO.IOException: Too many open files

at System.Diagnostics.Process.CreatePipe (System.IntPtr& read, System.IntPtr& write, System.Boolean writeDirection) [0x00014] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0

at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0012c] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0

[PID:00991:023 2018.01.15 23:55:39.386 DEBUG MonitoringService ] Could not get drive name for directory ‘/home/ubuntu/eventstore’ on Unix.

System.IO.IOException: Too many open files

at System.Diagnostics.Process.CreatePipe (System.IntPtr& read, System.IntPtr& write, System.Boolean writeDirection) [0x00014] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0

at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0012c] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0

[PID:00991:027 2018.01.15 23:56:09.388 DEBUG MonitoringService ] Could not get free mem on linux, received memory info raw string: []

System.IO.IOException: Too many open files

at System.Diagnostics.Process.CreatePipe (System.IntPtr& read, System.IntPtr& write, System.Boolean writeDirection) [0x00014] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0

at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0012c] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0

[PID:00991:027 2018.01.15 23:56:09.389 DEBUG MonitoringService ] Could not get drive name for directory ‘/home/ubuntu/eventstore’ on Unix.

System.IO.IOException: Too many open files

at System.Diagnostics.Process.CreatePipe (System.IntPtr& read, System.IntPtr& write, System.Boolean writeDirection) [0x00014] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0

at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0012c] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0

[PID:00991:036 2018.01.15 23:57:09.393 DEBUG MonitoringService ] Could not get drive name for directory ‘/home/ubuntu/eventstore’ on Unix.

System.IO.IOException: Too many open files

2

Upgraded to EventStore 4.0.3 hotfix 1 still happening. INcreased open file limit to 800000

3

Apparently wasn’t succesfull to increase file limit for eventstore, still at 1024. Have tried many options still won’t increase.

4

For anyone having the same issue:

Increase the ulimit by modifying /etc/init/eventstore.conf

// put in after start on runlevel

limit nofile 524288 1048576

limit nproc 524288 1048576

``

5

Yeah, we had this issue yesterday and increasing the open file limit in the service conf file helped.