EventStore keeps restarting ReadIndex Rebuilding

I have increased open files to 50000 but it seems not to be enough: Event Store 4.0.1.4
Do i have to keep on adding or increasing the open file?

Logs:

[PID:18126:023 2018.01.15 23:15:53.977 FATAL GLOBAL-LOGGER ] Global Unhandled Exception occurred.

System.IO.IOException: Too many open files

at System.IO.FileStream…ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOpti

ons options) [0x0025f] in <8f2c484307284b51944a1a13a14c0266>:0

at System.IO.FileStream…ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.IO.FileOptions options) [0x00000] in

<8f2c484307284b51944a1a13a14c0266>:0

at (wrapper remoting-invoke-with-check) System.IO.FileStream:.ctor (string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare,int,System.IO.FileOptions)

at EventStore.Core.Index.PTable+WorkItem…ctor (System.String filename, System.Int32 bufferSize) [0x00006] in :0

at EventStore.Core.Index.PTable+c__AnonStorey1.<>m__0 () [0x00000] in :0

at EventStore.Core.DataStructures.ObjectPool1[T]..ctor (System.String objectPoolName, System.Int32 initialCount, System.Int32 maxCount, System.Func1[TResult] factory, System.Action`1[T] dispose, Syst

em.Action`1[T] onPoolDisposed) [0x000d1] in :0

at EventStore.Core.Index.PTable…ctor (System.String filename, System.Guid id, System.Int32 initialReaders, System.Int32 maxReaders, System.Int32 depth) [0x00102] in :

0

at EventStore.Core.Index.PTable.MergeTo2 (System.Collections.Generic.IList1[T] tables, System.Int64 fileSize, System.Int32 indexEntrySize, System.String outputFile, System.Func3[T1,T2,TResult] upgrad

eHash, System.Func2[T,TResult] existsAt, System.Func2[T,TResult] readRecord, System.Byte version, System.Int32 cacheDepth) [0x002d6] in :0

at EventStore.Core.Index.PTable.MergeTo (System.Collections.Generic.IList1[T] tables, System.String outputFile, System.Func3[T1,T2,TResult] upgradeHash, System.Func`2[T,TResult] existsAt, System.Func

`2[T,TResult] readRecord, System.Byte version, System.Int32 cacheDepth) [0x00066] in :0

at EventStore.Core.Index.IndexMap.AddPTable (EventStore.Core.Index.PTable tableToAdd, System.Int64 prepareCheckpoint, System.Int64 commitCheckpoint, System.Func`3[T1,T2,TResult] upgradeHash, System.Fun

c2[T,TResult] existsAt, System.Func2[T,TResult] recordExistsAt, EventStore.Core.Index.IIndexFilenameProvider filenameProvider, System.Byte version, System.Int32 indexCacheDepth) [0x00069] in <caba33f27

9964903a4792ae6112d9f47>:0

at EventStore.Core.Index.TableIndex.ReadOffQueue () [0x0011b] in :0

[PID:00991:023 2018.01.15 23:55:39.378 DEBUG MonitoringService ] Could not get free mem on linux, received memory info raw string: []

System.IO.IOException: Too many open files

at System.Diagnostics.Process.CreatePipe (System.IntPtr& read, System.IntPtr& write, System.Boolean writeDirection) [0x00014] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0

at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0012c] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0

[PID:00991:023 2018.01.15 23:55:39.386 DEBUG MonitoringService ] Could not get drive name for directory ‘/home/ubuntu/eventstore’ on Unix.

System.IO.IOException: Too many open files

at System.Diagnostics.Process.CreatePipe (System.IntPtr& read, System.IntPtr& write, System.Boolean writeDirection) [0x00014] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0

at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0012c] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0

[PID:00991:027 2018.01.15 23:56:09.388 DEBUG MonitoringService ] Could not get free mem on linux, received memory info raw string: []

System.IO.IOException: Too many open files

at System.Diagnostics.Process.CreatePipe (System.IntPtr& read, System.IntPtr& write, System.Boolean writeDirection) [0x00014] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0

at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0012c] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0

[PID:00991:027 2018.01.15 23:56:09.389 DEBUG MonitoringService ] Could not get drive name for directory ‘/home/ubuntu/eventstore’ on Unix.

System.IO.IOException: Too many open files

at System.Diagnostics.Process.CreatePipe (System.IntPtr& read, System.IntPtr& write, System.Boolean writeDirection) [0x00014] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0

at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0012c] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0

[PID:00991:036 2018.01.15 23:57:09.393 DEBUG MonitoringService ] Could not get drive name for directory ‘/home/ubuntu/eventstore’ on Unix.

System.IO.IOException: Too many open files

Upgraded to EventStore 4.0.3 hotfix 1 still happening. INcreased open file limit to 800000

Apparently wasn’t succesfull to increase file limit for eventstore, still at 1024. Have tried many options still won’t increase.

For anyone having the same issue:

Increase the ulimit by modifying /etc/init/eventstore.conf

// put in after start on runlevel

limit nofile 524288 1048576

limit nproc 524288 1048576

``

Yeah, we had this issue yesterday and increasing the open file limit in the service conf file helped.