Too many open files

EventStoreDB
1

Hello community

It’s perhaps more an sysadmin question, but i’m running mad after I have tested a lot of scripts with no success

I got this error a while ago on a test instance, and since i was test, I handled it by flushing the databse. Now i wanna go production, I have too make it.

So after around 50 GB written with WRFL, finally it happens (here is the error log) :

[PID:31205:044 2017.05.05 13:14:37.555 DEBUG MonitoringService ] Could not get free mem on linux, received memory info raw string: []
System.IO.IOException: Too many open files
at System.Diagnostics.Process.CreatePipe (System.IntPtr& read, System.IntPtr& write, System.Boolean writeDirection) [0x00014] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0012c] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
[PID:31205:044 2017.05.05 13:14:37.556 DEBUG MonitoringService ] Could not get drive name for directory ‘/ges/’ on Unix.
System.IO.IOException: Too many open files
at System.Diagnostics.Process.CreatePipe (System.IntPtr& read, System.IntPtr& write, System.Boolean writeDirection) [0x00014] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0012c] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
[PID:31205:063 2017.05.05 13:15:07.584 DEBUG MonitoringService ] Could not get free mem on linux, received memory info raw string: []
System.IO.IOException: Too many open files
at System.Diagnostics.Process.CreatePipe (System.IntPtr& read, System.IntPtr& write, System.Boolean writeDirection) [0x00014] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0012c] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
[PID:31205:063 2017.05.05 13:15:07.584 DEBUG MonitoringService ] Could not get drive name for directory ‘/ges/’ on Unix.
System.IO.IOException: Too many open files
at System.Diagnostics.Process.CreatePipe (System.IntPtr& read, System.IntPtr& write, System.Boolean writeDirection) [0x00014] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0012c] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
[PID:31205:043 2017.05.05 13:15:37.630 DEBUG MonitoringService ] Could not get free mem on linux, received memory info raw string: []
System.IO.IOException: Too many open files
at System.Diagnostics.Process.CreatePipe (System.IntPtr& read, System.IntPtr& write, System.Boolean writeDirection) [0x00014] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0012c] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
[PID:31205:043 2017.05.05 13:15:37.631 DEBUG MonitoringService ] Could not get drive name for directory ‘/ges/’ on Unix.
System.IO.IOException: Too many open files
at System.Diagnostics.Process.CreatePipe (System.IntPtr& read, System.IntPtr& write, System.Boolean writeDirection) [0x00014] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0012c] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
[PID:31205:015 2017.05.05 13:16:01.903 ERROR StorageWriterService] Exception in writer.
System.IO.IOException: Too many open files
at System.IO.FileStream…ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) [0x0025f] in <8f2c484307284b51944a1a13a14c0266>:0
at System.IO.FileStream…ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.IO.FileOptions options) [0x00000] in <8f2c484307284b51944a1a13a14c0266>:0
at (wrapper remoting-invoke-with-check) System.IO.FileStream:.ctor (string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare,int,System.IO.FileOptions)
at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.CreateInternalReaderWorkItem () [0x00030] in :0
at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.CreateReaderStreams () [0x00019] in :0
at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.InitNew (EventStore.Core.TransactionLog.Chunks.ChunkHeader chunkHeader, System.Int32 fileSize) [0x0006b] in :0
at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.CreateWithHeader (System.String filename, EventStore.Core.TransactionLog.Chunks.ChunkHeader header, System.Int32 fileSize, System.Boolean inMem, System.Boolean unbuffered, System.Boolean writethrough) [0x00011] in :0
[PID:31205:015 2017.05.05 13:16:01.908 FATAL StorageWriterService] Unexpected error in StorageWriterService. Terminating the process…
System.IO.IOException: Too many open files
at System.IO.FileStream…ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) [0x0025f] in <8f2c484307284b51944a1a13a14c0266>:0
at System.IO.FileStream…ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.IO.FileOptions options) [0x00000] in <8f2c484307284b51944a1a13a14c0266>:0
at (wrapper remoting-invoke-with-check) System.IO.FileStream:.ctor (string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare,int,System.IO.FileOptions)
at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.CreateInternalReaderWorkItem () [0x00030] in :0
at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.CreateReaderStreams () [0x00019] in :0
at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.InitNew (EventStore.Core.TransactionLog.Chunks.ChunkHeader chunkHeader, System.Int32 fileSize) [0x0006b] in :0
at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.CreateWithHeader (System.String filename, EventStore.Core.TransactionLog.Chunks.ChunkHeader header, System.Int32 fileSize, System.Boolean inMem, System.Boolean unbuffered, System.Boolean writethrough) [0x00011] in :0
[PID:31205:015 2017.05.05 13:16:01.968 ERROR Application ] Exiting with exit code: 1.
Exit reason: Unexpected error in StorageWriterService: Too many open files
[PID:19475:008 2017.05.05 13:16:36.076 DEBUG MonitoringService ] Could not get drive name for directory ‘/ges/’ on Unix.
System.IO.IOException: Too many open files
at System.Diagnostics.Process.CreatePipe (System.IntPtr& read, System.IntPtr& write, System.Boolean writeDirection) [0x00014] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0012c] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
[PID:19475:026 2017.05.05 13:18:36.685 DEBUG MonitoringService ] Could not get free mem on linux, received memory info raw string: []
System.IO.IOException: Too many open files
at System.Diagnostics.Process.CreatePipe (System.IntPtr& read, System.IntPtr& write, System.Boolean writeDirection) [0x00014] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0012c] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
[PID:19475:026 2017.05.05 13:18:36.687 DEBUG MonitoringService ] Could not get drive name for directory ‘/ges/’ on Unix.
System.IO.IOException: Too many open files
at System.Diagnostics.Process.CreatePipe (System.IntPtr& read, System.IntPtr& write, System.Boolean writeDirection) [0x00014] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0012c] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
[PID:19475:025 2017.05.05 13:19:36.805 DEBUG MonitoringService ] Could not get free mem on linux, received memory info raw string: []
System.IO.IOException: Too many open files
at System.Diagnostics.Process.CreatePipe (System.IntPtr& read, System.IntPtr& write, System.Boolean writeDirection) [0x00014] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0012c] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
[PID:19475:025 2017.05.05 13:19:36.809 DEBUG MonitoringService ] Could not get drive name for directory ‘/ges/’ on Unix.
System.IO.IOException: Too many open files
at System.Diagnostics.Process.CreatePipe (System.IntPtr& read, System.IntPtr& write, System.Boolean writeDirection) [0x00014] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0012c] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
[PID:19475:043 2017.05.05 13:20:36.888 DEBUG MonitoringService ] Could not get free mem on linux, received memory info raw string: []
System.IO.IOException: Too many open files
at System.Diagnostics.Process.CreatePipe (System.IntPtr& read, System.IntPtr& write, System.Boolean writeDirection) [0x00014] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0012c] in <5071a6e4a4564e19a2eda0f53e42f9bd>:0
[PID:19475:043 2017.05.05 13:20:36.889 DEBUG MonitoringService ] Could not get drive name for directory ‘/ges/’ on Unix.

``

It’s talking about memory, but I have 15 GB on the machine, only 1.5 used last time I checked.

As mentioned here (https://groups.google.com/forum/#!searchin/event-store/Too$20many$20open$20files|sort:relevance/event-store/ESDA7iFpTrc/2PXBMmmn-ZMJ), I tried some ulimit tricks.

root@GES-TEST:/# su -l eventstore -s /bin/bash
eventstore@GES-TEST:~$ ulimit -n
64000

``

64 000 files seems big to me since i only have 195 chunk files in eventstore db directory right now …

Also tried some (https://underyx.me/2015/05/18/raising-the-maximum-number-of-file-descriptors)
cat > /etc/security/limits.conf <<- EOM

eventstore soft nofile 64000
eventstore hard nofile 64000

  • soft nofile 64000
  • hard nofile 64000
    eventstore soft nofile 64000
    eventstore hard nofile 64000
    root soft nofile 64000
    root hard nofile 64000

EOM

``

Then some

echo “session required pam_limits.so” >> /etc/pam.d/common-session
echo “session required pam_limits.so” >> /etc/pam.d/common-session-noninteractive

``

Or

echo “limit nofile 40000 65000” | sudo tee -a /etc/init/eventstore.conf
sudo sysctl -w fs.file-max=100000
sudo echo “sysctl -w fs.file-max=100000” >> /etc/sysctl.conf

``

Well, nothing works.

Right now after I restarted it manually, the eventstore service is unstable and periodicaly change it’s PID so it’s crashing … and nothing more appending to the logs …

I don’t know what to do …

any thoughts ?

2

Nobody on this one ?

3
  1. What OS are you on?
  2. Just out of curiosity what version of Event Store?
4

Ubuntu 16.04 - ES 4.0.1.0

installed like this :

curl -s https://packagecloud.io/install/repositories/EventStore/EventStore-OSS/script.deb.sh | sudo bash
sudo apt-get update
sudo apt-get install -y debian-archive-keyring
sudo apt-get install -y eventstore-oss

``

rm /etc/eventstore/eventstore.conf

cat > /etc/eventstore/eventstore.conf <<- EOM

RunProjections: All
ClusterSize: 1
ExtIp: $(ifconfig eth0 | grep “inet addr” | cut -d ‘:’ -f 2 | cut -d ’ ’ -f 1)
ExtTcpPort: 1112
ExtHttpPort: 2114
ExtHttpPrefixes: http://:2114/
IntTcpPort: 1111
IntHttpPort: 2113
IntHttpPrefixes: http://:2113/
AddInterfacePrefixes: false

EOM

``

sudo service eventstore start

``

5

The limits that you have specified should be fine but i’m curious as to whether they are actually in effect.

One way to check is to check the process’ limits via

cat/proc/{eventstore-pid}/limits

``

6

Hi Jerome,
try to add:

LimitNOFILE=65536 in /etc/systemd/system/eventstore.service

7

We’re getting this error all of a sudden on our production system. File limit set to 94,000… no idea what is wrong. Did you find a solution?

8

Appears somewhere in the eventstore software the limit is being set, as the system hard and soft limits for all users (including root) are set to 94,000. However, cat/proc/[eventstoreprocessid/limits is returning 1024/4096

The LimitNOFILE=65536 in /etc/systemd/system/eventstore.service solution didn’t work for us.

9

Anyone have any ideas what the config file command would be to set the open files? Tried

LimitNoFile: 94000

But fails…

10

ES does not handle file limits, the OS does. ES does not change the limits.

11

There are multiple levels of configuration in linux btw, you need to figure out where your effective settings are coming from.

12

Linux uses several different levels of resource limits, but I would imagine that you are running into either a system limit (less likely) or a user limit (more likely). For us (ES 4.1.4 on AWS Amazon Linux host) both needed increased. A very good overview on how to modify these specific limits can be found here: https://easyengine.io/tutorials/linux/increase-open-files-limit/

Hope this helps

– Mitch

13

If you run event store as a service, you need to increase the service limits as well.

Edit /etc/systemd/system/eventstore.service

Under [Services] add:

LimitNOFILE=500000

LimitNPROC=500000