We are experiencing a reappearing problem where GES (version 3.0.1 through 3.0.5) is taking up a lot of memory for apparently no good reason.
It is running in a single node configuration on Ubuntu 14.04.2 LTS using this command line:
./run-node.sh --db /var/lib/eventstore --cached-chunks=0
We added --cached-chunks=0 after seeing the same behavior and searching this group for a solution.
It runs inside a VM with 2 available cores at 2.8 GHz and 6 GB RAM. The underlying hardware is mostly idle, so it is not starving for IO.
We are using GES through its HTTP interface and right now the software is idle (as it is most of the time, it usually has only 2 bursts of activity throughout the day with a few hundred events happening in several minutes; i.e. it doesn’t strain the system nor GES at all).
The log contains the following bits repeatedly:
[23549,10,18:39:10.157] SLOW BUS MSG [MainBus]: PurgeTimedOutRequests - 57ms. Handler: HttpService.
[23549,11,18:39:10.168] SLOW BUS MSG [manager input bus]: RegularTimeout - 63ms. Handler: ProjectionCoreCoordinator.
[23549,10,18:39:10.177] SLOW QUEUE MSG [MainQueue]: PurgeTimedOutRequests - 207ms. Q: 0/5.
[23549,11,18:39:10.177] SLOW QUEUE MSG [Projections Master]: RegularTimeout - 80ms. Q: 0/1.
[23549,70,18:39:34.513] Couldn’t get drive name for directory ‘/var/lib/eventstore’ on Unix.
ApplicationName=‘df’, CommandLine=’-P /var/lib/eventstore’, CurrentDirectory=’’, Native error= Out of memory
Usually the Linux Out Of Memory killer will kill the event store somewhen if we leave this state for long enough. So it is not an issue with df, as assumed in another thread where probably the same problem came up but which never saw a conclusion.
In the last snapshot before the oom-killer has gone to do its work, GES was hogging 4787M of resident memory, after GES restarted it was using only 292MB of resident memory which is a bit less then what it usually uses when it works as expected.
Right now we are still evaluating GES, so these irregular crashes are not interfering with production. We really like the features of GES, but with that kind of stability problems we certainly will not be able to use it.
So are we just using (configuring) it wrong? Does GES need more memory than 6 GB in oder to handle several (at most) thousand messages per day? Do you need any more information?