Event Store memory leak?

EventStoreDB
1

I have a number of ubuntu AWS EC2 instances solely running Event Store and regularly experience a scenario where the server becomes unresponsive and requires a reboot to recover.

On investigation I can see that Event Store’s memory use increases as time goes by to a point where there are no more memory resources available on the server.

An example is as follows, where Event Store has gone from using 41.9% of memory to 75.5% in 6 days.

Ubuntu 16.04.5 LTS running Event Store 4.1.1.0

Wed Jan 23 09:39:20

ubuntu:~$ free -m

total used free shared buff/cache available

Mem: 3950 1723 222 40 2004 1931

Swap: 0 0 0

ubuntu:~$ top

top - 09:40:00 up 1 day, 22:12, 1 user, load average: 0.00, 0.03, 0.00

Tasks: 121 total, 1 running, 120 sleeping, 0 stopped, 0 zombie

%Cpu(s): 0.2 us, 3.2 sy, 0.0 ni, 96.3 id, 0.2 wa, 0.0 hi, 0.0 si, 0.2 st

KiB Mem : 4045040 total, 227640 free, 1765168 used, 2052232 buff/cache

KiB Swap: 0 total, 0 free, 0 used. 1977520 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

1442 eventst+ 20 0 0.157t 1.616g 27104 S 7.7 41.9 168:36.98 Main

1209 root 20 0 19472 256 0 S 0.3 0.0 0:05.17 irqbalance

1 root 20 0 37696 5448 3712 S 0.0 0.1 0:36.00 systemd

2

Mark,

I have been down this route quite a few times where the ES runs out of memory, and every time it’s been related to Projections (usually how I read the streams)

How many projections is your Event Store running, and could you share them (even privately)

If you have a search through the google group, you will find a few posts from me on the same issue.

3

One of my older posts:

4

What are your settings? much of how memory ES uses is configurable in
quite a few ways. As example by default 0.5 GB is used as immediate
caching for the last two chunk files. This can be changed.
--cached-chunk-count (IIRC check the command line args)

5

Thanks for that.
We are using fromAll() on our projection so are now looking to replace with a different selector.

6

Greg

Our settings are all defaulted. Our projection uses the fromAll() selector which perhaps is the source of our issue - I’m currently looking to replace with an alternative to focus on just the streams we really need however ordering of the events seem to change when being stream specific using fromStream({streamId}) .

7

fromStream should offer perfect ordering. Can you give an example?

8

Our projection (I’m a work colleague of MarkJ) uses the following structure, which on initial testing appeared to give us with the TransactionHasCompletedEvent before the TransactionHasStartedEvent. This however may not have been the case, so we are going to do some further testing and will provide an update when done. Best case for us is that we do get perfect ordering.

fromStreams("$et-TransactionProcessor.DomainEvents.TransactionHasStartedEvent",

“$et-TransactionProcessor.DomainEvents.TransactionHasCompletedEvent”)

.when(

{

$init: function (s, e) {

return { TransactionMappings: [] }

},

“TransactionProcessor.DomainEvents.TransactionHasStartedEvent”: processTransactionHasStartedEvent,

“TransactionProcessor.DomainEvents.TransactionHasCompletedEvent”: processTransactionHasCompletedEvent

}

)

9

Chris,

Is your worry about the order they get replayed on a Projection Reset?

10

Yes, that is correct Steven.

During original development the projection was being reset and I’m almost certain we saw events coming in out of order. Note that these were from different streams, so the order the events are written to ES are

TransactionProcessor.DomainEvents.TransactionHasStartedEvent
TransactionProcessor.DomainEvents.TransactionHasCompletedEvent

However, I’m sure I saw some of the TransactionHasCompletedEvent being replayed before the TransactionHasStartedEvent after a reset.

11

Hmm, I couldn’t say for certain the order then.