Event Store eating up all most available memory

EventStoreDB
61

I have just started it up without my Projection running, so I will keep an eye on it.

For reference, here is my only projection that’s running:

fromAll()

.when({

‘Tahoe.Eposity.SalesTransactions.DomainEvents.SalesTransactionSummaryAddedEvent’: function (s, e) {

var streamName = ‘AllSales_’ + e.data.MetaData.organisationId.replace(/-/gi,"");

var eventDate = new Date(e.data.CompletionDateTime);

var month = eventDate.getMonth() + 1;

var day = eventDate.getDate().toString().paddingLeft(“00”);

var monthlyStream = streamName + ‘_’ + eventDate.getFullYear() + month.toString().paddingLeft(“00”);

var dailyStream = streamName + ‘_’ + eventDate.getFullYear() + month.toString().paddingLeft(“00”) + day;

linkTo(monthlyStream, e);

linkTo(dailyStream, e);

}

});

String.prototype.paddingLeft = function (paddingValue) {

return String(paddingValue + this).slice(-paddingValue.length);

};

62

That shouldnt cause memory issues. (not using any state etc)

63

Another thing to try is turning off stats (its another periodic operation)

64

Not sure if this is any help, but the command window has a lot of activity going.
I have pasted a snippet out to the attachment.

ES Command Window Trace.txt (11 KB)

65

thats your projection writing events

66

Todat is the firts morning we have come in and our ES is still running.
Only difference is our only projection was turned off!

Memory usage is sitting roughly the same as it was when I went home last night, so it seems this is the culprit.

I need to investigate the code, but my start point will be this:

fromAll()

.when({

‘Tahoe.Eposity.SalesTransactions.DomainEvents.SalesTransactionSummaryAddedEvent’: function (s, e) {

//Our code

}

I have a funny feeling “fromAll” is not what I want. A SalesTransactionSummaryAddedEvent would only ever be written to a specific category stream ($ce-SalesTransactions) so I wonder if I am somehow hosing the Event Store, and as time goes on, the amount of streams I am creating will only increase (and maybe this is the source of the problem)

67

I’m running ES in docker container and today to the same error in my logging service.

Some facts:

  • I am running ES on a VM (DigitalOcean’s droplet) with 1GB max memory

  • I am running ES in docker container

Is there any way to limit the docker container’s memory size to 800MB max from inside Dockerfile that you guys provide?

[ AGENT ] 2018-08-27 16:32:01 UTC | INFO | (ntp.go:135 in Run) | There was an error querying the ntp host: read udp 172.18.0.4:41928->129.6.15.28:123: i/o timeout

ApplicationName=‘df’, CommandLine=’-P /var/lib/eventstore’, CurrentDirectory=’’, Native error= Out of memory

``
68

20/hour, 90/hour, 180/hour and now 430 errors / hour, saying “Out of memory”, but yet the ES keeps working normally and responds to queries…

69

The “out of memory” errors are usually when its trying to reserve memory and not being allowed. It should continue running but with degraded performance under most circumstances.

70

You likely also want to change some of the defaults for memory usage like how many chunks to cache etc.

71

Issue saying it might be version of Mono 4.6:
https://github.com/EventStore/EventStore/issues/1481

Will investigate.

72

Ok, the issue was I ran out of DISK SPACE hahaha!

73

That seems like a legitimate reason to run into issues :slight_smile: