Running/StateLoaded projections stopped

EventStoreDB
#1

Hello there,

We had a little issue here, 2 projections died on a “Running/StateLoaded” state (Event Store 3.8.1.0).

The only place I found info about this state was here :

https://groups.google.com/forum/m/#!topic/event-store/-Zm43xnEbjY

(and it doesn’t sound like a normal state)

It’s a little cluster of 2 nodes, and after stopped/restarted the master node (only this one) all is going smooth again.

Logs are in attachement if someone could say what to look for in order to understand this crash ?

Also, is there an event we can project somehow to warn us when a projection state change ?

Cheers !

Jérôme

logs.zip (4.68 MB)

#2

Will look. Out of curiosity why run a 2 node cluster? 2 nodes will
have less availability than a single node? :slight_smile:

#3

I have seen this issue occur from time to time and it appears that it’s related to the v8 process not shutting down properly and compiling the projections fails. I have also observed this issue on Windows, I don’t recall seeing this on Linux.
Have put an issue up for it so that it can be tracked for 4.0. https://github.com/EventStore/EventStore/issues/1032

#4

@Greg … we try to run GES on the cheapest (windows) configuration we could => An azure “S” VM,
After a while having the only server crying about performances, we added one node to catch up the first one.
We have to move to better machines, soon or later, we’ll probably move to linux machines witch are 40% cheaper in Azure (and better memory consumption …) but we right now are not linux guys, so one thing after another.

@Pieter, that exactly what happened.

Perhaps it’s a problem about not enought memory available to run the compilation ?

Our 2 nodes have 1.75 GB memory on windows, with 95% more the memory almost always consumed … it’s not enough, perhaps the V8 engine doesn’t answer in time because it’s just waiting for more memory ?

or … not that …

The same database (little one : 2GB) backuped and restore on my local computer runs just perfectly.

#5

@Jerome my point was that running two nodes has less availability than
running a single node :slight_smile: If you run two nodes and either goes down the
other will not be able to form a quorum and will be unavailable.
Assuming each node had 99% availability you would get 99% availability
on a single node and 98.01% availability with two nodes.