Event Store generally faulting after VM power cycle

EventStoreDB
#1

Environment

VirtualBox VM running Ubuntu 14.04 (dedicated to running Event Store)

Event Store 3.9.3, projections enabled

What happened

ES was running fine - we had written lots of events across multiple streams. At some point the host machine (Windows 10) was rebooted by IT - not entirely sure what happened to the VM at that point (i.e. whether a proper shutdown signal was sent or whether it was a hard reset - whatever VirtualBox does by default)

When we restarted the VM, Event Store no longer allowed us to log in to the admin site (we were using the default admin:changeit creds as it is a dev environment). We got the orange footer warning: “Server does not exist or incorrect user credentials” supplied.

System state

Closer inspection of the running VM revealed:

  • eventstore service is running
  • both standard log and error log being constantly written to (despite no ongoing usage of the system from external)
    The standard log appeared to be full of messages that appeared to indicate that the projections system was simply initializing over and over.

The error log was written to every few seconds. This is my colleague’s machine and I won’t have access for an hour or two but here are a couple snapshots of the error log:

#2

This looks like a kernel regression as noted https://github.com/EventStore/EventStore/issues/1085 and https://github.com/EventStore/EventStore/issues/678
What version of the kernel are you running?

There is a list of known good versions of the Kernel on the README right at the bottom.

#3

Looks like we are running 4.2.0-27-generic on our VBoxes. (This is based on just building a new Ubuntu box from VBox’s tools.)

We also have a test virtual Ubuntu server at kernel 3.13.0-32-generic. This one isn’t strictly on the list either but it’s a lot closer.

I thought we needed Ubuntu distribution 14.04 to be able to smoothly install the eventstore-oss package, so both systems are using that distribution. I am reading about potential issues combining Ubuntu 14.04 with kernel 4.4+.

#4

Update: just downgraded VM’s kernel to 3.19.0-66-generic. And so far it looks like it stabilized…

#5

Pieter, that list of known good versions of the kernel has now disappeared from the README. What am I to make of that? Is there a new version of Event Store that makes the kernel regression issue go away entirely, or has the list of known good versions simply been moved elsewhere?

#6

Hi Zach,
Thanks for mentioning this. As Event Store has been upgraded to mono 4, we have started testing on newer kernels, the list is no longer valid for 4.0.0.

Will submit a PR to bring back the kernel list and note that they pertain to releases of Event Store pre 4.0.0.

You can find the known good versions of the kernel here

Apologies about that.

#7

Found it in the diff :slight_smile: One final question: it says: " >= 4.4.27" - as in, anything greater than that version is OK. I assume this is a typo for: “4.4.0-27” - yes?

#8

This last question is pertinent because in one of the issues that you linked, someone was able to reproduce the issue with ES 3.9.0, Ubuntu 14.04.5, and kernel 4.4.0-34-generic.

It appears we will not be able to use 3.19.0-66-generic on production; that version has known vulnerabilities. We’ll have to go with 4.4 or later. But when I look at the list of supported kernel packages for Ubuntu 14.04.5, I see only 4.4.0-*, not 4.4.27 or later, so I need to double check if that listed “4.4.27” was a typo or not…

#9

Update: sadly it appears that 4.4.0-59-generic is susceptible to this issue, as indicated by the linked threads from Pieter’s post above.

Not sure if I can go to any higher kernel version for Ubuntu 14.04.5 - which puts me in a bind. Do I need to wait for ES 4 to come out before I can get a working system with projections on Ubuntu 14.04?