fatal error when using in docker

Ugo_Bourdon · April 3, 2015, 11:47am

I use geteventstore in docker but i had regularly fatal error with this stacktrace :

tacktrace:

Native stacktrace:

./clusternode() [0x612962]
./clusternode() [0x5beb0b]
./clusternode() [0x4584f3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fe48c8c6340]
./clusternode() [0x60fc0e]
./clusternode() [0x6139aa]
./clusternode() [0x5bcebd]
[0x41bc8377]

Debug info from gdb:

Ugo_Bourdon · April 3, 2015, 11:47am

The geteventstore version is 3.0.3

Greg_Young1 · April 3, 2015, 11:56am

Seen twice in two days on the list. Can you provide a backtrace?

Also knowing things like what operating system you are running in as
well as what version of docker you are using could help.

Ugo_Bourdon · April 3, 2015, 12:33pm

Sorry what you mean by backtrace ?
My system is : Ubuntu 14.04.2 LTS
Docker version : Docker version 1.5.0, build a8a31ef

Thx for help.

Ugo_Bourdon · April 3, 2015, 12:36pm

New crash for new stacktrace :

Object reference not set to an instance of an object
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
at System.String.FormatHelper (System.Text.StringBuilder result, IFormatProvider provider, System.String format, System.Object[] args) [0x00000] in :0
at System.String.Format (IFormatProvider provider, System.String format, System.Object[] args) [0x00000] in :0
at System.String.Format (System.String format, System.Object[] args) [0x00000] in :0
at EventStore.Common.Log.NLogger.ErrorException (System.Exception exc, System.String format, System.Object[] args) [0x00000] in :0
at EventStore.Common.Log.LazyLogger.ErrorException (System.Exception exc, System.String format, System.Object[] args) [0x00000] in :0
at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0
at System.Threading.Thread.StartInternal () [0x00000] in :0

Greg_Young1 · April 3, 2015, 12:40pm

From reading it seems lots of people with recent versions of docker
are experiencing segfaults in docker while not getting the same
segfaults in non containerized equivalent environments.

re back trace its basically just running gdb and typing backtrace on a
failure it says where things were at the time of the failure

also please give us:

docker version
uname -a
docker -D info

Greg_Young1 · April 3, 2015, 12:46pm

This looks like stack/heap corruption like due to same root cause as other issue

Greg_Young1 · April 3, 2015, 12:47pm

I'd be curious if building things in the container solves the issue as
opposed to using things built outside of a container

e.g. run with local mono and do a ./build.sh from source

Bob_Gregory1 · April 3, 2015, 5:52pm

I think Docker is a red herring.

I’ve just spent an hour and a half playing around with the tech-ops guy. He’s running Ubuntu 14.04 on a VM, no docker image, and gets an immediate segfault in libpthread when running WRFL 10 50000000.

I ran the docker image and got a segfault after 40M or so writes. I’m trying again on my local machine, which has a more recent kernel and a different version of glibc.

Bob_Gregory1 · April 3, 2015, 5:55pm

Nope, died after 9.8M reads. I’ll try an earlier version of event store and see what happens.

Greg_Young1 · April 3, 2015, 5:58pm

"on a vm"

which vm? Is a big question. We are dealing with something similar now
where the problem seems to only happen with ubuntu in azure (ubuntu in
AWS=fine, centos in Azure = fine).

Bob_Gregory1 · April 3, 2015, 6:16pm

I just reproduced on bare metal, arch linux, ES 3.0.3. Kernel is 3.18.6 x86_64. My glibc is Gnu Libc 2.21.

The VM was Ubuntu 14.04

– Bob

The VM was running Ubuntu 14.04.02

Greg_Young1 · April 3, 2015, 6:21pm

And if you build locally instead of using statically linked binaries
which is generally a bad idea anyways (just easier for people to do)

Bob_Gregory1 · April 3, 2015, 6:24pm

I’ll try that next, just want to test against ES 3.0 and see what happens.

Build instructions are on the

Greg_Young1 · April 3, 2015, 6:26pm

./build.sh

You need mono installed. You can either build
(/src/scripts/mono/get_mono.sh) or you can install from xamarin for
your release.

My guess here though is we are seeing slight differences in a
dependency which will go away if built locally

Bob_Gregory1 · April 3, 2015, 6:41pm

When running get-mono on a clean ubuntu 14.04 docker image, I receive:

Some packages could not be installed. This may mean that you have

requested an impossible situation or if you are using the unstable

distribution that some required packages have not yet been created

or been moved out of Incoming.

The following information may help to resolve the situation:

The following packages have unmet dependencies:

libtool : Depends: libc6-dev but it is not going to be installed or

                libc-dev

E: Unable to correct problems, you have held broken packages.

FAIL!

]0;root@1b3c32dda5e7: /tmp/eventstore/EventStore/src/Scriptsroot@1b3c32dda5e7:/tmp/eventstore/EventStore/src/Scripts# ^C

]0;root@1b3c32dda5e7: /tmp/eventstore/EventStore/src/Scriptsroot@1b3c32dda5e7:/tmp/eventstore/EventStore/src/Scripts# exit

If I try to explicitly install libc6-dev, I get

Some packages could not be installed. This may mean that you have

requested an impossible situation or if you are using the unstable

distribution that some required packages have not yet been created

or been moved out of Incoming.

The following information may help to resolve the situation:

The following packages have unmet dependencies:

libc6-dev : Depends: libc6 (= 2.19-0ubuntu6) but 2.19-0ubuntu6.6 is to be installed

         Recommends: gcc but it is not going to be installed or

                     c-compiler

Greg_Young1 · April 3, 2015, 6:52pm

Just use the xamarin ppa debugging dependency issues on your ubuntu
instance doesn't seem to be fruitful right now (we are way passed the
point of this having even the slightest thing to with eventstore or to
do with the problem).

Though @James the conflict is interesting on libc. Thinking back my
14.04 machine had problems as well we had chalked it up to 32 bit vs
64 bit but maybe something is umm problematic in 14.04 in particular

@Bob I run 14.10 here do you have 14.10?

Bob_Gregory1 · April 3, 2015, 7:47pm

I swapped the docker image to 14.10 and successfully wrote 50M events. I’ll do it again because I’m paranoid, and probably again and again after that, but it’s certainly a suggestive result.

I’m 99% sure I saw the same libpthread segfault issue occur on Arch though.

Greg_Young1 · April 3, 2015, 8:32pm

This sounds quite probable as running 14.10 i have never had issue but did in 14.04 (was only on for a few hours as i figured corrupted etc)

Ugo_Bourdon · April 4, 2015, 6:31pm

docker version
Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): a8a31ef
OS/Arch (client): linux/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): a8a31ef
uname -a
linux sd-58540 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
docker -D info
Containers: 8
Images: 18
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 34
Execution Driver: native-0.2
Kernel Version: 3.13.0-48-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 4
Total Memory: 15.64 GiB
Name: sd-58540
ID: 7SOH:EXDP:EBNJ:LDWJ:EWJ6:5J2L:R6SB:FFWR:YFQ7:7MOG:GPMF:I525
Debug mode (server): false
Debug mode (client): true
Fds: 12
Goroutines: 17
EventsListeners: 0
Init Path: /usr/bin/docker
Docker Root Dir: /var/lib/docker
WARNING: No swap limit support

My geteventstore crash regarly since i’ve installed it.