Testing eventstore cluster

EventStoreDB
#1

Hi

I am testing eventstore in a 3 node cluster on 3 ubuntu 14.04 virtualbox nodes on my laptop.

I am having some problems, two nodes seem to work fine, but when i connect the third node, they start disconnecting from eachother. See attached log.

I tried setting gossip timeout to 3000 and intervall to 2000.

There is also the issue that both nodes when idle use 40% cpu on my Intel® Core™ i7-2600K CPU @ 3.40GHz

Is anyone else having these problems?

192.168.33.11-2114-cluster-node-err.log (60 KB)

192.168.33.11-2114-cluster-node.log (447 KB)

192.168.33.11-2114-cluster-node-stats.csv (4.17 MB)

#2

So these nodes are having elections and cycling more than one time /
second. Also I assume thats 40% of one core. BTW measuring CPU usage
of anything when running in virtual box is "interesting"

My guess is its something in the config. Can you try this?

export MONO_THREADS_PER_CPU=100

on the nodes?

#3

So these nodes are having elections and cycling more than one time /
second. Also I assume thats 40% of one core. BTW measuring CPU usage
of anything when running in virtual box is "interesting"

My guess is its something in the config quite likely

#4

hmm this was in error...

#5

I changed a lot of timeouts and i think i nailed it now: ./run-node.sh --db /tmp/db --log /tmp/log --int-ip 192.168.33.13 --ext-ip 192.168.33.13 --int-tcp-port=1111 --ext-tcp-port=1112 --int-http-port=2113 --ext-http-port=2114 --cluster-dns escluster.mooo.com --cluster-gossip-port 2113 --run-projections All --cluster-size 3 --gossip-timeout-ms 3000 --gossip-interval-ms 2000 --int-tcp-heartbeat-timeout 3000 --ext-tcp-heartbeat-timeout 3000 --prepare-timeout-ms 3000 --commit-timeout-ms 3000 --projection-threads 12 --worker-threads 10 --http-prefixes http://*:2114/ --gossip-allowed-difference-ms 1200000

especially the allowed difference in ms, was key.

#6

I didn't see any errors about allowed difference... This is talking
about clock drift between the nodes.

#7

never mind, i was lucky. They can sometimes find out who is who and settle.

#8

btw why are you running 12 projection threads and 10 worker threads?

#9

Just increasing to see if that helped. It didn’t. Do you know whats making the nodes reelect all the time, before they can decide?

#10

They are deciding and then there is a communications problem. Why the
communications problem is happening can't be easily determined by
looking at just one nodes log.

In .11 I see

[PID:10368:008 2015.07.04 13:26:01.877 TRACE TcpConnectionManager]
Closing connection 'master-normal' [192.168.33.12:1111,
L192.168.33.11:53281, {ef70187d-0f25-49f0-b731-20a1ef8387f6}] cleanly.
Reason: HEARTBEAT TIMEOUT at msgNum 11
[PID:10368:008 2015.07.04 13:26:01.877 INFO TcpConnection ] ES
TcpConnection closed [13:26:01.877: N192.168.33.12:1111,
L192.168.33.11:53281, {ef70187d-0f25-49f0-b731-20a1ef8387f6}]:
Received bytes: 18982, Sent bytes: 1973
Send calls: 10, callbacks: 10
Receive calls: 12, callbacks: 11
Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 11

[PID:10368:008 2015.07.04 13:26:01.877 INFO TcpConnectionManager]
Connection 'master-normal' [192.168.33.12:1111,
{ef70187d-0f25-49f0-b731-20a1ef8387f6}] closed: Success.

But .12 was master so it depends what happened there.

#11

Another example:

192.168.33.13-2114-cluster-node.log (43.8 KB)

192.168.33.13-2114-cluster-node-err.log (185 Bytes)

192.168.33.12-2114-cluster-node.log (46.8 KB)

192.168.33.13-2114-cluster-node-stats.csv (17 KB)

192.168.33.12-2114-cluster-node-stats.csv (17 KB)

192.168.33.11-2114-cluster-node.log (71.1 KB)

192.168.33.11-2114-cluster-node-stats.csv (16.5 KB)

#12

You are getting failures on sending gossip.

(L=40054209,W=40055076,C=40055076,E135@40048594:{4cc1fd86-9b4d-4671-b0fa-574a5453c7c8}).
[PID:12916:010 2015.07.04 15:02:28.272 TRACE GossipServiceBase ]
Looks like node [192.168.33.11:2113] is DEAD (Gossip send failed).
[PID:12916:010 2015.07.04 15:02:28.272 TRACE GossipServiceBase ]
CLUSTER HAS CHANGED (gossip send failed to [192.168.33.11:2113])

The node then comes back later then another node disappears then comes
back then disappears then comes back.

[PID:11553:010 2015.07.04 15:02:27.671 DEBUG ElectionsService ]
ELECTIONS: (V=3) VIEWCHANGE FROM [192.168.33.13:2113,
{ce8cb83c-2b53-4706-84b1-bb795850ec9b}].
[PID:11553:010 2015.07.04 15:02:27.678 TRACE GossipServiceBase ]
Looks like node [192.168.33.12:2113] is DEAD (Gossip send failed).

You also lost TCP connections and are getting tons of errors over
http. Basically the nodes aren't really talking to each other well.

Did you export the threads setting mentioned above for mono? If
running on small vms that needs to be set.

Greg

#13

Thanks, that fixed it :slight_smile: I am starting to dislike mono a lot.

It settles after 3-4 seconds now.

What about setting that environment variable in run-node.sh in next version :slight_smile:

I am still getting 20 % cpu usage inside each vm when idle, and 40 % cpu outside the vm. Is this normal?

#14

We shouldn't put that in run-node but maybe a comment. It depends on
the machine you are running on whether or not it needs to be raised
(TBF 100 is a lot btw you can use a lower number).

Inside yes @ 20% though there are settings that can lower this.

Especially --run-projections=none should lower this.

You can also lower thread counts of various things. Remember 20% = 1
core @ 20% on most linux systems. Most of the load is normally context
switching

#15

Yep, i am just glad i got it working fine on Ubuntu. Hoping to get it working better on SLES 12 again on monday, when i am back at work.

Seems strange to have issues with glibc on SLES 12 and not on ubuntu 14.04 which have the same version 2.19.

#16

It will have issues on 14.04 on current binaries my guess is you are
running single core vms? This could make it less likely to happen (as
a guess)

#17

I am using 2 core per vm.

What os do i need to run for current binaries to be stable?

Are there plans for making binaries that will run stable on one of the enterprise distros? We have two options get it to run on SLES 12 or maby buy 3 appliances.