Manually electing a cluster master

EventStoreDB
#1

Hi,

I’m currently in the middle of trying to set up a cluster of EventStore database nodes across multiple datacentres for high availability. I’ve got most of it working quite happily, but we have a preferred datacentre for our all of our cluster master roles for data tier services (we use SQL and MongoDB here as well) and I can’t seem to find a way to manually force a particular database node to be the EventStore cluster master (leader?) so that the role lives in our primary datacentre.

Is it possible to force a particular database node to be the master, and if so, how do I go about doing it - e.g. is there an api call or a command line option?

Cheers,

Mike

#2

You can't *force* a master role onto a given node, doing so would
possibly lose data. What you can do is give a preference. If two nodes
are equally able to be a master the preference will be used.
--node-priority (higher values are picked)

#3

Hi Greg,

Thanks for the reply - that looks like the parameter I was searching for!

As a follow up, I’m thinking about how to test a DR scenario where our primary datacentre fails - is there a way to trigger an internal re-election? E.g. if I change the priority on a “secondary” node to be higher than the current master (and restart it), does the gossip protocol detect that there’s a node with a higher priority than the current master and automatically run an election? Or can I send a “re-elect” message of some sort, or should I simply stop the current master at the primary datacentre and let nature take its course?

Thanks again,

Mike

#4

Failovers are automatic. You can always force elections (just by
killing nodes). The failover however will always choose the best node
unless both are equal in which case the priority applies

#5

Ah, ok cool.

One last thing - when you say “The failover however will always choose the best node unless both are equal in which case the priority applies”, what determines the “best node” and whether they are equal? I got the “–node-priority parameter”, but I’m not sure if you mean there’s another metric that applies to determine the “best node unless both are equal” before the node-priority gets inspected?

Cheers,

M

#6

How much data they have.

Consider
Node A 557
Node B 580 master
Node C 580

So we kill node B

We have an election between A/C it doesn't matter what their
priorities are as if we select Node A as master we will lose data.

Cheers,

Greg

#7

Got you. That makes sense.

I guess I’m going to have to have a bit of a think about how I test datacentre failover, but I feel like I understand the process a bit better now.

Thanks again,

Mike

#8

Why don’t A and C “talk” after the election and make sure they are both up-to-date so election could always follow priority?

#9

If A is updating from C, so that it can become a viable candidate for master, then it’s too late - that decision has already been made!

What if a new Node joins the cluster before A is ready? The master has to be C, whether or not it is preferred.