How is an increase of cluster size handled?

EventStoreDB
1

We have some internal discussion about different databases and a question regarding how the clustering of event store works. Let’s put together a simple example. You start with three nodes: A, B, C. So far so good. But all those are created with a cluster size of 3, how do you go around creating a cluster of size 5 from this size 3 cluster? Do you have to take down A, B, C and change so they know they are in a five node cluster and then start them again and then add node D and E? Can you add D and E first and then increase the size of A, B, C? Is there a way to do this without downtime?

2

This is technically handled with view change messages (if you bring up
two extra nodes they will come up as clones). I have on a branch the
code to transition to a 5 node cluster in this case.

3

With “come up as clones” you mean the cluster will still have size 3 with the original nodes and the cluster will fail if two of the nodes A,B,C fails? Is change messages documented anywhere?

4

viewchange messages...

http://www.cs.cornell.edu/courses/cs6452/2012sp/papers/paxos-complex.pdf

The branch basically will realize this case and make it a 5 node
cluster at that point

5

Ok, I thought you meant that it was supported today with view change messages. I’ll be a sweet feature when implemented. But is there a recommendation today regarding increasing cluster size?

6

Its on a branch, needs testing. Haven't had anyone been really keen on
pushing it forward.

Today it depends in terms of complexity vs amount of possible
downtime. The easiest answer is just shutdown/restart cluster. You can
however get this time lower by bringing up say 2 clones first (at 5)
then shutdown 3 and bring up one at 5 (this can be < 5 seconds). This
is likely the reason noone has asked. Then bring up other nodes.

Cheers,

Greg

7

So the recipe is:

  • You have A, B and C

  • Bring up D and E with cluster size 5 (but connect to existing cluster? how is the existing cluster affected when there are two sizes specified?)

  • Take down A, B and C

  • Bring up A, B and C with size 5 one by one

  • Now A, B, C, D and E is a cluster of 5

8

Easy enough to test :slight_smile:

9

Of course, I was just thinking it was a recommended to how this should be handled.