We have some internal discussion about different databases and a question regarding how the clustering of event store works. Let’s put together a simple example. You start with three nodes: A, B, C. So far so good. But all those are created with a cluster size of 3, how do you go around creating a cluster of size 5 from this size 3 cluster? Do you have to take down A, B, C and change so they know they are in a five node cluster and then start them again and then add node D and E? Can you add D and E first and then increase the size of A, B, C? Is there a way to do this without downtime?
This is technically handled with view change messages (if you bring up
two extra nodes they will come up as clones). I have on a branch the
code to transition to a 5 node cluster in this case.
With “come up as clones” you mean the cluster will still have size 3 with the original nodes and the cluster will fail if two of the nodes A,B,C fails? Is change messages documented anywhere?
The branch basically will realize this case and make it a 5 node
cluster at that point
Ok, I thought you meant that it was supported today with view change messages. I’ll be a sweet feature when implemented. But is there a recommendation today regarding increasing cluster size?
Its on a branch, needs testing. Haven't had anyone been really keen on
pushing it forward.
Today it depends in terms of complexity vs amount of possible
downtime. The easiest answer is just shutdown/restart cluster. You can
however get this time lower by bringing up say 2 clones first (at 5)
then shutdown 3 and bring up one at 5 (this can be < 5 seconds). This
is likely the reason noone has asked. Then bring up other nodes.
So the recipe is:
You have A, B and C
Bring up D and E with cluster size 5 (but connect to existing cluster? how is the existing cluster affected when there are two sizes specified?)
Take down A, B and C
Bring up A, B and C with size 5 one by one
Now A, B, C, D and E is a cluster of 5
Easy enough to test
Of course, I was just thinking it was a recommended to how this should be handled.