Hi Guys, we are Hosting Eventstore with 3 Cluster Nodes. Is there a way to have a graceful shutdown so that the clients will not get a timeout when the leader role is switching to another node?
It’s less about how the node shuts down but more about who will be the next leader. The leader node shutdown or restart triggers the elections, so there will be some node elected as the new leader. Most of the time it will be the node with no replication lag, so theoretically it should be possible to know where the clients will need to switch, but in reality it works like this:
- The leader is considered dead
- Elections occur
- The new leader is elected
- A client tries to write to the old leader
- There’s no response, the client fails to write
- The client retries the operation (it should be done by the application code I believe)
- The client gossips with the cluster and finds the new leader
- Write goes through
Subscriptions also drop when the node is no longer available, and when resubscribing, the correct node will be selected based on the new cluster topology.
Overall, it’s all about retries, and I don’t think it would be possible to avoid it. Even if we say that the node is being shut down gracefully and sends a signal to its clients that they should stop writing to it, the clients won’t know where to write until elections happen and the new topology is available via gossip endpoint. One way or another, there will always be a delay between the writes must stop, and then resume.
1 Like