Running with two datacenters

Yes, I understand. (1) is based on there being manual intervention in the case where datacenter with 2 nodes goes down. In that case, the system would not accept writes and we either:

  • Bring the dead nodes back online, or, if that is not possible or would be too long:
  • Accept data loss and our new source of truth would be the single remaining node. Would have to shut down the 2 dead nodes completely and discard their data to ensure they don’t come back, then rebuild cluster.
    This option is still preferable in my mind, because we want to handle individual machine failures automatically.

Related question: If a node dies completely, how do you bring a fresh node into the cluster? Can’t see anything in the docs about this.

"Related question: If a node dies completely, how do you bring a fresh
node into the cluster? Can't see anything in the docs about this."

You would have to kill the original node completely (never bring it
back up) and bring up a new node.

This sounds like a nasty DR strategy when you consider it can be made
risk/"interesting" manual stuff free with the addition of a 3rd node
in a 3rd dc.

For me at this point I would just go with the two nodes in two dcs and
push the events between them (probably less work overall!)