For this to work, I’ve created an A record for cluster.somedomain.com with the IPs of the nodes manually. I need some advise of how I’d be able to update the A record in Route53 when one instance terminates & another takes its place.
I’ve tried using --gossip-seed option with DNS name instead of IP, but it doesnt seem to like it as its expecting IP address.
I am using cluster-dns option, the problem with that is I need to manually update the Route53 A record with IP of the nodes. I am looking for a self healing solution.
Is there an option in config to use the DNS name of the individual nodes instead of the IPs … example
you want to use the cluster-dns option and then have the node register
itself when it starts up. Note that these dns entries are only used
for iniitial discovery of the cluster, clients and the nodes
themselves use a gossip protocol to maintain actual cluster layout
To give you a background of what I’ve achieved & what I’m trying -
I have 3 nodes running 3 AZs in AWS, each in their own ASG. This way when a node goes down, its replacment node in the same AZ comes up with the same attached EBS & and same hostname/DNS. This way each one of them always have the same state apart from the IP that changes when a node terminates & another replaces it.
So, when an instance goes down, how would I be able to add the new node(with new IP) back into the cluster automatically ?
DNS is meant to be used to contain *all* the nodes, this is how
discovery works. You can also use ips as gossip seeds. There is no
option to use dns-gossip-seeds nor can I understand why someone would
want to as this situation is already covered via dns based discovery.
when an instance goes down, how would I be able to add the new node which replaces the terminated node (with its new IP) back into the cluster automatically
This is not a good idea. You want three instances in one autoscaling group and to deal with replacement via introspection of the ASG API in a wrapping startup script vs using DNS.
Thanks Greg, I was going down the route of Lambda to update the A record in Route53 until you mentioned -
'Note that these dns entries are only used
for iniitial discovery of the cluster, clients and the nodes
themselves use a gossip protocol to maintain actual cluster layout
’
earlier.
pardon me for my misunderstanding if there was, but i understood that the DNS is used to get the IPs of the nodes initially and then afterwards all gossip happens using the IP addresses, so if thats true, even if the new node comes up & a lambda updates the Route53 entry, the other 2 nodes would still try to reach the terminated node using the old (terminated instance’s) IP, unless the service is restarted so that all nodes pick up the new IPs from Route53.
Am I correct in this ?
@James, I’m extremely sorry I didn’t get your point.