Wait for EventStore to be ready?

EventStoreDB
#1

Hi guys,

We want to upgrade our production servers to the latest release, but
need to know how to wait for an instance to be actually ready.
Specifically, when we start the EventStore service that returns
immediately, but it seems like we then really should wait for
"ReadIndex building..." to be completed before upgrading the next
instance.

Is there a way to know when an EventStore is really really ready for
use? Are the ports only available once it is really ready for example?

thanks, Rickard

#2

Hi Rickard

For updating a cluster, bring down a node (preferably starting with non-master nodes), update it, and bring it back up.

Wait for it to rejoin the cluster and catch up.

Once it’s joined the cluster, it’s safe for you to bring down the next node and repeat the above process

#3

Hey

#4

You can monitor the state of the cluster by polling the gossip url of one of the nodes, e.g. http://nodeip:2113/gossip

Once the node you have restarted has the status of “Slave” or “Master” it means it has successfully joined the cluster.

#5

Thanks! Unfortunately the default wait_for Ansible module would not be
able to handle that as far as I can tell:
http://docs.ansible.com/ansible/wait_for_module.html

The regex search for log file might work, if I can generate the
current log file name correctly.

Any other ideas on how to do this would be appreciated!

Anyone else doing automated upgrades?

regards, Rickard

#6

Hi Rickard,

You can wait for arbitrary conditions by using until.

http://stackoverflow.com/questions/23919744/how-to-restart-jenkins-using-ansible-and-wait-for-it-to-come-back

If you’re interested in contributing some code, I’ve got the beginnings of a set of ansible modules here: https://github.com/madedotcom/ouroboros/

It would be neat if you could say

  • name: Wait for Eventstore node to be healthy

eventstore_status:

host: 127.0.0.1

register: node

delay: 5

retries: 60

until: “node.status in [‘slave’, ‘master’, ‘clone’]”

– B

#7

Hey Bob!

Fantastic! That's much more sensible, should work well. I'll try it out!

cheers, Rickard