Testing and verifying a cluster setup

Do anyone know of any open source projects that can aid in testing and verifying the stability of a cluster?
We’re looking for something that can:

  • look for signs that the cluster has not been configured properly

  • put a cluster under stress/continuous load

  • create chaos by killing nodes in the cluster

  • simulate flaky network

It does not have to tick all the boxes…

Any help would be appreciated!

So we do this kind of testing,

as example can simulate a partition.

We generally use actual power pulls though (can cause slightly
different behaviour than just having a partition). As example

At Xero we’ve asked the exact same questions.

I’ve pulled together a document which describes our intended approach to testing and monitoring ES.

This doc is going through final review internally before we start implementing everything described in it.

We’ve attempted to write this document in a way which could be useful to people outside of Xero, and it’s based on an informal meeting with PieterG from GetEventStore and one round of internal feedback here.

It’s not quite complete and ready for publishing - but since you’ve asked the question, i’ll post what we have to date. :slight_smile: Hopefully it’s of some use - it doesn’t provide the bits you’re after but it does describe what is hopefully a fairly exhaustive Ops story. And there’s some useful background info from ES that provides answers to what I imagine are fairly common questions.

Once a final version is done (and hopefully reviewed by GetEventStore) then i’ll post an update to this forum…



EventStore testing

This document presents an approach to verifying ES cluster performance, and also which ES internal metrics and logs we should be monitoring to detect cluster and/or node issues.

The current testing approach is based on a single ES cluster in a single region - multi-region ES will be considered later.

This document is written for a specific environment - which happens to be linux hosts running in AWS, sumologic and either Datadog or Prometheus for logs/metrics.

Open questions

  1. what is a reasonable drift for writers to be behind the master?

  2. How do we use the Manager and watchdogs?

  3. Is there a way to monitor HTTP response codes from the cluster? Or is there anyway to directly correlate these with the EventStore logs? (e.g. we’re seeing occasional HTTP 500’s in our Prod cluster - is there a message we’d see in the logs that would match these events?)

General notes

Useful info that came out of general discussion/questions with ES and AWS.


  • slow messages - in general low value. defn of “slow” not configurable. value = 150-250ms, anything above that is slow.

  • ebs - consider which ssd type. has a definite impact on ES cluster perf and likelihood of getting “slow” warnings.

  • typical prod issue is deposed/partitioned master.

  • storage has a big impact on perf and is often the cause of prod issues.

  • if writecheckpoint never catching up to master checkpoint - could indicate network-related, storage.

  • 2+ masters indicates a network partition.

  • /gossip contains all nodes known to cluster, incl status, master/slave, whether nodes are behind/current checkpoint.

  • if nodes flip/flop, go to logs - source of info why things flipping.
    eg if deposed master separated from rest of cluster but doesn’t die for a while. comes up, gets told to die and truncate, restart.

  • storagereaderqueues - 4 by default. In charge of servicing read requests from clients.
    if these start backing up and not draining, maybe slow disk, network. also likely logged out as warnings.

  • writerqueue - single

  • node startup has multiple stages.

  • firstly is a node

  • joined cluster - catching up

  • once status is master, clone, slave then caught up and accepting writes.
    clone = a node not req to meet configured cluster config size (ie extra node)

  • cluster replication is over tcp, everything else incl cluster communications and election, etc all over http.

  • ntp - problems if exceeding 60s time drift btwn nodes

  • after discussion, agreed prefer to put resources into monitoring ES cluster health over canary events

  • when writes/second gets > 4K/second need to start seriously thinking about storage perf.

  • 15k-20k reads/second should be no problem for a well-configured cluster and appropriate storage (for eg t2-med, std ssd esb, 3 node cluster)

  • stats are published every 60 sec

  • (large) stats published every 60s will count towards backup size. can put maxage or maxcount on stats-ipaddress stream to reduce volume.

  • one ES customer builds an entire new cluster each morning - rebuild cluster, restore backups from ebs - have large downtime each morning to be able to do this.

  • restoring is essentially taking a EBS volume snapshot, attaching volume, restoring snapshot, joining node to cluster.

  • to alter user logins;
    PUT /users//command/change-password (basic auth with default creds, then change them). As per https://github.com/EventStore/EventStore/blob/08c2bdf7dcadd154cffa549d273e3a8e4673c5a1/src/EventStore.ClientAPI/UserManagement/UsersClient.cs#L90


  • Use attached storage SSDs rather than EBS

  • SSDs have much higher IOPS which is important in restore scenario

  • Restoring an EBS snapshot will lazy load disk blocks as they are requested

  • When restoring an ES node from a backup, it is necessary to have all the chunks available on disk before starting up the node. Therefore all disk blocks must be restored (whether from EBS snapshot or S3) first.

  • Preference therefore is for explicitly copying all disk blocks from S3 rather than background copy via ESB snapshot.

  • Use EBS for the root volume.
    So the OS, EventStore binaries, any other management agents, etc, would be baked into that EBS volume.

  • Use Instance Store volumes for the EventStore data itself.
    When you create a new instance you’ll always be populating EventStore data from another source (e.g. another EventStore node, or S3) as opposed to having pre-populated EventStore data in an EBS snapshot.

  • So:
    / - EBS volume. OS, EventStore binaries in /bin, etc
    /data - InstanceStore volume0. EventStore data
    /spare - InstanceStore volume1. Spare, swap, whatever

  • AMI creation would add Instance Stores to the AMI: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/add-instance-store-volumes.html#adding-instance-storage-ami

  • Will probably start with
    i3.large (0.475 TB SSD) + EBS ami root volume + attached SSD for the eventstore data
    Some larger instances have multiple instance stores, so the i2.8xlarge comes with 8*800GB instance store volumes.

  • Instance store volunes are attached at instance creation time. I think the 0.475 is only one volume, so you’d have only two volumes:
    / - An EBS volume holding OS
    /Data - the 0.475TB ssd instance store volume

Infrastructure specifications


Every ES node contains

  • log agent

  • metrics agent


ES logs

via log agent on every node

logs on linux are in /var/log/eventstore

  1. ensure log agent is configured to pick up the ES error log (/err)

  2. also ensure log agent is configured to pick up the std logs. (we may want to remove this later, depending on log sizes/verbosity)

  3. use log rotation / deletion to reduce file storage required after longs have been collected by the log agent

ES metrics

Two possible modes of capture;

  1. Prometheus - pulls from an ES http monitoring endpoint

  2. put an ELB in front of all nodes for monitoring

  3. prometheus connects to the ELB, will resolve to a single node

  4. If currently resolved node dies, ELB will resolve to another live node

  5. Cron job that runs on each ES node and initiates local job

  6. Job first queries local ES to determine if node is ES master

  7. If node is not ES master then exit

  8. else write result of ES monitoring query to directory monitored by log agent

  9. write metric showing cron job result;

  10. metric label = persistence.eventstore.cron.*

  11. ES node type

  12. elapsed time

  13. job result (success/failure)

every 1 second

  • GET /gossip

every 1 minute

  • GET /stats

  • GET /stats/tcp

  • GET /projections/all-non-transient

  • GET /users

  • GET /subscriptions

  • GET /streams/$scavenges?embed=tryharder (Accept=application/json)

Automated tests

Automated tests run once as part of ES cluster provisioning. They are intended to prove the cluster is performant.

Tests are to write json output and push results to a directory monitored by log agent.

  1. wrfl
    5 clients, 500K requests, 200K streams, 1K payload

  2. rdfl
    5 clients, 10M streams, 50M events, 1K payload

  3. verify
    as implemented by TestClient

Once the cluster is approved for production use then test data should be cleaned out;

  1. Stop ES cluster

  2. Delete contents of ES data dir

  3. Start ES cluster


Each backup should be written to metrics

  1. Backup started

  2. metric label = persistence.eventstore.backup.started

  3. node identifier

  4. Backup finished

  5. metric label = persistence.eventstore.backup.finished

  6. size

  7. elapsedtime (seconds)

  8. node identifier


Each restore should be written to metrics

  1. Restore started

  2. metric label = persistence.eventstore.restore.started

  3. node identifier

  4. Restore finished

  5. metric label = persistence.eventstore.restore.finished

  6. size

  7. elapsedtime (seconds)

  8. status (success/failure)

  9. node identifier

Publish all restore logs to directory monitored by log agent

Chaos Monkey

Each chaos check and hit (where chaos decides to terminate something) should be written to metrics

  1. Publish chaos checks to metrics

  2. metric label = persistence.eventstore.chaos.check

  3. Publish chaos hits to metrics

  4. metric label = persistence.eventstore.chaos.hit

  5. node identifier for node being terminated

  6. Publish chaos hits to log agent, including node identifier



  1. Verify single master node (via GET /info), choose master as source of backup

  2. Run S3 Sync every 30min to S3 bucket in same AWS account

  3. Ensure S3 bucket has cross-account replication to separate AWS backup account

  4. Execute via SSM, or (this may be broken atm) cron job running on each node as per ES metrics capture.


Three restore scenarios;

  1. Every day restore a new node at 5PM UTC

  2. once complete then terminate oldest node in cluster

  3. Whenever chaos terminates a node then restore a new node from S3

  4. Failure scenario - testing what happens when backup is bad

  5. restore node from bogus S3 backup

  6. ensure node is rejected by cluster

  7. determine exactly what logs/gossip should be looked for to indicate node restore/cluster join failure

Restore procedure

  1. Stand up new node instance, with ES service not running

  2. Restore node storage from latest S3 backup

  3. Overwrite truncate.chk with chaser.chk

  4. Start ES service to join node to cluster
    Node will be in catchup mode, catchup will verify node data matches

  5. Success indicated by node reaching slave status
    Failure indicated by cluster rejecting the node - will cause node to suicide and truncate data - this will be found in the logs


Following dashboards and queries to be built


  1. EBS / Instance Store (SSD) volumes

  2. CloudWatch metrics for bandwidth / throughput / queue length / latency / consumed ops / burst balance

  3. Individual nodes

  4. cpu load/perf

  5. GC

  6. storagereaderqueues

  7. writerqueue

  8. writes/second

  9. reads/second

  10. node cluster status

  11. Cluster

  12. connections

  13. node statuses

  14. node uptime

  15. node catchup point

  16. ntp differential

  17. backup size and result and elapsed time

  18. restore result and elapsed time

  19. Clusters
    (all environments)

  20. node statuses

  21. Backup

  22. SSM Association running backups to S3. Could monitor last write time to S3 and alert if it’s > backup interval?
    If not using SSM for backups then monitoring results of backup cron job


  1. ES error logs

  2. ES gossip

  3. ES stats

  4. Automated test results

  5. ES users/connections

  6. Backup failure logs


Alerts for following conditions to be setup;

  1. not exactly 1 master in cluster

  2. not at least 1 slave
    (for 5-node cluster actually want at least 3 slaves)

  3. actual cluster size < configured cluster size for longer than 1 hour

  4. configured cluster size != actual cluster size / asg size

  5. cluster health GET /gossip fails/stops producing monitoring data

  6. node status flip/flopping for any node

  7. storagereaderqueues backing up and not draining

  8. any node status is not one of master, clone, slave and node is > 30 min old

  9. ntp diff between any node > 30 sec

  10. writes/second > 4K

  11. reads/second > 20K

  12. restore.elapsedtime > 30 min

  13. backup.elapsedtime > 1 hour

  14. avail disk space < 30% total disk space

  15. no nodes are < 2 days old (would indicate nodes not being refreshed from daily backup)

Chaos Monkey

Purpose of chaos monkey is to randomly kill a node and verify that automated recovery/restore/rejoining the cluster/catchup/monitoring is working

  1. Update Chaos monkey to find and kill an ES node no more than once/day. Can be any node, ie do not exclude master.

  2. Do not execute chaos if actual cluster size < configured cluster size

  3. Do not execute chaos if any nodes not in healthy state

  4. Provide a semi-automated way to manually disable chaos invocations