I don’t have anything written up, but happy to share my experiences.
We are now running EventStore under Kubernetes on Google Cloud, AWS should be very similar. We have a 3 node cluster for our production environment (running in an isolated Kubernetes cluster), and an identical EventStore cluster for our “non production” Kubernetes cluster that hosts dev and test environments.
Each node has its own Kubernetes deployment, each with 1 replicas, I.e. we have three deployment files, identical except for names in each, to form the 3 node cluster. This was done before Kubernetes 1.3, PetSet looks like a nicer way to do this now.
The image used is based on https://github.com/EventStore/EventStore docker, with some tweaks, for example to enable system projections and start projections automatically.
We use Google persistent disks, ssd based, and take snapshots nightly. Persistent disks allow us to completely recreate the Kubernetes cluster, e.g. for Kubernetes upgrades, without having to worry about data loss. Killing nodes and Kubernetes pods “just works”, and the EventStore cluster recovers as expected.
EventStore is configured to use dns for initial peer discovery, and a Kubernetes service with clusterIP: none is created for this purpose (allowing EventStore to find the IPs of the individual pods, rather than a load balancing service proxy).
A separate Kubernetes service is set up that allows clients to connect to the cluster, this one with load balancing, to route clients to a random healthy node. We’ve also recently enabled cluster discovery in our clients (backend services), but may disable this for services that we know are read only (with regards to EventStore).
Overall things work well! One lesson learned is, persistent disk is very slow compared to local disk. This doesn’t cause issues for us in steady state as transaction rates are still low, but does mean that recovery when a node restarts is very slow, as checksums are checked and indexes rebuilt.
So far this all runs in a single availability zone, on the road map is setting up a kubernetes cluster spanning zones, and spread the EvrngStore nodes across zones, though I’m not sure how badly this will affect latency.