Is a clustered, production, OSS EventStore realistic?

EventStoreDB
1

Hi,

We are really trying to use EventStore on a new application. However, clustering ES is proving to beat us down. Here’s what I’d love to do: Create an AWS EC2 Container Service cluster with my three nodes that is auto-scaled by ECS. However, I can’t do this b/c ES needs to know the IP addresses of the other seeds before it’ll let any seed start. This makes me sad. It’d be nice if it acted something like Consul, and let me say “Don’t try to do anything until all the nodes check in”. This is complicated by Docker’s rudimentary networking, but, if I could just link containers, this would be very, very nice.

I have tried to use discover-via-dns, but I presume that means I have to setup my own, local DNS server (I couldn’t, for example, put in Google’s DNS server) which seems to be a a lot of work to do just to get up a cluster. I get why someone might do it, but it feels like overkill for us.

As such, my only real option (on the open source side) seems to be to provision 3 separate EC2 instances, get their IPs, then run the clusternode command on each one. This means that ES would be outside our Docker based infrastructure, which makes me sad. This also introduces issues if one of these goes down and the IP changes for the replacement…

You should know that all of this is compounded by my lack of DevOps knowledge. However, I have been able to deploy almost every other piece of software in our stack, outside of ES, on ECS. So, while I am sure a more experienced DevOps person could get farther than me, I don’t think it’s the only issue. Although, I’d be really happy if the issue is I am stupid.

Adding to my suspicions that an OSS-based ES cluster is not a viable production option is the utter lack of tutorials, posts in this group, etc. around it. There’s a bit, but not much. My guess is most people are paying for support and the better clustering tools.

So, is anyone running an ES cluster in production (preferably on AWS) that is willing to share knowledge? My client is going to balk at $2000/year.

Any help is appreciated.

Thanks,
Glenn

2

Whilst many people are indeed paying (and to be honest 2000 a year is pocket change for any project paying for developers - the AWS bill alone will be 2-300/month for a cluster that performs well) there are orders of magnitudes more people using the OSS
builds in production. One thing of note is that the commercial editions now include AMIs preconfigured as event store nodes.

Let me correct some fundamentals you have wrong though:

  • you don’t need to know nodes’ addresses up front, you can use DNS. There is a specific section about this in the OSS setup guide in the documentation.

  • you don’t want to auto scale. It’s a quorum system. Keep 3 or 5 nodes running all the time.

3

"
As such, my only real option (on the open source side) seems to be to
provision 3 separate EC2 instances, get their IPs, then run the
clusternode command on each one. This means that ES would be outside
our Docker based infrastructure, which makes me sad. This also
introduces issues if one of these goes down and the IP changes for the
replacement...

You should know that all of this is compounded by my lack of DevOps
knowledge. However, I have been able to deploy almost every other
piece of software in our stack, outside of ES, on ECS. So, while I am
sure a more experienced DevOps person could get farther than me, I
don't think it's the only issue. Although, I'd be really happy if the
issue is I am stupid."

Why would this be the only way of running ES? And if you could do this
on an ams instance why couldn't you do it in a docker instance?

4

Also look into route 53. If you're in AWS you should be using it anyway.

5

Thanks for the reply, gents.

James, you’re right. $2000 is not much, annually. I am exploring that possibility with the client (ugh, that sounds douchey)

Greg, the issue I have running a cluster on docker is having to know the IPs ahead of time (not going the DNS route, obvs) There’s no clean/easy way to do this with Docker.

If we go the DNS route (and we will), do all the nodes have to be on “well-known” ports, meaning 2113 and 1113? If so, this kills running it on ECS b/c I may only have 2 machines in the ECS cluster running 3 containers. Again, maybe that’s stupid, but it’s an example of why I am struggling getting this deployed via Docker and ECS (from both a knowledge and tooling issues)

Again, thanks for your help and patience.

6

"If we go the DNS route (and we will), do all the nodes have to be on
"well-known" ports, meaning 2113 and 1113? If so, this kills running
it on ECS b/c I may only have 2 machines in the ECS cluster running 3
containers. Again, maybe that's stupid, but it's an example of why I
am struggling getting this deployed via Docker and ECS (from both a
knowledge and tooling issues)"

What is the benefit of running two nodes on one physical node? I would
imagine the faults between the two nodes would be highly correlated.

7

The benefit is I can get a ECS cluster of ES nodes up and play with it without having to create 3 EC2 instances. You’re right, of course, that having > ES node on a physical node is questionable, at best, but if I want to put up a staging ES cluster and not worry about that, I can’t. Maybe putting 5 ES nodes across 3 EC2 instances with ECS would be somewhat less crazy. I dunno, maybe I am just being dense.

I guess the only way to do this with Docker and ECS would be to create 3 (or however many you want) ECS clusters that each run a single ES cluster node and get Route 53 DNS to have them discover each other.

8

You can run on any ports you want (see documentation here that runs 3
nodes on a single machine)

http://docs.geteventstore.com/server/3.1.0-pre/cluster-without-manager-nodes/

9

The benefit is I can get a ECS cluster of ES nodes up and play with it without having
to create 3 EC2 instances. You’re right, of course, that having > ES node on a physical

node is questionable, at best, but if I want to put up a staging ES cluster and not worry

about that, I can’t. Maybe putting 5 ES nodes across 3 EC2 instances with ECS would

be somewhat less crazy. I dunno, maybe I am just being dense.

If you have three instances run three nodes and all will be well. Running more than one on one box actively harms throughput (EBS is pretty slow all things considered) and will cost you a ton of money for precisely zero benefit. If you run two on one instance you may as well run a cluster of two machines and just accept downtime when one isn’t available.

10

Glenn,

I went through a very similar situation earlier this year. I had to deploy a cluster on OSS nodes on AWS Opsworks for our staging and production environments. Being a total beginner with Chef and AWS, I had quite a hard time getting everything to work. Hopefully, with help from the community I finally got the whole thing up and running. Here’s what I’ve learned :

  • both --int-ip and --ext-ip have to be set to the Private-IP of the EC2 instance. On Opsworks, you can programmatically get this value via the node[:opworks][:instance][:private_ip] attribute.
  • set --discover-via-dns to false and set --gossip-seed to a comma separated list of the addresses (ip:port) of the (at least one) other nodes in your cluster. Again, on Opsworks you can build this list by iterating over the value of node[:opsworks][:layers][XXX][:instances] (replace XXX with the shortname of the layer you’re running your ES cluster on). Make sure you use the internal http port of your ES nodes (which is 2112 by default).
  • on “little” instances (t2.micro, t2.small, etc) you’ll have to set the MONO_THREADS_PER_CPU environment variable to a “high” value (100 is perfectly fine).
  • to be able to access the web interface, since --ext-ip is set to an IP that is not accessible from outside AWS, you may set --http-prefixes=‘http://localhost:2113/’ and browse it through a ssh tunnel ( ssh -L 2113:localhost:2113 your_username@ec2_instance_ip ). But there may be a more convenient way that I’m not aware of.
  • do not forget to configure your security groups so that your instances can talk to each other en internal HTTP (2112) and TCP (1112) ports.
    That’s pretty much it. I run the nodes using daemonize, but writing an init.d script should not be so much trouble. And of course, you’ll also need to take care of monitoring and alerting yourself, which is another story.

Hope this helps.

Valentin

11

"

to be able to access the web interface, since --ext-ip is set to an IP
that is not accessible from outside AWS, you may set
--http-prefixes=‘http://localhost:2113/’ and browse it through a ssh
tunnel ( ssh -L 2113:localhost:2113 your_username@ec2_instance_ip ).
But there may be a more convenient way that I’m not aware of."

Set your elastic ip as a http-prefix.

12

The only way I’ve been able to get OSS to cluster is with a setup like described here. This is very helpful to have written down.

The only change I would suggest is just making the http-prefix *:2113. You already should have a couple layers of network security, and the http-prefix setting on ES just seems to cause more problems than it solves.

Oh, I also had to raise my gossip timeout a lot when running on medium instances - to something like 1200ms.

Justin

13

To summarise:

  • Don’t expose Event Store to the internet without a proxy server in front of it, ever. This is just plain a bad idea, even for development.

  • We rely on either having two cores minimum, or you’ll need to set MONO_MAX_THREADS_PER_CPU to avoid deadlocking

  • The defaults are set up well to work on real hardware on a real network (specifically, our appliances)

  • Network performance on medium instances means you’ll need to turn up the timeouts

  • HTTP permissions on Windows and .NET are a shit show, at some point we intend to switch away from it in favour of using something like libev directly and this will stop being a problem.

  • You need to understand your deployment environment and what it means to bind to an interface.

  • If you buy commercial support this is all take care of as you get images for AWS…

14

Hi,

I’m very new to the project and I’m looking at deploying EventStore in production (AWS) so naturally interested in what options are available to recover from potential failures. Apologies if I get any concepts wrong as I’m new to EventStore.

Some ideas:

  • Run a single node (ASG of min=1 max=1) with an ELB in front of it. The ELB is simply so that if the instance dies and a new one comes up, the ELB address that clients point to doesn’t change. For backups, I would periodically copy *.chk files to S3. If the instance dies, a new one will spin up and attempt to fetch *.chk files from S3 and restore. Of course the biggest pitfall is that if the node dies, you have to wait for a new one to spin up, restore and back into service.

  • Run an ASG of min=3 max=3 (again with an ELB in front). Instances could add themselves on boot to a DNS record that points to all the nodes. Would need to do some magic to ensure that if a node dies, it is removed from the record. Slight concern is that since they would all be launched at the same time, how would they determine if they are the first node in the cluster or if they need to connect to an existing node?

James, you mentioned that with commercial support, images are provided. I assume that solves the trouble of getting an instance together with all the dependencies to run the software, but it doesn’t solve the clustering bit right?

I appreciate any feedback on the above.

Thanks!
Gonzalo

15

how would they determine if they are the first node in the cluster or
if they need to connect to an existing node?

They need a majority of nodes to agree on that. So with a cluster of 3
it would be when the second node came up. The two nodes will then talk
to each other to figure out if they are both new and should start a
new cluster or if one of them has data in which case that node will
become the master and replicate its data to the other node.

Cheers,

Greg