Building highly available windows services with embedded event store

Just a load balancer would be ideal, so the client can be ignorant of the details.

Internal forwarding allows that (and can even put it on a different nerwork than public)

Just to be clear, I’m talking about my own service hooking into the master election mechanics of ES. The issue is that my own service also needs to implement some kind of forwarding if I’m also depending on the same round-robin DNS entry that ES if using for the cluster. But whatever kind of forwarding I do, I need to know who the master is to know who to forward to.

The other alternative I was thinking of was to create another cluster IP that my own service would Gratuitous ARP when it became master. But apparently doing a GARP from .NET is hard.

Why not just write and let es forward the put you service behind a load balancer? Is firwarding not working in embedded?

How do I go about that? I wasn’t aware that I could use it to forward non-ES traffic.

Just let es do it an load balance on top.

Lets say you run embedded 3 servers… just push to es it handles everything transparently to you

just push to es

Can you point me to a place to start looking for how to do that? Sorry for being dense. I’m digging around in the code, looking at things hanging off of ClusterVNode, and it’s just not coming to me.

You are hosting a ckuster embedded yes? Arw writes on a slave forwarding to the master? If not lets make that work like http or tcp now.

I’m doing research at the moment. I have some single instances of event store here and there for dev, but have not deployed a cluster yet. I’m trying to work how to integrate before we decide how we will deploy.

Okay, I found this class, which appears to do the forwarding and also listens for cluster state changes to know who the master is to forward to.

https://github.com/EventStore/EventStore/blob/master/src/EventStore.Core/Services/HttpSendService.cs#L47

https://github.com/EventStore/EventStore/blob/master/src/EventStore.Core/Services/HttpSendService.cs#L134

But it is not exposed publicly for me to use off ClusterVNode.

https://github.com/EventStore/EventStore/blob/master/src/EventStore.Core/ClusterVNode.cs#L267

(Still not quite smelling what you are stepping in, so thinking out loud below.)

So I assume I new up my own copy of it to use (along with the ES internal way of representing HTTP requests/entities). But from your comments, it seems like it should be easier than that. That leads me to think I should just be POSTing my commands to my URLs on the ES HTTP port and let them auto-forward to the master. But then I am unsure how I am going to handle those URLs on the other side… how to get Nancy to listen to those URLs on a port that is already in use by ES’s HTTP server. So then I think maybe you want me to use ES’s version of HTTP controllers, but I’m unsure how to hook them up. Then I am back to the “it’s less simple than your comments seem to indicate” conclusion. Hence my confusion.

If you do a write over clientapi as well it will forward.

Don’t think how to use ES’s controllers. Just write to ES (say using client api in your controller?) and it will handle all the forwarding for you

When you say “Just write to ES”, I’m hearing: (on a non-master node) go ahead and process the command, save the events (write to ES) (which will get forwarded to the master), and let optimistic concurrency failures deal with concurrency. But depending on the workflow, 2 nodes being able to process commands on the same aggregate could have undesirable consequences.

So if that’s not what you mean, can you be more explicit on what “just write to ES” means? I’m recovering from being sick, so I’m probably just being really thick about it.

"When you say "Just write to ES", I'm hearing: (on a non-master node)
go ahead and process the command, save the events (write to ES) (which
will get forwarded to the master), and let optimistic concurrency
failures deal with concurrency. But depending on the workflow, 2 nodes
being able to process commands on the same aggregate could have
undesirable consequences."

They will get linearized on the master. This is also why expected
version exists!

So side effects of command processing like charging a credit card… should these be initiated by a Process Manager or something similar to ensure they only happen once?

"So side effects of command processing like charging a credit card...
should these be initiated by a Process Manager or something similar to
ensure they only happen once?"

They can be.. However its highly unlikely on an operation such as
charging a credit card that you can reach only once semantics

/s/only/exactly

Hmm, I’ll have to think about that. I was originally going for single node processing a given aggregate at a time. My opcon was detected on load rather than on save so I could avoid a class of duplicated side effects (due to retry).

opcon was also on save, I should say.