Clustered EventStore Configuration with Separate Int/Ext Networks

EventStoreDB
#1

I am attempting to set up my cluster following the recommendation of https://eventstore.org/docs/server/4.0.0/ports-and-networking/ and the cluster DB nodes documentation. I have several clusters with the following config (two networks to separate internal/external traffic) - the below is the config of one of the nodes:

[09791,01,00:18:23.272]
ES VERSION: 4.0.0.7 (HEAD/aa8cda89f70ddef785d4e673bf9a336bf88da59f, Thu, 23 Mar 2017 12:08:13 +0200)
[09791,01,00:18:23.287] OS: Linux (Unix 4.11.0.1011)
[09791,01,00:18:23.292] RUNTIME: 4.6.2 (Stable 4.6.2.16/ac9e222) (64-bit)
[09791,01,00:18:23.292] GC: 2 GENERATIONS
[09791,01,00:18:23.292] LOGS: /data/eslogs
[09791,01,00:18:23.305] MODIFIED OPTIONS:
INT IP: 10.0.3.5 (Config File)
EXT IP: 0.0.0.0 (Config File)
INT HTTP PREFIXES: http://10.0.3.5:2112/ (Config File)
EXT HTTP PREFIXES: http://*:2113/ (Config File)
ADD INTERFACE PREFIXES: false (Config File)
RUN PROJECTIONS: All (Config File)
DB: /data/esdb (Config File)
INDEX: /data/esindex (Config File)
LOG: /data/eslogs (Config File)
CLUSTER SIZE: 2 (Config File)
GOSSIP SEED: 10.0.3.7:2112 (Config File)
DISCOVER VIA DNS: False (Config File)

The networks are:

  • 10.0.2.0/28 - external – IPs are 10.0.2.5, 10.0.2.6
  • 10.0.3.0/28 - internal – IPs are 10.0.3,6, 10.0.3.7

There is a Load Balancer that contains an SSL cert resolution for https, 443 becomes 2113 through the LB, pointing to the external http port (2113) of each machine. The internal network is isolated to only the cluster itself (nothing from the outside can reach the internal network).

My question is: what connection string do I use externally to both access the gossip (I thought it was on external http) and to connect via tcp to the cluster? I’ve tried seeding the Gossip with the IP address of the load balancer and port :443 but the connection can’t resolve.

#2

Following up as this is a blocker for me getting the clusters fully configured in our environment. Here is more detail that might be helpful. I have been attempting to set up the cluster per the documentation under the clustered node section where it talks about separating internal/external traffic. The internal/external configuration is specified in my first post (or at the bottom). We have 3 connection points as described below – note that I distinguish between external and internal which is defined per the ES documentation (external != internet)

  1. “Internet” HTTP traffic – goes through a Gateway on :443, resolves SSL at the gateway, and then forward to :2113 on the ES external IPs (10.0.2.x).
  2. “Internet” TCP traffic - goes through a NATed Load Balancer on :1114-1116 which maps to :1113 on each of the ES servers external IP (10.0.2.x).
  3. “External” TCP traffic - comes from a “peered” network to the external ES servers IP (10.0.2.x) on ports :1113.

The documentation states that “You can control whether gossip is enable on external with the gossip-on-ext option (though you normally want it).” and that true is the default option.

Problem 1: For scenario 1 (and 2), what connection string do I use to gossip and connect from the “internet”? When I try “GossipSeeds=gateway-ip:443; HeartBeatTimeout=500” I get 10 attempts and nothing returned. I would like

Problem 2: For scenario 3, I think I can gossip on the external IPs to get back the ‘master’ and that should work, so the sees would be 10.0.2.6, 10.0.2.7.

Thanks in advance!

Josh

#3

Another follow up, here is a third option that might work, but it does open up the internal network, which may not be preferred:

Is it possible to configure ES to do the following:

  • For “Internet” connections, Gossip seeds go against a LB that NATs to the external IP range (10.0.2.x) on ports :1114, :1115, etc and it returns the internet-ip AND NAT PORT of the master node
  • For “external” connections (not coming from internet but from a peered network), gossip seeds go against the internal IP range (10.0.3.x) and it returns the internal ip of the master node.
    Is this a recommended configuration that would support connections coming both from the internet and a network “internal” (NOT ES internal) to the cluster? Do we gain anything by allowing internal vs external connectivity? We are thinking connections may be faster and more resilient when going internal vs external.

Thank again, sorry for all the posts!

#4

Any feedback on this?

I noticed that not being able to have a separate advertise IP for the external http and tcp ports made the gossip on the http useless because the web would attempt to redirect if the IP was set to the load balancer. :frowning:

#5

There is a setting for this advertise as which allows you to put a different address than the one that you actually bind to.

#6

I didn’t see a way to advertise a different IP between the TCP and HTTP endpoints… only different for external vs internal. Is there a new setting to allow a different advertise IP for external TCP and external HTTP?

#7

See the varying advertise-as arguments here: https://eventstore.org/docs/server/4.0.0/command-line-arguments/

Basically it allows you to do things like bind to a port 192.168.0.1 but advertise another address say 10.114.2.28. This can be useful in nat’ed and other similar environments.