Cluster Configuration

EventStoreDB
#1

Hello

I am trying to find a configuration that would work on Azure virtual machines, but it just seems to evade me.

So far, here is a configuration on one machine, with which I communicate from my computer which has full network access (Network security group allows any communication between my computer and the VM) to said event store machine.

Could you guys help me figure what is wrong in my configuration ?

Best regards

Omar

I have three sessions of event store on one machine.

CONFIG: /etc/eventstore/eventstore0.conf (Command Line)
START STANDARD PROJECTIONS: True (Config File)
RUN PROJECTIONS: ALL (Config File)
DB: /datadrive0/db/eventstore (Config File)
LOG: /datadrive0/log/eventstore (Config File)
STATS PERIOD SEC: 3600 (Config File)
MAX MEM TABLE SIZE: 500000 (Config File)
INT IP: 192.168.2.4 (Config File)
EXT IP: 192.168.2.4 (Config File)
INT HTTP PORT: 1113 (Config File)
EXT HTTP PORT: 1114 (Config File)
INT TCP PORT: 1111 (Config File)
EXT TCP PORT: 1112 (Config File)
INT HTTP PREFIXES: http://:1113/ (Config File)
EXT HTTP PREFIXES: http://:1114/ (Config File)
ADD INTERFACE PREFIXES: false (Config File)
DISCOVER VIA DNS: false (Config File)
PREPARE TIMEOUT MS: 7500 (Config File)
COMMIT TIMEOUT MS: 7500 (Config File)
SKIP DB VERIFY: true (Config File)
SKIP INDEX VERIFY: true (Config File)
EXT TCP HEARTBEAT TIMEOUT: 6000 (Config File)
EXT TCP HEARTBEAT INTERVAL: 3000 (Config File)
GOSSIP INTERVAL MS: 1000 (Config File)
GOSSIP TIMEOUT MS: 2000 (Config File)
GOSSIP SEED: 192.168.2.4:2114,192.168.2.4:3114 (Config File)
CLUSTER SIZE: 3 (Config File)
EXT IP ADVERTISE AS: XXX.XXX.XXX.XXX (Config File)
INT IP ADVERTISE AS: 192.168.2.4 (Config File)
EXT TCP PORT ADVERTISE AS: 1112 (Config File)

#2

Seeing logs would help but I am guessing an issue is coming from

EXT TCP PORT ADVERTISE AS: 3112 (Config File)
EXT IP ADVERTISE AS: XXX.XXX.XXX.XXX (Config File)
INT IP ADVERTISE AS: 192.168.2.4 (Config File)

Why do you need to use advertise as here? Also why are you setting internal advertise as ip to the same thing on all nodes?

#3

Hi Greg

As an initial test, the three nodes run on the same machine.

The adverstising is a suggestion from support

I am joining the log files hoping they are helful

Best regards

Omar

reproblemswithaclustercreation.zip (248 KB)

#4

There is no reason to use advertise as here (its setting it to what it would be anyways). So the use of advertise as is for when your actual address may not be what clients as example talk to you as. Let me give a concrete example…

We have a node which knows itself as 192.168.1.2 … The clients are on the other side of a router, due to this they must address the machine as 10.114.77.22. This setting allows the clients to know this. This is also not a very common setup and it can be handled in other ways as well.

Does this make sense?

#5

Ok let’s say I unset the advertise.

The cluster works, I noticed it yesterday, but I can’t connect from outside the network hosting the virtual machine with the .net client

Omar

#6

OK what are you connecting to? Is external traffic allowed into that network? Are you doing any NAT/port mapping etc? What do the client logs say? Most of this is basic tcp questions …

#7

Hi Greg,

Let me start by saying that we are paying customers of your company.

I am trying to set up what should be an easy recipe to apply, which is, create three nodes in a cluster on Azure and connect to them.

The cluster was up and running with default config (No Advertising etc.)

All communications to the machine are open to my development machine.

You lost me with the NAT/port management, all communication is direct to the vm from my computer, I get an exception (Tried 10 time and failed).

So I have no idea what I did wrong, but I would love a little help

Thanks

Omar

#8

I am asking about NAT etc because you have

EXT IP ADVERTISE AS: 52.177.203.170 (Config File)

This is telling the clients that they should connect to this IP address not the IP address that the server knows itself to be running on which is 192.168.2.4 for the first node. This implies that there is some sort of routing between the clients and the node as clients should address it as 52.177.203.170 but it is on 192.168.2.4.

If clients are using gossip to find nodes they will attempt to use this ip address to connect instead of what the node knows itself as.

Does this make sense?

#9

Not really

I would expect clients to connect to 52.177.203.170 as they are not in the network

so does this mean that I should keep the advertising

Omar

#10

Can you telnet to the port from outside of the network?

#11

Yes I could

I will redeploy the said configuration (Vanilla) without any advertise and get back to you with all the information

Cheers

Omar

#12

if you can telnet to the port from an outside machine that should be the same info that you give to the client…

#13

The machine is up with these logs

Telnet result

image.png

Client connection failure

image.png

Exactly the problem I struggled with yesterday

Omar

192.168.2.4-1114-cluster-node.log (59.9 KB)

#14

The port 1111 has the same behaviour

And all ports are open to my public IP Address as displayed below

image.png

#15

OK so these are unrelated.

When doing the telnet you are getting expected behaviour. You are connecting then it is sending a message it looks like…

You are setup to connect with Discover it looks like which requires being able to gossip with the cluster (to get nodes etc). You can enable verbose logging on the connection to see this failing …

Try just using the IP/port just like via telnet and my guess is it will work. When doing discovery the setup is a bit different…

#16

Ok So it works when I use the following code :

var connectionSettings = ConnectionSettings.Create()
.SetDefaultUserCredentials(new UserCredentials(“admin”, “changeit”))
.EnableVerboseLogging();
var esConnexion = EventStoreConnection.Create(connectionSettings, new Uri(“tcp://104.209.128.163:1111”));
//var esConnexion = EventStoreConnection.Create(connectionString).EnableVerboseLogging();

        esConnexion.ConnectAsync().Wait();

How would I go around to make it work with Discover ?

Is it unnecessary ?

What if I wanted to add dynamically extra machines and set up a Dns as I planned initially ?

Omar

#17

Just to be more bummed

I connected to one of the nodes. It happened not to be the master

When I tried to write an event I got kicked out

DO I have to address the Master only to write events ?

Thanks

Omar

#18

So there are multiple things for discovery…

  1. discovery must be able to talk to the http port (it gossips with the cluster to find nodes)

  2. discovery will continue gossiping and depending on changes/settings might drop connections/reconnect to different nodes etc as example if you want a master connection and the master moves it will reconnect to the new master

  3. discovery can survive nodes being added/removed without config changes

#19

No writing to a slave should forward to the master (if it knows of one) by default.

#20

Actually, here is how I got it to write to any random node

var connectionSettings = ConnectionSettings.Create()
.SetDefaultUserCredentials(new UserCredentials(“admin”, “changeit”))
.PerformOnAnyNode()
.UseConsoleLogger()
.EnableVerboseLogging();
var esConnexion = EventStoreConnection.Create(connectionSettings, new Uri(“tcp://104.209.128.163:1111”));

Otherwise I was unable to read not write from any node.

For the gossiping, The nodes are gossiping on the internal IP already. Should that be modified to be on the public IP if I wanted to use discovery ?

Would the discovery be helpful only if I have a DNS setup ?

Thanks by the way you directed me to finding the proper answer.

Omar