Will eventstore keep asking the dns server for ips if it fails to resolve? (playing with kubernetes)

Tomas_Jansson · December 18, 2015, 11:25pm

I’m playing around with kubernetes and thought setting up an eventstore cluster would be a fun exercise. I’m basing it on wkruse/eventstore docker container without modification. My replication controller definition looks like this:

apiVersion: v1

kind: ReplicationController

metadata:

labels:

component: eventstore

spec:

replicas: 3

selector:

component: eventstore

template:

metadata:

labels:

component: eventstore

spec:

serviceAccount: eventstore

containers:

name: eventstore

image: wkruse/eventstore

command: ["./run-node.sh","–ext-ip=0.0.0.0","–int-ip=0.0.0.0","–cluster-size=3","–cluster-gossip-port=2113","–cluster-dns=eventstore.default","–http-prefixes=http://*:2113/","–run-projections=all"]

ports:

containerPort: 2113

protocol: TCP

containerPort: 1113

protocol: TCP

I also have this service description:

apiVersion: v1

kind: Service

metadata:

labels:

component: eventstore

spec:

type: LoadBalancer

clusterIP: None

selector:

component: eventstore

ports:

name: http

port: 2113

protocol: TCP

name: transport

port: 1113

protocol: TCP

and this service account set up:

apiVersion: v1

kind: ServiceAccount

metadata:

And this works almost. The only problem is that at the moment when eventstore tries to resolve the dns name it seems like the dns hasn’t yet been configured, so it can’t find any nodes to connect to. If I wait for the kubernetes pods to be in the running state I can make a dns query and get all three ips back. So it seems like the dns is working.

How is the eventstore cluster supposed to work? If a client is dropped and then come back online but with a different IP will the cluster be able to resolve that? Also, if there are no other nodes available at start, will the node query again later? It sort of makes sense to do so if I have defined it to be in a cluster using dns and if it can’t resolve the other nodes it’s no point start gossiping.

jen20 · December 18, 2015, 11:37pm

How is the eventstore cluster supposed to work? If a client is dropped and then come back online but with a different IP will the cluster be able to resolve that? Also, if there are no other nodes available at start, will the node query again later? It sort of makes sense to do so if I have defined it to be in a cluster using dns and if it can’t resolve the other nodes it’s no point start gossiping.

Clusters never try to resolve addresses of clients?

https://github.com/EventStore/EventStore/blob/release-v3.5.0/src/EventStore.Core/Services/Gossip/DnsGossipSeedSource.cs is how the DNS resolution is done.

Tomas_Jansson · December 19, 2015, 10:34pm

Thanks for answering James. But what happens if there are no ip addresses from the dns resolution result? Will the nodes try again every time they gossip?

Greg_Young1 · December 19, 2015, 10:36pm

There are back off strategies etc there... There are also retry count
strategies in case it can't connect etc

Tomas_Jansson · December 19, 2015, 10:40pm

Ok. I’ll just play around some more during the holidays if I get some time. It is most likely something I’m doing wrong :). I should also use put together an image with the newest version.

Greg_Young1 · December 19, 2015, 10:47pm

Can you explain more the exact issue you are running into?

Tomas_Jansson · December 19, 2015, 10:59pm

I can give the show the logs when I get some computer time.

When the nodes starts it seems like they all didn’t get hold of the IPs since the IPs for the nodes hadn’t yet been registered. After that I looked like they just failed to establish the cluster.

I will write the logs as soon as I can.

Tomas_Jansson · December 21, 2015, 11:17pm

I finally had the time to recreate the logs. Note that this is version 3.0.1 of eventstore. I do think that I might have found the problem, and that is that the internal communication is on port 1112 and 2112 and isn’t that what is used for communicating between nodes? I forgot to add that those ports should be exposed from the container. Either way, here are the arguments and log from one of the node.

Arguments: “./run-node.sh”,"–ext-ip=0.0.0.0","–int-ip=0.0.0.0","–cluster-size=3","–cluster-gossip-port=2113","–cluster-dns=eventstore.default","–http-prefixes=http://*:2113/","–run-projections=all"

Log:

ES VERSION: 3.0.1.0 (release-oss-v3.0.1/7fa876c111888dd5980dbd86d126e6abe13b05ab, Thu, 23 Oct 2014 22:27:04 +0100)

OS: Unknown (Unix 3.17.4.301)

RUNTIME: 3.8.0 ((detached/45d0ba1) (64-bit)

GC: 2 GENERATIONS

LOGS: /data/logs

HELP: False ()

VERSION: False ()

LOG: /data/logs (Environment Variable)

CONFIG: ()

DEFINES: ()

WHAT IF: False ()

INT IP: 0.0.0.0 (Command Line)

EXT IP: 0.0.0.0 (Command Line)

INT HTTP PORT: 2112 ()

EXT HTTP PORT: 2113 ()

INT TCP PORT: 1112 ()

INT SECURE TCP PORT: 0 ()

EXT TCP PORT: 1113 ()

EXT SECURE TCP PORT: 0 ()

INT TCP HEARTBEAT TIMEOUT: 700 ()

EXT TCP HEARTBEAT TIMEOUT: 1000 ()

INT TCP HEARTBEAT INTERVAL: 700 ()

EXT TCP HEARTBEAT INTERVAL: 2000 ()

FORCE: False ()

CLUSTER SIZE: 3 (Command Line)

NODE PRIORITY: 0 ()

MIN FLUSH DELAY MS: 2 ()

COMMIT COUNT: -1 ()

PREPARE COUNT: -1 ()

ADMIN ON EXT: True ()

STATS ON EXT: True ()

GOSSIP ON EXT: True ()

DISABLE SCAVENGE MERGING: False ()

DISCOVER VIA DNS: True ()

CLUSTER DNS: eventstore.default (Command Line)

CLUSTER GOSSIP PORT: 2113 (Command Line)

GOSSIP SEED: ()

STATS PERIOD SEC: 30 ()

CACHED CHUNKS: -1 ()

CHUNKS CACHE SIZE: 536871424 ()

MAX MEM TABLE SIZE: 1000000 ()

DB: /data/db (Environment Variable)

MEM DB: False ()

SKIP DB VERIFY: False ()

RUN PROJECTIONS: All (Command Line)

PROJECTION THREADS: 3 ()

WORKER THREADS: 5 ()

HTTP PREFIXES: [ http://*:2113/ ] (Command Line)

ENABLE TRUSTED AUTH: False ()

CERTIFICATE STORE LOCATION: ()

CERTIFICATE STORE NAME: ()

CERTIFICATE SUBJECT NAME: ()

CERTIFICATE THUMBPRINT: ()

CERTIFICATE FILE: ()

CERTIFICATE PASSWORD: ()

USE INTERNAL SSL: False ()

SSL TARGET HOST: n/a ()

SSL VALIDATE SERVER: True ()

AUTHENTICATION TYPE: internal ()

PREPARE TIMEOUT MS: 2000 ()

COMMIT TIMEOUT MS: 2000 ()

UNSAFE DISABLE FLUSH TO DISK: False ()

GOSSIP INTERVAL MS: 1000 ()

GOSSIP ALLOWED DIFFERENCE MS: 60000 ()

GOSSIP TIMEOUT MS: 500 ()

[00007,01,22:57:27.951] Quorum size set to 2

[00007,01,22:57:27.955] Can’t find plugins path: plugins

[00007,01,22:57:27.975]

INSTANCE ID: 71f39985-d770-4894-996c-a27746a68265

DATABASE: /data/db

WRITER CHECKPOINT: 0 (0x0)

CHASER CHECKPOINT: 0 (0x0)

EPOCH CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF)

TRUNCATE CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF)

[00007,01,22:57:28.120] MessageHierarchy initialization took 00:00:00.1104338.

[00007,01,22:57:28.201] CACHED TFChunk #0-0 (chunk-000000.000000) in 00:00:00.0014867.

[00007,01,22:57:28.310] Starting MiniWeb for /web/es/js/projections ==> /opt/EventStore-OSS-Linux-v3.0.1/projections

[00007,01,22:57:28.312] Starting MiniWeb for /web/es/js/projections/v8/Prelude ==> /opt/EventStore-OSS-Linux-v3.0.1/Prelude

[00007,01,22:57:28.312] Starting MiniWeb for /web/es/js/projections/resources ==> /opt/EventStore-OSS-Linux-v3.0.1/web-resources/js

[00007,01,22:57:28.313] Binding MiniWeb to /web/es/js/projections/{*remaining_path}

[00007,01,22:57:28.313] Binding MiniWeb to /web/es/js/projections/v8/Prelude/{*remaining_path}

[00007,01,22:57:28.313] Binding MiniWeb to /web/es/js/projections/resources/{*remaining_path}

[00007,01,22:57:28.314] Binding MiniWeb to /web/es/js/projections/{*remaining_path}

[00007,01,22:57:28.314] Binding MiniWeb to /web/es/js/projections/v8/Prelude/{*remaining_path}

[00007,01,22:57:28.314] Binding MiniWeb to /web/es/js/projections/resources/{*remaining_path}

[00007,01,22:57:28.323] Starting MiniWeb for /web ==> /opt/EventStore-OSS-Linux-v3.0.1/clusternode-web

[00007,01,22:57:28.323] Binding MiniWeb to /web/{*remaining_path}

[00007,01,22:57:28.323] Starting MiniWeb for /web/users ==> /opt/EventStore-OSS-Linux-v3.0.1/Users/web

[00007,01,22:57:28.323] Binding MiniWeb to /web/users/{*remaining_path}

[00007,01,22:57:28.323] Starting MiniWeb for /web ==> /opt/EventStore-OSS-Linux-v3.0.1/clusternode-web

[00007,01,22:57:28.323] Binding MiniWeb to /web/{*remaining_path}

[00007,01,22:57:28.324] Starting MiniWeb for /web/users ==> /opt/EventStore-OSS-Linux-v3.0.1/Users/web

[00007,01,22:57:28.324] Binding MiniWeb to /web/users/{*remaining_path}

[00007,10,22:57:28.341] ========== [0.0.0.0:2112] SYSTEM INIT…

[00007,10,22:57:28.359] Starting Normal TCP listening on TCP endpoint: 0.0.0.0:1113.

[00007,10,22:57:28.374] Starting Normal TCP listening on TCP endpoint: 0.0.0.0:1112.

[00007,15,22:57:28.386] TableIndex initialization…

[00007,15,22:57:28.413] ReadIndex building…

[00007,15,22:57:28.414] ReadIndex rebuilding done: total processed 0 records, time elapsed: 00:00:00.0005910.

[00007,10,22:57:28.428] Starting HTTP server on [http://*:2113/]…

[00007,10,22:57:28.435] HTTP server is up and listening on [http://*:2113/]

[00007,10,22:57:28.451] SLOW BUS MSG [MainBus]: SystemInit - 60ms. Handler: HttpService.

[00007,10,22:57:28.451] Starting HTTP server on [http://0.0.0.0:2112/]…

[00007,10,22:57:28.451] HTTP server is up and listening on [http://0.0.0.0:2112/]

[00007,10,22:57:28.474] SLOW QUEUE MSG [MainQueue]: SystemInit - 130ms. Q: 0/7.

[00007,10,22:57:28.474] ========== [0.0.0.0:2112] Service ‘StorageWriter’ initialized.

[00007,10,22:57:28.474] ========== [0.0.0.0:2112] Service ‘StorageReader’ initialized.

[00007,10,22:57:28.475] ========== [0.0.0.0:2112] Service ‘StorageChaser’ initialized.

[00007,10,22:57:28.477] ========== [0.0.0.0:2112] SYSTEM START…

[00007,10,22:57:28.480] ========== [0.0.0.0:2112] IS UNKNOWN!!! WHOA!!!

[00007,10,22:57:28.503] ELECTIONS: STARTING ELECTIONS.

[00007,10,22:57:28.503] ELECTIONS: (V=0) SHIFT TO LEADER ELECTION.

[00007,10,22:57:28.503] ELECTIONS: (V=0) VIEWCHANGE FROM [0.0.0.0:2112, {71f39985-d770-4894-996c-a27746a68265}].

[00007,06,22:57:28.522] Error while retrieving cluster members through DNS.

Could not resolve host ‘eventstore.default’

[00007,10,22:57:29.506] ELECTIONS: (V=0) TIMED OUT! (S=ElectingLeader, M=).

[00007,10,22:57:29.506] ELECTIONS: (V=1) SHIFT TO LEADER ELECTION.

[00007,10,22:57:29.506] ELECTIONS: (V=1) VIEWCHANGE FROM [0.0.0.0:2112, {71f39985-d770-4894-996c-a27746a68265}].

[00007,10,22:57:29.549] CLUSTER HAS CHANGED

Old:

VND {71f39985-d770-4894-996c-a27746a68265} [Unknown, 0.0.0.0:1112, n/a, 0.0.0.0:1113, n/a, 0.0.0.0:2112, 0.0.0.0:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-12-21 22:57:28.484

New:

MAN {00000000-0000-0000-0000-000000000000} [Manager, 10.246.1.9:2113, 10.246.1.9:2113] | 2015-12-21 22:57:29.536

VND {71f39985-d770-4894-996c-a27746a68265} [Unknown, 0.0.0.0:1112, n/a, 0.0.0.0:1113, n/a, 0.0.0.0:2112, 0.0.0.0:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-12-21 22:57:28.484

Tomas_Jansson · December 30, 2015, 12:34am

I created my own docker images with 3.4.0 which also has some code so I can set the --ext-ip to the ip of the container. It’s available on docker hub under mastoj/eventstore.

That didn’t solve my issue though. Running one node works fine, but trying to set up the cluster doesn’t work. I think I do the same as described in the documentation using these arguments:

“–ext-ip=$eth0”,"–int-ip=$eth0","–cluster-size=3","–cluster-gossip-port=2113","–cluster-dns=eventstore.default.svc","–run-projections=all"]

``

$eth0 will expand to the ip of the container it executes in. nslookup on eventstore.default.svc gives:

Name: eventstore.default.svc
Address 1: 10.246.52.12
Address 2: 10.246.52.11
Address 3: 10.246.52.10

The logs from one of the is below. Almost the same problem as before but now I have provided the container IPs insead of 0.0.0.0 for int and ext ip. Do I also need to set advertise ip here to get it working? Enough with int advertise or ext advertise as well? Do I need to set advertise port? Does the sample code from the documentation work, and why does that differ from the way I do it?

Greg_Young1 · December 30, 2015, 12:39am

See previous reply re advertiseas

you are not setting them:

EXT IP ADVERTISE AS: <empty> (<DEFAULT>)
EXT TCP PORT ADVERTISE AS: 0 (<DEFAULT>)
EXT HTTP PORT ADVERTISE AS: 0 (<DEFAULT>)
INT IP ADVERTISE AS: <empty> (<DEFAULT>)
INT SECURE TCP PORT ADVERTISE AS: 0 (<DEFAULT>)
INT TCP PORT ADVERTISE AS: 0 (<DEFAULT>)
INT HTTP PORT ADVERTISE AS: 0 (<DEFAULT>)

Tomas_Jansson · December 30, 2015, 10:27am

Thank for taking your time Greg,

I have the same issue when advertising the IP, or am I advertising the wrong IP?

Greg_Young1 · December 30, 2015, 11:06am

Your nodes are not talking to each other.

You should not set advertise as to the same thing you are bound to
that is the default. An example of where advertise as is needed can be
seen with elastic ips.

I am bound to 192.168.1.1 but I am addressable from everyone else as
10.72.23.11 (no traffic can route to 192.168.1.1). In this case I
would want to bind to 192.168.1.1 but advertise myself as 10.72.23.11.
There are also cases with NAT where although I am bound to port 2113 I
am only externally visible as port 7777 which is why you can also
change the advertisement for ports.

Cheers,

Greg

Tomas_Jansson · December 30, 2015, 11:15pm

The setup I’m using doesn’t use NAT. If I fire up just one node, then that node is accessible from other containers in the cluster, verified with curl from another container. So I think it is weird that the container can’t talk to each other. Also, I don’t think I would need the advertise as stuff since they should be able to reach each other with the IP they are bound to, otherwise I wouldn’t be able to curl from a container to the single node setup, right?

For the single node part I can both curl against the prefix/dns name and to the IP directly to the container. It feels like there is something really obvious part I’ve missed.

Networking in kubernetes is described here if interested, https://github.com/kubernetes/kubernetes/blob/release-1.1/docs/design/networking.md. For the setup I’m using it is Pod to Pod communication. A pod is most often one container, but you can also group pods but I don’t want that in this case. I want to put the eventstore pod behind a kubernetes service which should make it really easy to setup any type of cluster in kubernetes.

Greg_Young1 · December 30, 2015, 11:38pm

DNS name has nothing to do with it. As I said your conifg is also
wrong as you have both ext-ip and advertise as the same thing, the
point of advertise as is when they are different.

Greg_Young1 · December 30, 2015, 11:39pm

although I must laugh a bit as well, "I am not using NAT, I am in a
docker container..."

Tomas_Jansson · December 31, 2015, 12:02am

But do I need advertise as if the container should be able to communicate to each other on the IP they are bound to?

My setup should be something like:

Container ES1: IP X

Container ES2: IP Y

Container ES3: IP Z

All containers expose port 2112, 2113, 1112, 1113, which means they can talk to each other internally on those ports in the network kubernetes has set up. To communicate with the cluster I have set up a kubernetes service of type load balancer on top of those containers.

This works perfectly fine for one node. Do I really need to use advertise as if the containers can reach each other on IP X, Y and Z? If I need it, what should it be?

Is it possible to update the cluster documentation with this information?

Tomas_Jansson · December 31, 2015, 12:20am

It’s ok to laugh, one have to be stupid to learn ;). I just took that from the documentation of kubernetes networking: https://github.com/kubernetes/kubernetes/blob/release-1.1/docs/design/networking.md#pod-to-pod.

When any container calls ioctl(SIOCGIFADDR) (get the address of an interface), it sees the same IP that any peer container would see them coming from — each pod has its own IP address that other pods can know. By making IP addresses and ports the same both inside and outside the pods, we create a NAT-less, flat address space. Running “ip addr show” should work as expected.

As I understand it, the network internally in the cluster works like they are not NAT. Or am I totally mislead?

Greg_Young1 · December 31, 2015, 7:50am

Really easy way to try. From one node run curl http://anothernode:2113/gossip

Tomas_Jansson · January 2, 2016, 12:05am

I have my node running on:

Name: eventstore.default

Address 1: 10.246.5.7

Address 2: 10.246.5.8

Address 3: 10.246.5.9

Gossip req to 10.246.5.x:2113/gossip where x is 7, 8 or 9 works perfectly fine from another container not running eventstore. They are all on the same network so I really don’t see any reason for the gossip to fail between the nodes. I really don’t see this as a problem with eventstore since I know it works, it is just frustrating not to get my kubernetes plyaground up and running :).

Greg_Young1 · January 2, 2016, 8:23am

If you can gossip from one to another then you should be able to bring
up a cluster without advertise as etc. Is ES bound to 10.246.5.7? What
does gossip return?