SSL Termination

EventStoreDB
1

I have an issue around connecting to the cluster.

Each node has eventstore and haproxy providing tls termination; which on the face of it works fine; but when it comes to the client fails (configs below).

The cluster works (status has all 3 nodes) but it fails at the client on discovery.

Is this related to https://github.com/EventStore/EventStore/issues/1017 ?

I’ve tried flipping settings around http-prefixes but I’m assuming this only affects the UI component.

Using this code

static void Main(string[] args)

{

// var seed = DNS GET NODES FROM CONSUL

var esS = EventStoreConnection.Create(seed.FirstOrDefault());

esS.ConnectAsync().Wait();

var settings = ConnectionSettings.Create()

.KeepReconnecting()

.KeepRetrying()

.SetHeartbeatInterval(TimeSpan.FromSeconds(30))

.SetGossipTimeout(TimeSpan.FromMinutes(5))

.SetHeartbeatTimeout(TimeSpan.FromMinutes(5))

.SetTimeoutCheckPeriodTo(TimeSpan.FromMinutes(1))

.UseSslConnection(“ges-haproxy-tcp.service.consul”, false)

.SetDefaultUserCredentials(new UserCredentials(“admin”, “changeit”))

.UseConsoleLogger().EnableVerboseLogging()

.Build();

var clusterSettings = ClusterSettings.Create()

.DiscoverClusterViaGossipSeeds()

.SetGossipSeedEndPoints(seed.Select(x => new IPEndPoint(x.Address, 2113)).ToArray())

.SetGossipTimeout(TimeSpan.FromMinutes(5))

.SetMaxDiscoverAttempts(500)

.Build();

var eventStoreConnection = EventStoreConnection.Create(settings, clusterSettings, “testconnection”);

eventStoreConnection.ConnectAsync().Wait();

for (var i = 0; i < 1000; i++)

{

var s = eventStoreConnection.AppendToStreamAsync(Guid.NewGuid().ToString(“N”), -2,

new List() {new EventData(Guid.NewGuid(), “t”, false, new byte[0], new byte[0])}).ConfigureAwait(false);

}

Console.ReadLine();

}

With 2113 set (which goes via HTTP to the GES nodes) it works fine and resolves the nodes to 1114 which is TLS terminated TCP at haproxy.

Set to 2114 which I’d expect to be right it just throws discovery errors.

So am I right in thinking (and going off the github issue) that you can’t terminate TLS discovery?

Is there anyway around this currently? Why do I even need discovery when I’ve fetched the cluster to feed the seeds?

EVENTSTORE

RunProjections: system

StartStandardProjections: true

Log: “/var/log/eventstore”

Db: /gesdata

IntIp: 0.0.0.0

ExtIp: 127.0.0.1

IntIpAdvertiseAs: 10.0.0.41

ExtIpAdvertiseAs: 52.0.0.1

IntHttpPort: 2112

ExtHttpPort: 2113

IntTcpPort: 1112

ExtTcpPort: 1113

ExtTcpPortAdvertiseAs: 1114

ExtHttpPortAdvertiseAs: 2114

IntTcpHeartbeatTimeout: 2000

ExtTcpHeartbeatTimeout: 2000

IntTcpHeartbeatInterval: 5000

ExtTcpHeartbeatInterval: 5000

GossipTimeoutMs: 2000

DiscoverViaDns: true

AdminOnExt: true

GossipOnExt: true

ClusterSize: 3

ClusterDns: “eventstore.service.consul”

ClusterGossipPort: 2112

IntHttpPrefixes: http://*:2112/

ExtHttpPrefixes: http://*:2113/

AddInterfacePrefixes: false

HAPROXY

global

daemon

tune.ssl.default-dh-param 2048

user haproxy

group haproxy

defaults

log 127.0.0.1 local0

log 127.0.0.1 local1 notice

timeout connect 5000ms # max time to wait for a connection attempt to a server to succeed

timeout client 50000ms # max inactivity time on the client side

timeout server 50000ms # max inactivity time on the server side

frontend http_frontend

bind *:2114 ssl crt /etc/certs/server.pem ca-file /etc/certs/ca.crt

mode http

option forwardfor

option http-server-close

option httpclose

we add this so the backend servers know the request was ssl

otherwise we could end up in a redirect loop

reqadd X-Forwarded-Proto:\ https

reqadd X-Forwarded-Scheme:\ https

reqadd X-Forwarded-Port:\ 2114

default_backend http_backend

backend http_backend

mode http

timeout connect 5s

timeout server 30s

server ui 127.0.0.1:2113 check

frontend gestcp_frontend

bind *:1114 ssl crt /etc/certs/server.pem ca-file /etc/certs/ca.crt

mode tcp

option tcplog

timeout client 1m

default_backend ges_backend

backend ges_backend

mode tcp

option tcplog

option log-health-checks

option redispatch

log global

timeout connect 10s

timeout server 1m

server i-x 127.0.0.1:1113 check

2

just FYI the first

var esS = EventStoreConnection.Create(seed.FirstOrDefault());

esS.ConnectAsync().Wait();

was just to check a random test.

I’ll add though that over https on the external port of a node

https://52.ext:2114/gossip

produces

3

Ok looking at the netcore client it connects and calls out to the gossip url on http

{Method: GET, RequestUri: ‘http://52.xxxxx:2114/gossip?format=json’, Version: 1.1, Content: <nu

Which spawns from the ClusterDnsEndpointDiscovery > TryGetGossipFrom using HTTP schema regardless of the ssl settings passed

   private ClusterMessages.ClusterInfoDto TryGetGossipFrom(GossipSeed endPoint)

    {

        //_log.Debug("ClusterDnsEndPointDiscoverer: Trying to get gossip from [{0}].", endPoint);

        ClusterMessages.ClusterInfoDto result = null;

        var completed = new ManualResetEventSlim(false);

        var url = endPoint.EndPoint.ToHttpUrl(EndpointExtensions.HTTP_SCHEMA, "/gossip?format=json");

        _client.Get(

            url,

            null,

            response =>

            {

                if (response.HttpStatusCode != HttpStatusCode.OK)

                {

                    //_log.Info("[{0}] responded with {1} ({2})", endPoint, response.HttpStatusCode, response.StatusDescription);

                    completed.Set();

                    return;

                }

                try

                {

                    result = response.Body.ParseJson<ClusterMessages.ClusterInfoDto>();

                    //_log.Debug("ClusterDnsEndPointDiscoverer: Got gossip from [{0}]:\n{1}.", endPoint, string.Join("\n", result.Members.Select(x => x.ToString())));

                }

                catch (Exception)

                {

                    //_log.Info("Failed to get cluster info from [{0}]: deserialization error: {1}.", endPoint, e.Message);

                }

                completed.Set();

            },

            e =>

            {

                //_log.Info("Failed to get cluster info from [{0}]: request failed, error: {1}.", endPoint, e.Message);

                completed.Set();

            }, endPoint.HostHeader);

        completed.Wait();

        return result;

    }

The inability to pass in a custom httpclient handler of course makes it a bugger to alter the client cert validation / pass client certs up.

On the upside bashing that chunk of code with a rock fixes the issue around tls termination

4

Raise this a a github issue its not a lot of work and is reasonable.

5

I’ve done the code but for the netcore client

6

Added with PR https://github.com/EventStore/ClientAPI.NetCore/issues/28