Cluster cannot elect leader when using HTTP Proxy

Hello all,

I have recently converted our cluster to run through an NGINX Reverse Proxy, ever since the cluster will not elect a leader. I have attached log files below along with NGINX Config and Eventstore Config

The NGINX is running on a Docker Container in HOST Mode.

Many Thanks

Josh

[10565,12,15:30:51.583] ELECTIONS: (V=552) VIEWCHANGE FROM [127.0.0.1:8082, {3e4e8296-f041-4602-9340-f3aae388b0d2}].
[10565,12,15:30:52.041] Time difference between us and [192.168.33.14:6113] is too great! UTC now: 2018-01-23 15:30:52.041, peer’s time stamp: 2018-01-23 15:22:11.558.
[10565,12,15:30:52.589] ELECTIONS: (V=552) TIMED OUT! (S=ElectingLeader, M=).
[10565,12,15:30:52.589] ELECTIONS: (V=553) SHIFT TO LEADER ELECTION.
[10565,12,15:30:52.589] ELECTIONS: (V=553) VIEWCHANGE FROM [127.0.0.1:8082, {3e4e8296-f041-4602-9340-f3aae388b0d2}].
[10565,12,15:30:53.060] Time difference between us and [192.168.33.14:6113] is too great! UTC now: 2018-01-23 15:30:53.060, peer’s time stamp: 2018-01-23 15:22:11.558.
[10565,12,15:30:53.593] ELECTIONS: (V=553) TIMED OUT! (S=ElectingLeader, M=).
[10565,12,15:30:53.593] ELECTIONS: (V=554) SHIFT TO LEADER ELECTION.
[10565,12,15:30:53.593] ELECTIONS: (V=554) VIEWCHANGE FROM [127.0.0.1:8082, {3e4e8296-f041-4602-9340-f3aae388b0d2}].
[10565,12,15:30:54.051] Time difference between us and [192.168.33.11:6113] is too great! UTC now: 2018-01-23 15:30:54.051, peer’s time stamp: 2018-01-23 15:22:22.327.
[10565,12,15:30:54.603] ELECTIONS: (V=554) TIMED OUT! (S=ElectingLeader, M=).
[10565,12,15:30:54.604] ELECTIONS: (V=555) SHIFT TO LEADER ELECTION.
[10565,12,15:30:54.604] ELECTIONS: (V=555) VIEWCHANGE FROM [127.0.0.1:8082, {3e4e8296-f041-4602-9340-f3aae388b0d2}].
[10565,12,15:30:55.055] Time difference between us and [192.168.33.13:6113] is too great! UTC now: 2018-01-23 15:30:55.055, peer’s time stamp: 2018-01-23 15:22:22.327.
[10565,12,15:30:55.606] ELECTIONS: (V=555) TIMED OUT! (S=ElectingLeader, M=).
[10565,12,15:30:55.606] ELECTIONS: (V=556) SHIFT TO LEADER ELECTION.
[10565,12,15:30:55.606] ELECTIONS: (V=556) VIEWCHANGE FROM [127.0.0.1:8082, {3e4e8296-f041-4602-9340-f3aae388b0d2}].
[10565,12,15:30:56.059] Time difference between us and [192.168.33.13:6113] is too great! UTC now: 2018-01-23 15:30:56.059, peer’s time stamp: 2018-01-23 15:22:22.327.
[10565,12,15:30:56.614] ELECTIONS: (V=556) TIMED OUT! (S=ElectingLeader, M=).
[10565,12,15:30:56.614] ELECTIONS: (V=557) SHIFT TO LEADER ELECTION.
[10565,12,15:30:56.614] ELECTIONS: (V=557) VIEWCHANGE FROM [127.0.0.1:8082, {3e4e8296-f041-4602-9340-f3aae388b0d2}].
[10565,12,15:30:57.064] Time difference between us and [192.168.33.13:6113] is too great! UTC now: 2018-01-23 15:30:57.064, peer’s time stamp: 2018-01-23 15:22:22.327.

``

http {

sendfile on;

tcp_nopush on;

tcp_nodelay on;

keepalive_timeout 65;

types_hash_max_size 2048;

include /etc/nginx/mime.types;

default_type application/octet-stream;

access_log /var/log/nginx/access.log;

error_log /var/log/nginx/error.log;

gzip on;

gzip_disable “msie6”;

include /etc/nginx/conf.d/*.conf;

server {

listen 6114;

ssl off;

location / {

proxy_set_header X-Forwarded-For $remote_addr;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forwarded-Host $http_host;

proxy_pass http://localhost:8081;

proxy_redirect off;

}

}

server {

listen 6113;

ssl off;

location / {

proxy_set_header X-Forwarded-For $remote_addr;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forwarded-Host $http_host;

proxy_pass http://localhost:8082;

proxy_redirect off;

}

}

}

stream {

server {

listen 6112;

proxy_pass localhost:8083;

proxy_buffer_size 16k;

}

server {

listen 6111;

proxy_pass localhost:8084;

proxy_buffer_size 16k;

}

}

``

cluster-size: 3
discoverViaDns: false
GossipSeed: [‘192.168.33.11:6113,192.168.33.14:6113’]
ext-http-port: 8081
ext-ip: 127.0.0.1
ext-tcp-port: 8083
int-http-port: 8082
int-ip: 127.0.0.1
int-tcp-port: 8084
db: /var/lib/eventstore/test
log: /var/log/eventstore/test
run-projections: All

``

Hi Josh,

Could you try add an advertised IP address to your configuration, and set these to the ip addresses used by your proxy.

For example :

ExtIpAdvertiseAs: 192.168.33.11
IntIpAdvertiseAs: 192.168.33.11

``

These options set the address the node will send to the other nodes in the cluster as its address.

This worked a treat! Thank You very much!