Connecting a read replica to the ES Cloud cluster

EventStoreDB V22.X
1

Hello :wave:,

I’m stuck trying to connect external read-only replica to the cluster hosted on ES Cloud

2023-10-23T18:42:56Z app[2874deea62d798] lhr [info][  405,16,18:42:56.720,INF] CLUSTER HAS CHANGED "gossip send failed to [192.168.21.3:2113/cjxxxlg.mesdb.eventstore.cloud]"
2023-10-23T18:42:56Z app[2874deea62d798] lhr [info]Old:
2023-10-23T18:42:56Z app[2874deea62d798] lhr [info]["Priority: 0 VND {aade97f6-857f-4bdd-9207-c65c30c6d2d4} <LIVE> [ReadOnlyLeaderless, Unspecified/172.19.5.10:1112, n/a, Unspecified/:::1113, n/a, Unspecified/:::2113, (ADVERTISED: HTTP::0, TCP::0), Version: 23.6.0.0] 22271/22423/22423/E15@21623:{52ef371f-4eaa-41a8-969c-24c746670ba7} | 2023-10-23 18:42:56.649", "MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 192.168.21.4:2113/cjxxxlg.mesdb.eventstore.cloud] | 2023-10-23 18:42:56.525", "MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 192.168.21.3:2113/cjxxxlg.mesdb.eventstore.cloud] | 2023-10-23 18:42:56.525", "MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 192.168.21.2:2113/cjxxxlg.mesdb.eventstore.cloud] | 2023-10-23 18:42:56.525"]
2023-10-23T18:42:56Z app[2874deea62d798] lhr [info]New:
2023-10-23T18:42:56Z app[2874deea62d798] lhr [info]["Priority: 0 VND {aade97f6-857f-4bdd-9207-c65c30c6d2d4} <LIVE> [ReadOnlyLeaderless, Unspecified/172.19.5.10:1112, n/a, Unspecified/:::1113, n/a, Unspecified/:::2113, (ADVERTISED: HTTP::0, TCP::0), Version: 23.6.0.0] 22271/22423/22423/E15@21623:{52ef371f-4eaa-41a8-969c-24c746670ba7} | 2023-10-23 18:42:56.649", "MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 192.168.21.4:2113/cjxxxlg.mesdb.eventstore.cloud] | 2023-10-23 18:42:56.525", "MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 192.168.21.3:2113/cjxxxlg.mesdb.eventstore.cloud] | 2023-10-23 18:42:56.720", "MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 192.168.21.2:2113/cjxxxlg.mesdb.eventstore.cloud] | 2023-10-23 18:42:56.525"]
2023-10-23T18:42:56Z app[2874deea62d798] lhr [info][  405,16,18:42:56.720,INF] Looks like node ["192.168.21.4:2113/cjxxxlg.mesdb.eventstore.cloud"] is DEAD (Gossip send failed).
2023-10-23T18:42:56Z app[2874deea62d798] lhr [info][  405,16,18:42:56.721,INF] CLUSTER HAS CHANGED "gossip send failed to [192.168.21.4:2113/cjxxxlg.mesdb.eventstore.cloud]"
2023-10-23T18:42:56Z app[2874deea62d798] lhr [info]Old:
2023-10-23T18:42:56Z app[2874deea62d798] lhr [info]["Priority: 0 VND {aade97f6-857f-4bdd-9207-c65c30c6d2d4} <LIVE> [ReadOnlyLeaderless, Unspecified/172.19.5.10:1112, n/a, Unspecified/:::1113, n/a, Unspecified/:::2113, (ADVERTISED: HTTP::0, TCP::0), Version: 23.6.0.0] 22271/22423/22423/E15@21623:{52ef371f-4eaa-41a8-969c-24c746670ba7} | 2023-10-23 18:42:56.649", "MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 192.168.21.4:2113/cjxxxlg.mesdb.eventstore.cloud] | 2023-10-23 18:42:56.525", "MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 192.168.21.3:2113/cjxxxlg.mesdb.eventstore.cloud] | 2023-10-23 18:42:56.720", "MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 192.168.21.2:2113/cjxxxlg.mesdb.eventstore.cloud] | 2023-10-23 18:42:56.525"]
2023-10-23T18:42:56Z app[2874deea62d798] lhr [info]New:
2023-10-23T18:42:56Z app[2874deea62d798] lhr [info]["Priority: 0 VND {aade97f6-857f-4bdd-9207-c65c30c6d2d4} <LIVE> [ReadOnlyLeaderless, Unspecified/172.19.5.10:1112, n/a, Unspecified/:::1113, n/a, Unspecified/:::2113, (ADVERTISED: HTTP::0, TCP::0), Version: 23.6.0.0] 22271/22423/22423/E15@21623:{52ef371f-4eaa-41a8-969c-24c746670ba7} | 2023-10-23 18:42:56.649", "MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 192.168.21.4:2113/cjxxxlg.mesdb.eventstore.cloud] | 2023-10-23 18:42:56.720", "MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 192.168.21.3:2113/cjxxxlg.mesdb.eventstore.cloud] | 2023-10-23 18:42:56.720", "MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 192.168.21.2:2113/cjxxxlg.mesdb.eventstore.cloud] | 2023-10-23 18:42:56.525"]
2023-10-23T18:42:56Z app[2874deea62d798] lhr [info][  405,16,18:42:56.721,DBG] "NO LEADER" found in READ ONLY LEADERLESS state, making further attempts.
2023-10-23T18:42:56Z app[2874deea62d798] lhr [info][  405,16,18:42:56.721,DBG] "NO LEADER" found in READ ONLY LEADERLESS state, making further attempts.
2023-10-23T18:42:56Z app[2874deea62d798] lhr [info][  405,16,18:42:56.962,INF] Looks like node ["192.168.21.2:2113/cjxxxlg.mesdb.eventstore.cloud"] is DEAD (Gossip send failed).
2023-10-23T18:42:56Z app[2874deea62d798] lhr [info][  405,16,18:42:56.963,INF] CLUSTER HAS CHANGED "gossip send failed to [192.168.21.2:2113/cjxxxlg.mesdb.eventstore.cloud]"
2023-10-23T18:42:56Z app[2874deea62d798] lhr [info]Old:
2023-10-23T18:42:56Z app[2874deea62d798] lhr [info]["Priority: 0 VND {aade97f6-857f-4bdd-9207-c65c30c6d2d4} <LIVE> [ReadOnlyLeaderless, Unspecified/172.19.5.10:1112, n/a, Unspecified/:::1113, n/a, Unspecified/:::2113, (ADVERTISED: HTTP::0, TCP::0), Version: 23.6.0.0] 22271/22423/22423/E15@21623:{52ef371f-4eaa-41a8-969c-24c746670ba7} | 2023-10-23 18:42:56.953", "MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 192.168.21.4:2113/cjxxxlg.mesdb.eventstore.cloud] | 2023-10-23 18:42:56.720", "MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 192.168.21.3:2113/cjxxxlg.mesdb.eventstore.cloud] | 2023-10-23 18:42:56.720", "MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 192.168.21.2:2113/cjxxxlg.mesdb.eventstore.cloud] | 2023-10-23 18:42:56.525"]
2023-10-23T18:42:56Z app[2874deea62d798] lhr [info]New:
2023-10-23T18:42:56Z app[2874deea62d798] lhr [info]["Priority: 0 VND {aade97f6-857f-4bdd-9207-c65c30c6d2d4} <LIVE> [ReadOnlyLeaderless, Unspecified/172.19.5.10:1112, n/a, Unspecified/:::1113, n/a, Unspecified/:::2113, (ADVERTISED: HTTP::0, TCP::0), Version: 23.6.0.0] 22271/22423/22423/E15@21623:{52ef371f-4eaa-41a8-969c-24c746670ba7} | 2023-10-23 18:42:56.953", "MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 192.168.21.4:2113/cjxxxlg.mesdb.eventstore.cloud] | 2023-10-23 18:42:56.720", "MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 192.168.21.3:2113/cjxxxlg.mesdb.eventstore.cloud] | 2023-10-23 18:42:56.720", "MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 192.168.21.2:2113/cjxxxlg.mesdb.eventstore.cloud] | 2023-10-23 18:42:56.962"]
2023-10-23T18:42:56Z app[2874deea62d798] lhr [info][  405,16,18:42:56.963,DBG] "NO LEADER" found in READ ONLY LEADERLESS state, making further attempts.
2023-10-23T18:43:01Z app[2874deea62d798] lhr [info][  405, 6,18:43:01.033,DBG] Segments count: 1, buffers count: 20, should be when full: 20

Here is the config I’m using:

  EVENTSTORE_CLUSTER_SIZE=3
  EVENTSTORE_READ_ONLY_REPLICA="true"
  EVENTSTORE_CLUSTER_DNS="cjxxxlg.mesdb.eventstore.cloud"
  EVENTSTORE_RUN_PROJECTIONS="System"
  EVENTSTORE_START_STANDARD_PROJECTIONS=true
  EVENTSTORE_EXT_TCP_PORT=1113
  EVENTSTORE_HTTP_PORT=2113
  EVENTSTORE_EXT_IP="::"
  EVENTSTORE_INSECURE=true
  EVENTSTORE_ENABLE_EXTERNAL_TCP=false
  EVENTSTORE_ENABLE_ATOM_PUB_OVER_HTTP=true
  EVENTSTORE_LOG="/data/eventstore/log"
  EVENTSTORE_DB="/data/eventstore/db"

Following a conversation on the community discord the assumption is that something is missing on the security side. Maybe they need to share root certificate file?

Any tips and hints are appreciated!

Best,
Yevhenii

2

Unfortunately, it won’t work due to the fact that cluster members authorise each other with X.509 certificate (the same one used for TLS), and you don’t have that certificate on the RoR node. We will be working on bringing RoRs to Cloud, but it will only realistically happen in Q1 next year.

1 Like
3

That makes sense, thanks!