hi, I’m getting “subscription never got confirmation from server” when the connection I’m using to create the subscription has been made to a slave node, and then it keeps trying until the retry limit is reached. The subscription is successfully created when connected to the master node. Has anyone experienced this ?
Thanks,
Javier.
which type of subscription?
maybe a verbose client log?
Persistent subscription.
I’m using the .net client, just noticed that setting PerformOnAnyNode (_requireMaster = false) in the connection I’m using to create the subscription allows the subscription to connect to a slave node.
Well yes 
RequireMaster requires it to be done on the master.
That said creating a persistent subscription on a slave should require
master. What will happen is it will be created on the master anyways
(the slave will forward it to the master) persistent subscriptions are
run on the master.
Well yes, that sounds about right :), thanks a lot for your help, awesome piece of software.
I ran a couple of more tests and it looks like the persistent subscription sometimes fails to reconnect or get created with a slave node connection (even though there’s a master node up), here’s a sample log of what I’m seeing :
[11,11:07:03.214,ERROR] EventStoreConnection ‘ES-472ff0e9-7dee-41be-8d9b-f9d46790c467’: subscription never got confirmation from server.
UTC now: 11:07:03.214, operation: Subscription ConnectToPersistentSubscriptionOperation (eb3b16d5-68ee-4455-b470-8eeb11406fcd): EventStore.ClientAPI.ClientOpera
tions.ConnectToPersistentSubscriptionOperation, is subscribed: False, retry count: 0, created: 11:06:55.604, last updated: 11:06:55.966.
[11,11:07:03.218,DEBUG] EventStoreConnection ‘ES-472ff0e9-7dee-41be-8d9b-f9d46790c467’: RemoveSubscription Subscription ConnectToPersistentSubscriptionOperation
(eb3b16d5-68ee-4455-b470-8eeb11406fcd): EventStore.ClientAPI.ClientOperations.ConnectToPersistentSubscriptionOperation, is subscribed: False, retry count: 0, c
reated: 11:06:55.604, last updated: 11:06:55.966, result True…
[11,11:07:03.221,DEBUG] EventStoreConnection ‘ES-472ff0e9-7dee-41be-8d9b-f9d46790c467’: retrying subscription Subscription ConnectToPersistentSubscriptionOperat
ion (eb3b16d5-68ee-4455-b470-8eeb11406fcd): EventStore.ClientAPI.ClientOperations.ConnectToPersistentSubscriptionOperation, is subscribed: False, retry count: 0
, created: 11:06:55.604, last updated: 11:06:55.966…
[11,11:07:03.223,DEBUG] EventStoreConnection ‘ES-472ff0e9-7dee-41be-8d9b-f9d46790c467’: StartSubscription SUBSCRIBING Subscription ConnectToPersistentSubscripti
onOperation (546aa423-678c-409f-aaba-e2aa43ce1818): EventStore.ClientAPI.ClientOperations.ConnectToPersistentSubscriptionOperation, is subscribed: False, retry
count: 1, created: 11:06:55.604, last updated: 11:07:03.223…
[17,11:07:04.041,DEBUG] EventStoreConnection ‘ES-472ff0e9-7dee-41be-8d9b-f9d46790c467’: enqueueing message EventStore.ClientAPI.Internal.HandleTcpPackageMessage
…
[04,11:07:04.042,DEBUG] EventStoreConnection ‘ES-472ff0e9-7dee-41be-8d9b-f9d46790c467’: HandleTcpPackage connId 7700e20a-3ff8-413f-9175-793dcf44c9e5, package He
artbeatResponseCommand, 4b97daa8-156f-416c-84d9-33f0de097b49…
[17,11:07:06.527,DEBUG] EventStoreConnection ‘ES-472ff0e9-7dee-41be-8d9b-f9d46790c467’: enqueueing message EventStore.ClientAPI.Internal.HandleTcpPackageMessage
…
[13,11:07:06.534,DEBUG] EventStoreConnection ‘ES-472ff0e9-7dee-41be-8d9b-f9d46790c467’: HandleTcpPackage connId 7700e20a-3ff8-413f-9175-793dcf44c9e5, package He
artbeatResponseCommand, 433e526b-5fb3-454a-9074-19c267e6de49…
[14,11:07:09.003,DEBUG] EventStoreConnection ‘ES-472ff0e9-7dee-41be-8d9b-f9d46790c467’: enqueueing message EventStore.ClientAPI.Internal.HandleTcpPackageMessage
…
[04,11:07:09.005,DEBUG] EventStoreConnection ‘ES-472ff0e9-7dee-41be-8d9b-f9d46790c467’: HandleTcpPackage connId 7700e20a-3ff8-413f-9175-793dcf44c9e5, package He
artbeatResponseCommand, 3574f3b0-19e1-4bae-ac46-cbd34e985222…
Can that slave talk to the master? Again persistent connections run on
the master always. I am not sure why you are trying to run with slave?
I have a cluster with 2 nodes, there’s one connection initially established with master, the subscription is created using this connection. Then, I bring down master, and the subscription tries to reconnect as expected. Then I bring master back up. In this scenario, the subscription is able to reconnect intermittently. I’m using the same connection all the time, I might need to create a new connection if the subscription reaches its reconnection limit.
You realize a cluster with 2 nodes has lower availability than a single node?
even having a cluster with 3 nodes, if I kill them all and bring them back up, the subscription sometimes fails to reconnect.
What are your connection settings? And did it call your subscription
dropped handler?
Note there is a handler you pass in that gets called when the
subscription is dropped.
Once your subscription is dropped you need to reconnect the subscription.
yes, and I’m reconnecting the subscription, see https://github.com/javier-troconis-q2/ddd/blob/master/src/infra/PersistentSubscription.cs
in fact, sometimes it manages to reconnect, sometimes it keeps trying forever.
Is the node its connecting to actually up yet? There is a time during
start up where its not actually up yet.
Note:
[11,11:07:03.214,ERROR] EventStoreConnection
'ES-472ff0e9-7dee-41be-8d9b-f9d46790c467': subscription never got
confirmation from server.
yep, the 3 nodes that form the cluster are fully up. After trying to reconnect the subscription 10 times, then I proceed to recreate it and repeat the process.
Are you seeing anything in server logs? Basically there is no response
coming back from server.
I tried to reproduce this here but cannot. When I have a node and shut
it down then bring it back the connecttosubscription works. ... Its
also important to note this is not "reconnecting" this is a new
connection to the subscription.
Try starting with a full cluster, kill 2, and leave 1 up. Then, bring back the other 2, you’ll see that the subscription gets stuck in “subscription never got confirmation from server”.