Hey Hayley-Jean,
I apologize for disappearing; we were forced to put this down for a bit to deal with a couple other higher priority items. We’re still seeing the problem, but only under load, and we only have the one service that generates that kind of load and we’ve shut it down because it was causing the other services to time out as well. Without this service running, things are a bit more stable with only an occasional timeout. We did do a test with the server running on Google cloud and it had the same problems so I don’t think it’s an Azure issue. The last time I looked at the logs, they appeared to be exactly the same. The client logs said they never got a response from the server; the server logs didn’t show any errors but did log out the fact that the client connection was lost. We’ll get you another set of logs as soon as we can pick it back up.
We may just go ahead and just pay for the support to get this resolved because this is obviously going to hinder us going forward. In the meanwhile, if there’s anything I can do to assist with the troubleshooting and help the 24hr latency between replies, let me know. I can grant you read-only repo access if that would help. I can be reached at jstafford at digital air strike dot com (no spaces)
I’ve done similar testing and had decent success in recent times. Some key elements:
-
use KeepReconnecting (I assume that’s obvious/noncontroversial)
-
consider using LimitConcurrentOperationsTo in order to get more predictable behavior (it’s never helped me; I prefer my SetQueueTimeout, see below)
-
consider SetHeartbeatTimeout to > 1.5s (2.5 is recommended e.g. if you’re traversing Regions. I’ve not found resilience to be significantly impacted by this setting IME; YMMV)
-
use FailOnNoSeverResponse - IMO this should be a default, though I can see people disagreeing and it would be breaking. This is more predictable if the connection is lost and you’re doing request processing
-
4.1.1 client / pending PR from me: SetQueueTimeout, ConcurrentQueue https://github.com/EventStore/EventStore/pull/1582
-
4.1.1 client new feature: KeepDiscovering https://github.com/EventStore/EventStore/pull/1578 (think SetMaxDiscoverAttempts(int.MaxValue))
-
4.1.1 client fix for issue that can happen under high load or via a server or client induced heartbeat timeouts: https://github.com/EventStore/EventStore/commit/8a5a5ab2ebfa0e718b13ed3f6f6298f8269e9550
I do not handle Closed / connecting events / objectdisposedexceptions - the OOTB config is such that you should be able to do normal request processing without such arrangements.
This is obviously likely not directly useful to you at this time; I hope this is useful to someone, even as a checklist of things to consider - there is no one size fits all for this unfortunately!
–Ruben