We have a couple applications that subscribe to the $all stream with filters that lose their connection every day or two and the SubcscriptionDropped callback is never invoked. The application is running on .NET 5.0 and using EventStore.Client.Grpc v21.2.0. Our EventStore instance in ES Cloud is 20.10, and we have a self-hosted instance of ES on the same version that sits within our vnet.
On our self hosted instance our connections are never dropped. With the same code deployed to our Azure environment connected to ES Cloud, this happens at least 2 or 3 times a week. The only way we’ve been able to resolve has been to restart the application, which reconnects to ES and continues processing from the last checkpoint that has been recorded.
This is what the test subscriber looks like: https://gist.github.com/teeroddesigns/4eb19238ead16c5ff4ed588d4b7014ac
I started this up on Thursday afternoon (Aug 12, 2021) and when I came back this morning and created some new events they were not picked up by the application. Furthermore, there were no Error level logs that I would have expected from the SubscriptionDropped callback.
We have had the same code running against our self hosted environment for well over a year and have never once ran into this issue. We have two hosted ES clusters and this is an issue for both environments. Is there something I’m missing here?