Hi
We are seeing a lot of Subscription dropped exceptions in the .net tcp client at the moment. Drop reason: ProcessingQueueOverflow.
Can you share some details on this specific drop reason?
Thank you!
What version of the client are you using and what are your catch up subscription settings set to?
It normally means there are more events waiting to be processed than
the queue is allowed to hold (eg back pressure)
We are using version 3.6.3.0.
Which settings are you thinking of?
The only non standard settings we have at the moment:
settings.KeepReconnecting();
settings.FailOnNoServerResponse();
Thanks. That definitely makes sense in this scenario.
In particular this tends to happen when you are on a live subscription
(eg the server is pushing messages) and the consumer is not keeping up
with the messages being pushed from the server, at some point we have
to say "this is enough" or else we would just eat up all memory in the
client.
There are a few ways of dealing with this.
1) manage the queue on your own. when you receive from the
subscription queue the message and have something reading from the
queue asynchronously. This of course runs the risk of blowing up
memory on your own so you would need to put your own policies
2) manage the reconnect on subscription dropped. This drops you back
into polling mode.
3) (putting up an issue for it) change the default queue size.
We are actually doing 2) at the moment.
Our streams are wrapped in IObservables and we have an rx operator that takes care of resubscribing if the subscription is dropped (not in the case of SubscriptionDropReason.UserInitiated).
Rx.net is a pure push model, no backpressure interfaces as in the case of rx. java. But it’s pretty easy to add an implicit queue with an operator… But I think I prefer the current solution that goes into polling mode as you mentioned 
Not strictly relate to the problem of falling behind a live subscription, but in case anyone’s interested:
We’re using Java and Project Reactor (Rx implementation) that does support backpressure - but not with operations such as buffer. We have a custom implementation of top of this that fills up a local buffer, passing on whole chunks of events to process (for performance reasons, so that we can use things like a single DB transaction).
In the live case, as long as the subscriber is faster than the updates, new events will be processed without delays (with a batch containing a single event). When falling behind slightly, the batch will automatically grow allowing it to process events at a higher throughput. If falling well behind, it will switch to catching up.
Cheers,
Kristian
+1 this is a common strategy especially for projections that tend to
be IO bound. ES actually does this same strategy internally.