Problem with subscription getting dropped with a "client buffer too big" exception message

EventStoreDB
#1

Hi

I having problems with a SubscribeToStreamFrom ending with with a dropped subscription. The DropReason is “UserInitiated” (which seems a little weird given the exception–>) and the exception message is “client buffer too big”. Can anyone explain what happens here…?

/Anders

#2

There is a buffer in the subscription, if the client can't keep up and
the buffer is full then the subscription will be dropped.

#3

Could you explained this concept a bit more..?

I Would expect the subscription to just try to keep the buffer full - otherwise how does one consume a stream, where processing of the events is not fast enough to "keep up"?

#4

In historical mode you won't have any issue only in live mode (at
which point the subscription buffer will just fill up forever).

I believe there is an issue up to make it drop back to historical mode
at this point (and I could have sworn it did at some point).

#5

Specifically my problem is, that once a day a system will produce somewhere between 1 and 2 million events - these are written to a stream. Another part of the system subscribes to this stream, and processes the events. The problem seems to occur when these events are being produced, since they are beeing produced faster than they are being processed. As I understand it this will cause the buffer to fill up, and thus the subscription is dropped.

If this is the case, how can I handle the above scenario..?

#6

Handle dropped subscription and recreate? Though I agree that it
should handle this case.

#7

OK - thank you very much!!

By the way as long as the “client buffer too big” exception occurs in these cases - should’nt it have a dropreason other than UserInitiated? This seems counter intuitive…

#8

I take it that I have to track my current position in the subscription myself (as a client) and recreate the subscription from that position when it is dropped?
Alternatively I’m thinking about creating persistent subscriptions and deleting + recreating them from index 0 whenever the service restarts (or from the last index that the client sees in its read cache). I cannot rely on the ES subscription index matching up with my read cache, in case of restored backups, switched databases (during development/debugging) etc.

#9

You always have to track your position on the subscription (that is
where the from field is). You would normally hand back on restart etc.

" I cannot rely on the ES subscription index matching up with my read
cache, in case of restored backups, switched databases (during
development/debugging) etc."

Store the position with your read cache.

Using persistent subscriptions is likely a terrible idea here (lose
ordering assurances etc)