Hi guys,
I'm currently doing a migration of existing data, and noticed a few
things that I'd like to ask you about. The input is CSV files, so I
can generate events at a very high rate. However, if I do this then
quite quickly the subscriber will get a disconnect, and eventstore
server log shows heartbeat timeout. Could this be because
SubscriptionObserver.onEvent takes a long time? Specifically, in that
method it puts the event on a bounded size queue and the db updater
gets events from there in batches. During import that queue can reach
maximum size, and hence onEvent blocks as a whole. Can that cause
timeouts for the connection, because it cannot process other events
(like heartbeat), or are those done separately?
Which brings me to the general question: what is the correct way to
indicate back pressure to the subscription? Currently onEvent simply
blocks, but maybe there's a nicer way to do it?
To get around the above I added a throttle to the CSV importer, to slow it down.
Another thing I noticed is that during this import I get thousands of
calls to onLiveProcessingStart in the subscriber. Is this normal? What
do people usually do in that callback?
Thanks for any help in understanding the above better!
regards, Rickard
"Another thing I noticed is that during this import I get thousands of
calls to onLiveProcessingStart in the subscriber. Is this normal? What
do people usually do in that callback?"
This is when you switch from paging mode to processing a live
subscription. Live subscriptions have back pressure on them which if
too big will then go back to paging. Likely you are going back and
forth between them. Also I am guessing this is with the akka client?
What version are you running?
Hi Greg!
"Another thing I noticed is that during this import I get thousands of
calls to onLiveProcessingStart in the subscriber. Is this normal? What
do people usually do in that callback?"
This is when you switch from paging mode to processing a live
subscription. Live subscriptions have back pressure on them which if
too big will then go back to paging. Likely you are going back and
forth between them. Also I am guessing this is with the akka client?
Yes, this is with the Akka client. How is the back pressure defined?
In my case it is effectively that onEvent becomes slow, indicating
that the db updater has trouble keeping up.
What version are you running?
Just updated to 3.5.0, which is where I saw this. I was getting the
onLiveProcessingStart many many times per second, so really
flip-flopping between the states.
/Rickard
Drop an issue up on the jvm client github. My guess is this is
configurable (but the behaviour with back pressure sounds reasonable
on a slow subscriber). This should not however be causing heartbeat
timeouts in my opinion (heartbeats are only sent when there is no data
on the connection)