I'm currently doing a migration of existing data, and noticed a few
things that I'd like to ask you about. The input is CSV files, so I
can generate events at a very high rate. However, if I do this then
quite quickly the subscriber will get a disconnect, and eventstore
server log shows heartbeat timeout. Could this be because
SubscriptionObserver.onEvent takes a long time? Specifically, in that
method it puts the event on a bounded size queue and the db updater
gets events from there in batches. During import that queue can reach
maximum size, and hence onEvent blocks as a whole. Can that cause
timeouts for the connection, because it cannot process other events
(like heartbeat), or are those done separately?
Which brings me to the general question: what is the correct way to
indicate back pressure to the subscription? Currently onEvent simply
blocks, but maybe there's a nicer way to do it?
To get around the above I added a throttle to the CSV importer, to slow it down.
Another thing I noticed is that during this import I get thousands of
calls to onLiveProcessingStart in the subscriber. Is this normal? What
do people usually do in that callback?
Thanks for any help in understanding the above better!