Hello,
When I supply all 3 callbacks to connection.SubscribeToStreamFrom(), are there any multi-threaded guarantees?
E.g., can EventAppeared be called while another EventAppeared is executing?
Can any 2 of the callbacks be called simultaneously?
Thanks,
Igor.
Hi Igor,
EventAppeared for the each subscribe request is guaranteed to be called sequentially (i.e. only when the previous one completes), but subscriptions are not bound to a thread (i.e. thread may differ on the next call).
EventAppeared from different subscribe requests are completely independent. There is no optimizations for subscriptions to the same stream.
So, 2 callbacks can be called on simultaneously. Callbacks are called on the thread pool.
-yuriy
(somewhat) Related question: When do heartbeat timeouts occur? Is it when the EventAppeared takes too long to process? Or all callbacks?
/Peter
Heartbeat timeouts are on the tcp connection to the store in client API. If you are too slow etc there are various queues along the way that if they reach a maximum size will drop your live subscription and return to polling
Heartbeat timeouts are on the tcp connection to the store in client API. If you are too slow etc there are various queues along the way that if they >reach a maximum size will drop your live subscription and return to polling
Too slow in the callbacks? Or could I get a timeout due to eventstore client not getting enough cpu-time to actually call my code? (have been trying to balance out the number of subscriptions, seems when I have too many I get heartbeat timeouts frequently, even though all I do is drop the event in a queue of my own)
/Peter
How many subscriptions are a lot? Also what is your heartbeat set to? Remember things like go pauses can also go into this
How many subscriptions are a lot?
Think I had about 20 at the most (high priority projections had their own). Now I’m done to 3, and it seems like I’m rid of the timeouts (but with higher latency due to bigger transactions).
Note that the write side runs in the same process, so high load means a lot of activity, not just projections.
Also what is your heartbeat set to?
Tried from 10 sec to 5 minutes, didn’t make much difference under heavy load, the timeouts came either way
Remember things like go pauses can also go into this
?
/Peter
What’s heavy load? I can try to reproduce things here
Don’t know if I have any meaningful metric , I’ll setup an events per seconds counter or similar and get back…
/Peter
Understood. So, EventAppeared is not re-entrant.
What about concurrency between EventAppeared and LiveProcessingStarted? Can they be called at the same time?
It would be nice to have these 2 callbacks serialized.
I’m seeing what looks like the same problem.
If I try to establish multiple catch-up subscriptions in a tight loop, up to around 15 works fine.
More than that, and I start seeing this:
no callbacks are called for a long time
connection times out
client reconnects
EventAppeared gets fired for some number of subscriptions
if too many subscriptions, client disconnects again
client reconnects
another batch of subscriptions get EventAppeared
etc
Also, the delay before client detects the timeout seems to be longer the more subscriptions I try to establish.
Can you provide a standalone program showing this issue?
Id have to look at this to see if they are serialized. My guess is no but it could be.
We were seeing the same issue here. Seems to be related to the firewall in debian detecting it as a DDOS of some sort and quarantines the client for a period of time.
-Svein Arne
Wrong thread replied to i dont follow