I am using event store cloud.
I have one persistent subscription with 3 consumers and 2 catch up subscription.
I am trying to process around 100k messages by adding them to a command stream and then processing them by using a persistent subscription to all with filtering on event types.
I am using the default settings for the persistent subscription.
Each message is taking about 150 - 450 ms to process.
Each message that gets processed is adding it’s own stream and adding about 10 events.
When doing this after a couple of minutes the node becomes unreachable and the connection to the event store stops working. I get a critical notification in the cloud ui with a message node 0 can not be reached.
Then after awhile the status for the cluster is “down”.
Right now I am in test so I am using a M8 cluster with 1 node but I have tried using a M16 cluster with 3 nodes and I still get the same error it is just that 1 of the 3 nodes becomes unreachable.
I do not know what I am doing wrong and what I need to do to not make the cluster stop working.
If I process 1k events instead of 100k everything works fine.
Thanks for any help I can get.