EventStore persistent Subscription problems

EventStoreDB
1

Hi everyone,

I’m facing a weird situation, my persistent subscriptions hangs or slow down sometimes (from ~300 e/s to ~5 e/s)

My configuration hosted on GCP is:

CPU: [email protected]

RAM: 16GB

Disk: 1TB HDD

My database has been 3 times scavenged during last month, it is composed of ~100GB of chunks (~35M events, ~10M streams), index size is 5.2GB

I have also several projections enabled:

  • $by_category

  • ByDate (from any, stateless): which links 1/15 event to new stream per minute (e.g. Date-20180917-1304)

  • BySession (from $ce-Message, stateless): which links 1/300 event to a new stream

  • ByPeer (from $ce-Message, stateless): which links 2/3 event to existing streams (~6M)

  • Tracking (from $ce-Message, stateless): which emit 1/15 event to a new stream

  • Stats (from: $ce-Message, $ce-Session, $ce-Queue, statefull): which emit 1 event every minute

I tried to run database on local SSD, I joined cleaned logs and a screenshot from this plateform showing some errors

Any idea to solve my problem ?

Screenshot from 2018-09-17 13-24-38.png

eventstore-logs.txt (48.3 KB)

2

I am not seeing anything about persistent subscriptions in the log? Perhaps you mean something else?

As for why projections etc are failing here is a read operation taking 21 seconds:

[09866,07,09:44:34.317] ---!!! VERY SLOW QUEUE MSG [StorageReaderQueue #1]: ReadStreamEventsForward - 21813ms. Q: 3/104.
3

It’s true, I forgot it, now it’s attached.

I made more test with SSD, and the following configuration:

CommitTimeoutMs: 30000

MinFlushDelayMs: 5

ProjectionThreads: 10

It seems to be better, but I’ve not yet tested with heavy load.

eventstore-logs.txt (83.9 KB)