Performance problems on vmware hosted v5 cluster


We are hosting eventstore now on vmware and some form of network storage. I get 200mb/s sequential read performance on the servers.

But eventstore has big problems with performance. I now can’t activate the two last system projections as eventstore then becomes so slow that i get timeouts. Read can take 1 min.

When reading from a by category stream i max out at 2.7mb/s. Its very very slow. I know this isn’t the recommended setup. Is there anything we can do?
16gm ram and 4 cpus, ubuntu 18.04. DB directory is 580gb, and scavenged every month. Takes 8 days to scavenge.

What will the preformance be on cloud solutions?

It could be a lot of things and normally NAS is not the best storage for databases.

For example, reads from category streams are in no way sequential. You effectively read link events and then resolve them to physical events, which are scattered around multiple chunk files, using the index.

For Event Store Cloud we only use SSD disks with low read seek latency and it works just fine. Surely, when the dataset grows and events get older, reading any stream, which contains old events, becomes slower, but still acceptable.

Btw, the only sequential read in ESDB is read/subscription from $all.

I have now tested locally on two nwme disks. Which is much faster and not prone to hangs for extended periods of time.

When i start to write i get to a point were it stops for 20-60s. See log below print time in ms every 1000 events. Is this normal?
Written 65000 events in 3
Written 66000 events in 45
Written 67000 events in 2
Written 68000 events in 2
Written 69000 events in 35277
Written 70000 events in 5
Written 71000 events in 2
Written 72000 events in 3

We have disabled two system projections in production $by_event_type and $stream_by_category. These restart when the master node restarts. Is it possible to specify that these should never start. We don’t need the data and they makes the eventstore cluster unusable. We get lots of timeouts when they are active. (they are far behind) Read and write operations takes longer then a minute.