Hi
We are hosting eventstore now on vmware and some form of network storage. I get 200mb/s sequential read performance on the servers.
But eventstore has big problems with performance. I now can’t activate the two last system projections as eventstore then becomes so slow that i get timeouts. Read can take 1 min.
When reading from a by category stream i max out at 2.7mb/s. Its very very slow. I know this isn’t the recommended setup. Is there anything we can do?
16gm ram and 4 cpus, ubuntu 18.04. DB directory is 580gb, and scavenged every month. Takes 8 days to scavenge.
What will the preformance be on cloud solutions?
It could be a lot of things and normally NAS is not the best storage for databases.
For example, reads from category streams are in no way sequential. You effectively read link events and then resolve them to physical events, which are scattered around multiple chunk files, using the index.
For Event Store Cloud we only use SSD disks with low read seek latency and it works just fine. Surely, when the dataset grows and events get older, reading any stream, which contains old events, becomes slower, but still acceptable.
Btw, the only sequential read in ESDB is read/subscription from $all.
I have now tested locally on two nwme disks. Which is much faster and not prone to hangs for extended periods of time.
When i start to write i get to a point were it stops for 20-60s. See log below print time in ms every 1000 events. Is this normal?
Written 65000 events in 3
Written 66000 events in 45
Written 67000 events in 2
Written 68000 events in 2
Written 69000 events in 35277
Written 70000 events in 5
Written 71000 events in 2
Written 72000 events in 3
We have disabled two system projections in production $by_event_type and $stream_by_category. These restart when the master node restarts. Is it possible to specify that these should never start. We don’t need the data and they makes the eventstore cluster unusable. We get lots of timeouts when they are active. (they are far behind) Read and write operations takes longer then a minute.
Just a note on scavenging. Not sure which “v5” you have, but 8 hours scavenge time means you have lots of deleted events. Disable writing stats to the database is you are on 5.0.8+ and you won’t need to scavenge them https://developers.eventstore.com/server/v20/server/diagnostics/stats.html#write-stats-to-database
We are on 5.0.8 and Last scavenge took 2021-03-01 03:07 -> 2021-03-05 11:09, 4 days give or take.
Takes 1-3 min for each 256mb chunk.
We have not enabled statistics saving.
We would love to move to v20 if the client libararies works with v20. Then we can also upgrade to grpc. But as esjc and Eventstore.JVM clients don’t work with tls on v20 its not a possible upgrade path yet.
What I meant is that saving stats to the database is enabled by default in v5. It is directly mentioned in the docs. You have to explicitly disable it and you will get a lot less events to scavenge.
https://developers.eventstore.com/server/v5/server/diagnostics/stats.html Says its disablend by default. But i may be mistaken.
The time it takes to scavenge is the same for chuncks that have 0 bytes release as those that have 40mb released.
I see its enabled by default in the log when it start. Disabling now.
I think it is disabled for 5.0.9, but it’s enabled by default in all the previous versions.
Concerning the performance topic.
The number you’re getting for a sequential read benchmark has very little to do with the read and write speed for EventStoreDB.
Consider the following:
- Writes are indeed sequential, as we append events to the active chunk. If your writes are slow, you might want to look at measuring the write performance of your disks, not reads.
- Each system projection produces (1) link events (2) checkpoint events. All of those are written to the database in addition to the original event. That’s what we call “write amplification” and that’s why we suggest to consider server-side filtering (v20) instead of using projected streams, like the category stream.
- When you read from the category stream, you don’t read events. Physically, the server gets a link event and resolves it to the original event. Original events could be scattered across many different chunks. Therefore, the speed of sequential reads from a large file is not relevant here. You need to check the random access speed of your disks.
- Also, when resolving the original event, the server will read the stream metadata, to ensure that you are not getting a deleted event. Links are not deleted when you delete the original event, as truncation doesn’t work for events in the middle of the stream. If you have links to many streams in the category stream, all those reads are again random access reads from many chunks.
Both points about the reads aren’t relevant when you subscribe to $all, as in that case you indeed will be reading from a single file, sequentially, until the end of the chunk. That’s one more reason to prefer reading from $all and use filters, instead of using category or event type projections.
1 Like
We also changed from cfq to noop(none) io scheduler. That helped a lot. Thanks for the help.