Disclaimer: I am in no way trying to use the data I’ve collected to promote or disparage any of the options tested. I am primarily concerned with understanding and diagnosising any performance issues with my current configuration.
After experiencing slower than expected write speeds with EventStore I setup some performance benchmarks to compare it to other event stores and see if it was my machine or something about how I was configuring/using event store. You can see the full benchmark implementation in this repo, and you can also run the benchmarks for yourself.
The benchmarks test a couple different functions, but the primary ones are sequential appends and batch appends of varying numbers of events. For the moment I’ve only compared EventStore against MartenDB since that’s the other main event store implementation I’m considering.
Here are the results I’m seeing:
BenchmarkDotNet v0.13.11, Windows 11 (10.0.22631.2861/23H2/2023Update/SunValley3)
13th Gen Intel Core i7-13700K, 1 CPU, 24 logical and 16 physical cores
.NET SDK 8.0.100
[Host] : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX2
DefaultJob : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX2
| Method |
NumEvents |
EventStorage |
Mean |
Error |
StdDev |
Median |
| AppendSequential |
1 |
EventStoreDb |
14,908.5 μs |
1,412.22 μs |
4,163.95 μs |
12,436.0 μs |
| AppendSequential |
1 |
MartenDb |
1,136.5 μs |
21.90 μs |
18.29 μs |
1,142.6 μs |
| AppendBatch |
1 |
EventStoreDb |
16,919.3 μs |
1,558.40 μs |
4,594.97 μs |
17,931.6 μs |
| AppendBatch |
1 |
MartenDb |
1,134.9 μs |
21.77 μs |
19.30 μs |
1,139.1 μs |
| ReadFromStart |
1 |
EventStoreDb |
511.3 μs |
6.11 μs |
5.72 μs |
509.4 μs |
| ReadFromStart |
1 |
MartenDb |
415.9 μs |
6.92 μs |
5.78 μs |
414.8 μs |
| AppendSequential |
100 |
EventStoreDb |
1,210,249.7 μs |
123,018.81 μs |
362,723.74 μs |
997,167.0 μs |
| AppendSequential |
100 |
MartenDb |
146,864.1 μs |
2,758.25 μs |
2,951.29 μs |
146,364.9 μs |
| AppendBatch |
100 |
EventStoreDb |
16,471.5 μs |
324.37 μs |
475.46 μs |
16,414.6 μs |
| AppendBatch |
100 |
MartenDb |
3,604.7 μs |
42.88 μs |
40.11 μs |
3,593.1 μs |
| ReadFromStart |
100 |
EventStoreDb |
1,125.9 μs |
45.53 μs |
132.08 μs |
1,093.7 μs |
| ReadFromStart |
100 |
MartenDb |
720.6 μs |
24.41 μs |
71.22 μs |
699.9 μs |
| AppendSequential |
1000 |
EventStoreDb |
13,092,525.6 μs |
909,273.26 μs |
2,681,012.72 μs |
12,447,555.2 μs |
| AppendSequential |
1000 |
MartenDb |
1,472,928.1 μs |
26,174.24 μs |
23,202.78 μs |
1,480,464.4 μs |
| AppendBatch |
1000 |
EventStoreDb |
39,707.5 μs |
941.09 μs |
2,760.06 μs |
39,777.7 μs |
| AppendBatch |
1000 |
MartenDb |
26,720.9 μs |
256.16 μs |
227.08 μs |
26,653.9 μs |
| ReadFromStart |
1000 |
EventStoreDb |
4,037.8 μs |
79.60 μs |
189.19 μs |
4,016.1 μs |
| ReadFromStart |
1000 |
MartenDb |
2,716.8 μs |
151.67 μs |
440.03 μs |
2,652.9 μs |
I was pretty shocked when I saw these results. In the single event append EventStore was in the neighborhood of 13-14x slower than Marten.
The setup is pretty barebones. I’m running the selected event store using testcontainers and docker and I’m not doing any additional configuration beyond what the testcontainer is defaulted to. You can check out the implementations for each event store, but they’re pretty standard usage from what I can tell.
Can anyone help me understand where this poor performance may be coming from?
I don’t know for sure if the ESDB code is correct (have my doubts), but what does these benchmarks show? How fast can one append events to an empty stream, starting from an empty database? 1, 100, 1000 events? Reading 1, 100, 1000 events?
A typical scenario, as I see it, is read a stream and append one event. I would not expect to read more than a couple of hundred events, otherwise I’d do snapshot + events. I would expect to add 1, 2, maybe 5 events for a single transaction. Not 100, not 1000.
Another thing is that Benchmark.NET is a benchmarking framework. You don’t want to benchmark a database, you want to load-test it. There are other tools for that, like NBomber.
So, if you want to make a choice of an event store when you plan to build something for production, you probably would want:
- Create a system stub with simulated command processing but real persistence, using the necessary abstractions that you will need anyway
- Discover your system operational expectations (like number of streams, number of events per stream, how many ops you expect per sec/min/hour, how many subscriptions you will use, etc)
- Build stubs for subscriptions too
- Start calling the command processor with your expected ingress level, then try to double, then do like 10 fold. Measure time needed to execute one transaction.
- Ensure your subs are on par with ingress. Measure them.
Only then you would know. But you don’t stop there.
You weight other things:
- Maintenance costs and TCO
- Licensing
- Support
- Future plans (both yours and the product’s you decide to use)
That’s what I was doing when I worked as a software architect and I had to make decisions about infrastructure.
Btw, when I was measuring ESDB performance a few years back, I built a simulation tool that implemented the go-to scenario (read stream, append event) with configurable number of streams and number of events per stream. Then, I used an in-memory message bus to simulate concurrent load. As I have to run the load test on a database, so I used the closed system load testing.
Then, I instrumented the simulation with metrics and traces, deployed it in Kubernetes and ran it for a week. After that, I got a pretty good idea how the actual system would run in production for a prolonged period of time. Of course, the simulation was loaded way more than the system I planned to build because it has no seasonality built-in, but it was good enough load test.
Thanks for the response Alexey.
To answer your questions on what this benchmark is testing:
- AppendSequential = Appends NumEvents to an empty stream sequentially (wait for each event to be appended before appending the next)
- AppendBatch = Appends NumEvents to an empty stream in a single batch (passes all events to a single Append call)
- ReadFromStart = Reads all events starting from the beginning of a stream with NumEvents
I should be more transparent about my intentions here since it might help explain some of my rationale with how I’ve been testing this.
I’m working on a library that will extend the event sourcing built-in to Orleans to allow using actual event stores such as ESDB. You’re welcome to check it out here if you’re interested, and here’s a link to some information about Orleans event sourcing if you’re not familiar.
The event sourcing implementation that is currently provided with Orleans does not rely on an event store, but instead stores either the current state of the grain in storage (essentially only storing a snapshot) or it stores the entire log of events serialized into a single database row. Both of these implementations have their drawbacks, so I thought I’d pursue an implementation that uses a true event store to store events.
This should explain why I’m doing benchmarking here and not load testing. I don’t have a database instance I’m trying to verify, nor do I have a real use case I need to simulate. I’m merely trying to compare this event storage based approach against the built-in event sourcing mechanisms already provided, and provide some numbers to users of the library to give them an idea of the performance difference between my library and the built-in options.
My initial hypothesis was that this event storage based approach would be at least as fast as the built-in snapshotting approach, and likely much faster than the serialized log-based approach. What I’ve found so far however is that ESDB seems to perform much slower than the other options by a significant margin. I added a Marten implementation as a comparison recently because I wanted to see if it was my implementation that was the problem, or if it was the event store.
EventStore should be installed and not run in a container unless something has changed in the past few years. Running the EventStore docker container is great for local development but not performance.
Also, you should check out this Orleans EventStoreDB repo. I’m not sure who the owner is, but you can use the event sourcing and persistence portions of it. You’ll just want to modify to use a typemapper. I was able to do it and it worked great in a prototype. I do not have that code available in a public repo but can help if you have questions.
hongliyu2002/Orleans.EventStore (github.com)
Hope that helps.
Zach