Event store for large volumes of events

EventStoreDB
1

We are looking at using the event store to store large volumes of events that are generated by a trading system. There will likely be around a million streams, each containing potentially millions of events. The event store looks like it would be a good solution to persisting this data.

However before we dive to deep into proof of concept, we have a few basic questions:

  • Is the event store suitable for this type of activity, and/or has anyone made use of it for this?
  • As storage cost will be critical for us, how does the data size compare to compressed raw data (zipped csv for example)
  • Can single streams by dumped to an archive, removed, but restored on demand when the data is required?

Any thoughts on this would be much appreciated

2

What is the number of events coming in / second? Millions of streams is no issue.

I would imagine you would want to run multiple stores at this point. In terms of storage you can run on a compressed file system which will give you compression. The format of storage is up to you (unless you want to use projections which requires json). ES itself just sees data as a vector of bytes.

Cheers,

Greg

3

Greg, can you explain why the solution would be to run multiple stores?
I’m too in evaluating for high load, but I want a single cluster, so… what would be the problem/bottleneck with, say, a 100-node cluster?