High-frequency Trading (HFT) using EventStoreDB

Hi All,

I’m trying to setup HFT platform for crypto derivatives. Basically i found @Greg_Young1 videos and couple of presentation wayback almost 10 years ago discussing about HFT with Eric Evans.

Can we use EventStoreDB as a event storage for HFT? We also have some research about Aeron. But looks like that one only for transport.

We got inspired by LMAX architecture. We also would love to use .NET stack if possible and do C/C++ if necessary.

Any thought on this one?

Cheers

I’ve got reply from @Greg_Young1 on twitter. In case anyone interested with this one.

Funnily enough, we run a crypto currency system using eventstore, but I suspect no where near the volume you are thinking about.

Could you share more about that? I would love to learn.

Cheers

Sure thing.
Is there anything specific you would like to know.
Our system is a closed loop, where a number of Organisations can float their own crypto currency, and the users of the system can trade between these different currencies, buy and selling in their own currencies ($, £ etc)

That’s nice!

So it some kind of closed crypto exchange?

I would like to know how to do projection from multiple of time-series based on time? Is there a way to do this in EventStoreDB ? Something like as-of in pandas but in realtime…

Cheers

Yeah close crypto exchange is a good description of the system.
As for the structure of our system, the buys and sells are registered immediately, and we have a subscriber listening for these events (our “trading service”) which does the matching, and once a suitable match is found, a “trade” is created. From there, we have a settlement service listening for “trades” that works out where the funds are being moved.

So central to the way this system works, is our Trade service, listening on customers Buys and Sells.

Nice!

How about your domain model, is that publishing event like in memory bus or it put the event in collections and the repository collect all the events and store it? Like @alexey.zimarev wrote in this blog post

I had to go lok at the code again to remind myself of how this part works.
We have a projection susbcriber to the Orders being raised.
Each projection has a partition per “Product” keeping track of of open orders, which keeps the requestsin datetime order.
So, simply put, the projection is the pending orders, which gets read by another service, forcing the system to look at the oldest orders first when finding a match.

Right, it’s clean separation of concern. Thanks that’s useful

Partitioning projections is a good strategy to handle the load in parallel, aiming for casual consistency in the scope of a single product, where the causality implies the events order.

When the write side cannot afford rehydrating the entity from all the events (even with snapshots), people tend to use in-memory, in-proc entity state updates (they happen anyway). LMAX ended up doing something similar. But it applies some advanced routing to the event delivery system if the system needs to scale horizontally. Applying patterns that have such a routing built-in (like Actor Model) could simplify the implementation. Of course, it comes with the cost of bringing the framework to implement the chosen pattern as a dependency.

1 Like

What Greg wrote makes total sense. It all depends on what “high-frequency” means in your particular case. Some say it is 3000 events per second, some expect 3 or even 30 millions per second. Persistence is a huge performance killer, of course. But trading has two parts - figuring out the trade and executing it. The first part doesn’t really need any persistence on the ingress side. Execution, on the other hand, needs strong consistency, at least within the scope of some limited boundary (portfolio, f.ex).

When looking through Martin Thompson talks I stumbled upon https://github.com/real-logic/Aeron and it is built with performance in mind, so it might be worth looking at it, at least for inspiration.

1 Like

Yes, I currently trying to setup meeting with Adaptive as Martin Thompson only handle Java, C/C++ basically.

They create Aeron .NET that handle Archieving and Clustering. Trying it right now…

Yeah, I’ve seen it https://github.com/AdaptiveConsulting/Aeron.NET
The project seems to be actively maintained.

Ha, I remember that I’ve seen the Adaptive logo somewhere, and definitely heard of their ReactiveTrader demo app. It uses EventStoreDB btw.

Yeah, i see the ReactiveTrader from Greg @Greg_Young1 tweets long time ago