Could Someone Give me Advice for Scaling Event Store in a Microservices Architecture?

EventStoreDB
1

Hello there,

I have been working with Event Store as part of a microservices architecture in a project where event sourcing plays a central role. As our application grows; we are facing challenges with scaling and managing data efficiently across services. I was hoping to get some insights from the community on best practices and potential pitfalls when scaling Event Store in a distributed environment.

We are using an Event Store cluster; but I am curious about how others have handled scaling it to support larger loads. Are there any specific configurations or tools you recommend to ensure high availability and fault tolerance? :thinking:

How do you approach partitioning data across multiple services? :thinking: Are there any strategies to minimize cross-service communication when querying event data? :thinking: We are considering using categories or streams to segregate events.

Also; I have gone through this post; https://discuss.eventstore.com/t/catchup-subscriptions-in-a-scalable-microservices-environment-ccsp/ which definitely helped me out a lot.

We are using projections to aggregate event streams; but we are unsure how to manage their performance as data grows. Have others encountered challenges with projections at scale; and how do you manage them efficiently? :thinking:

Thanks in advance for your help and assistance. :innocent:

2

Without going into details from specific customers:

  • we have people that have accumulated 10s of TB & billions of events using a single store for all their services.

  • the main point of contention is usually : using the same store for data and integration

    • it’s ok to use ESDB as a way to hold integrations streams
    • it is ok to use 1 ESDB for data from multiple different services
      • caveat below applies
    • it is strongly recommended to separate the data from integration messages in completely sperate ESDB clusters

  • When using a single store for the data make sure to have strong prescriptive stream & event naming convention to avoid services append or read from each other streams