Building an application release api/datastore

[bryan.oliver]

Hi friends!

I’m working on building an application in Golang with Eventstore. This application would be used in the context of a large company that has many microservice teams. Each team, when they create a new release of their microservice, will generate a series of events corresponding to that release. My thinking, to keep streams small, is that each release constitutes a new stream. For example, you might have one stream like so (assume that the team’s application is named di-application):

Stream name: di-application:0.0.1 (stream is created on the TagCreated event, which has an expected revision of NoStream in it’s append options)

TagCreated
ReleasedToProd
ProdClusterDeployment (event data would contain where, like clustername: us-east-2)
ProdClusterDeployment (… us-east-3)
ProdClusterDeployment (… eu-central-1)
ProdClusterDeployment (… etc.)

Building out this logic was fairly straightforward, I was able to create the functions for these events in a couple of days. I’m now working on wrapping these events in an API (so they can be called from the upstream systems that generate these events). All of this has been really fun so far, and I’m enjoying it.

Now, my question.

I’ll need also to provide apis that allow users (and systems) to find things like:

• show me the latest version of di-application that went to prod (i.e. the latest stream of format di-application:* that has a ReleasedToProd event)
• for all applications, show me the current version that is released to prod (i.e. for all streams, assuming the application pattern is <app_name:version>, find the latest <app_name:version> for each <app_name> that has a ReleasedToProd event)
• show me the last version of di-application that went to us-east-3

From what I’ve read and understood so far, it seems to me like the best way to accomplish this sort of thing is to do it with a REST API + database like Postgres, or an indexable user query focused search system (like Lucene, Elastic, etc). And that trying to build this sort of API in front of EventStore isn’t really a good idea (or what it’s meant to do), I should instead think of EventStore as my source of truth and provide convenience apis that help teams/machines find complex queries and patterns with an API+Engine that is intended for that purpose.

[alexey.zimarev]

I would challenge your schema design a little. I think you can get better results by keeping one stream per application/service. It will allow you:

• Keeping the whole history of service versions and deployments in one place
• Easily query the last service version by reading the stream backwards
• Query the deployments for a service by reading the stream backwards

You’d still to need to have a query model for keeping the list of all services and their versions, etc. For that, you can use any other database, which allows you to run indexed queries. Alternatively, you can create a custom projection with state, where you can use so-called “bi-state” with partitioning (per service in your case). The state object, however, might become too large when the number of services grows.

Unless you are interested in the whole history of service versions, you can still have the stream length limited by setting the max count or max age on those streams and run scavenging regularly.