Stream ids and concurrency

EventStoreDB
1

In the examples given in both series of blog posts on projections, streams are categories (“ponies”, “github-commits”, “strategy”), which may then be partitioned into more specific streams. As far as I can tell there are a couple of potential problems with this:

  1. If commands need to know the current state of an aggregate to calculate the outcome of a command then the command service will need some sort of cache of the current state of all the aggregates, to avoid having to read in all events in the category for every command to find the current state.

  2. If the write service is scaled horizontally (i.e. multiple writers) then either writes need to use ExpectedVersion.Any, or otherwise they rely on subscribing to all events from other writers to keep track of the current version.

Some possible solutions to these issues are:

a) Use a separate stream name for each aggregate (projections may still be used to aggregate streams within a category). (1) is then solved because each aggregate can be rebuilt at the time of each command from its own stream. (2) is partly solved as simultaneous commands on the same aggregate are less likely than simultaneous writes on the same category, though it will still sometimes happen unless writers are sharded by aggregate.

b) The reverse of the above, use a stream per category but then use projections to create a stream per aggregate. This mostly solves (1), but not (2).

c) Combine command and query services, and then keep a shared in-memory read model for use by both commands and queries. This solves (1) because commands can use the same cached read view as the queries, and (2) is somewhat solved because the command services are now reusing the same subscriber logic as the query service to keep track of the current version. Though even then simultaneous writes are possible, which will still result in expected version exceptions.

d) Avoid commands that need to know the current state of the aggregate, which can solve (1) - e.g. changing a name could be considered to always succeed, without needing to check the current state of the aggregate.

d) Avoiding multiple writers and only scale query services solves (2), but this may not always be possible or desirable for scalability or reliability reasons.

Is there some other solution? Is there any recommended best practice?

Finally, FWIW I’ve seen multiple people make the same mistake as me and not enabling projections at the command line before trying them out, and then being confused as to why their projection doesn’t work. Maybe the need to use a special command line argument should be made more prominent somehow, perhaps with a message in the UI?

Thanks,

2

If you take approach a (separate stream name per aggregate), and enable system projections on the server then you can make use of the $by_category projection, which will create a stream per category (a category is everything before the final - in a stream name, e.g. stream ‘aggregate-a’ would have a category of ‘aggregate’, while stream ‘github-commits-a’ would have a category of ‘github-commits’).

If possible, being able to take approach d will make your system much simpler, however in some cases it won’t be possible.

3

I feel like I must be missing some subtlety here, due to the multiple writers. But, if you have a stream per aggregate, you can just run your command and save it with the proper expected version. If the save fails, then reload the aggregate, rerun the command (or whatever retry logic you have), and resave with the new expected version.

4

" If the save fails, then reload the aggregate,"

Reload any events you don't already have. e.g. if you sent 243 and its
at 246 reload 243->246 not all 246

5

Thanks to all three responders. So it seems a stream per aggregate is indeed the recommended usage, and that the recommended way to deal with multiple writers is to reload the new events before retrying the save.

6

Thats one way of dealing with it (or run stateless). There are a lot
of "it depends" there.

What are latency requirements? How many messages/second? How many
servers? Are messages going to many aggregates or only to a few but
quickly?

There isn't really a "do it this way solution". As an example if you
have a few aggregates say 1000 but they get hit a lot (say tickers in
a financial market) I would lean towards giving up stateless and
becoming stateful.

But it really depends on the problem.