My service grabs commands from Kafka. I use event sourcing pattern to persist the aggregates. To enhance the audit in my service, there are suggestions to persist commands in event store and then, with the help of correlation id/causation id/message id, I could visualize the entire conversation in my service.
If I store commands in a separate stream and the events as part of aggregate’s stream, I believe, I need to either:
Put command stream and aggregate stream append operations in a transaction.
Split these two operations in the following way: first append the command in the commands stream, then subscribe to commands stream(s) and actually execute the command.
Option #2 seems a bit heavy, but does not use transactions. Also, Kafka offers scaling by partitioning, which is not present in Event Store, hence competing consumers will not be an option without partitioning as it would impact commands ordering.
I would definitively separate command & aggregate streams .
option 2 is not heavy but it
does introduce some latency
(mis)use the stream as a persistent queue, so you need to manage the checkpoint
sine it’s (mis)used as a queue , you need to handle failed commands / retries / …
.
there is a 3way to improve the audit and reduce the burden of the above points
do store command on arrival and keep on processing as you normally would in your pipeline
-> if the command is not saved for whatever reason you can choose to continue or just return some error to the sending process.
Really like the thought of logging commands. Our services currently support HTTP and RabbitMQ communication. In the other posts about this topic I read about the possibility to replace RMQ with (or at least prefer) ESDB as well by subscribing to a command stream. Supporting both HTTP and ESDB would create two flows. The HTTP flow would receive a command over http, stores it in a command type stream and handles the command. The ESDB flow would receive the command by subscription (no store operation required, as it is already been stored) and handles the command. Using a command type stream, the HTTP request would also trigger the services subscription. What would be the best way to filter these “already handled HTTP commands” out? I read about the possibility to filter subscriptions for the $all stream, but for other streams there seems to be no such sort of option?