Hi,
I’m in the process of working out how to interact with Event Store in a system I’m designing. In a nutshell, I have accounts, and each account can have a set of projects, and each project can have a set of entities, with further entities nested inside parent entities, do some arbitrary hierarchical depth. I will have a global event stream representing changes to my system as a whole, and there will be a decision process that looks at each event in the stream, as it is received, and if the entity that the event relates to is “active”, the event will be passed to that entity’s individual event stream for processing by a worker process. Now, whether or not an entity is active will depend on a hierarchical list of lookups; i.e. is that parent’s entity active? Is the project that the entity belongs to active? Is the account to which the account belongs to active? This is pretty trivially handled by an external subscribing process, as it can make that determination using appropriate queries to either the Event Store or perhaps to the database where the current data is flattened to. So in that sense, I can get away without using projections at all as I’m just looking at events and deciding what to do with them.
Here’s my question. I have good reason to expect that my global event stream could get very, very busy, which means I start looking at distributing the workload, either in terms of Event Store’s clustering features, and/or in adding additional decider processes to parallelise processing of the global event stream. My concern is that if I do parallelise the global event stream processing, I’m not sure how I ensure that events forwarded to other streams stay in the same order in which they were received. My first thought is that a clustered Event Store probably handles correct ordering even though projections may be working in parallel on different machines, though that’s just an assumption. The problem is that, correct me if I’m wrong, I wouldn’t be able to use projections for my use case, because they only operate on data they’re fed, forwarding projected data to other streams. Trying to determine where to project based on the state of a different stream isn’t going to work because there is no querying mechanism… again this is only what I’ve gathered from looking at Rob Ashton’s blog posts and so forth.
So, if I am correct and I can’t really use projections to ensure correctly-ordered event forwarding in a clustered setup, does Event Store provide any other mechanism whereby two parallel processes can be forwarding events from the same event stream source and yet ensuring that they are forwarded in the correct order? Perhaps this is an architectural problem that is outside the scope of what Event Store can do for me… any advice would be greatly appreciated.