I want to emphasize again that I don’t have a complete grasp of the solution and problem that you’re facing. I can speak with more confidence to the patterns than the particulars of your situation.
I should also say that I’m not in a position to conclude whether you’re doing things the “right way”. Again, I wouldn’t want to take such a strong stance until I felt as intimate with the business and the systems as you do. And it’s likely that such a level of knowledge transfer might take more time than I have.
To some specific points…
“InventoryItem” is a class of individual instances of an aggregate root. “inventoryItem-123” is a particular instance. The stream names are ideally named for the instance. And each instance is given it’s own stream. That’s how you can get the state of a single entity by directly querying that entity’s events.
PS: You can get all events for all instances of the InventoryItem class (or “category”) from the stream named “$ce-inventoryItem”. Note, though, that this feature (category streams) is not supported yet. They’re based on projections, and projections are not supported at this time (though they’re in-development, I understand).
There’s no issue of having one hell of a lot of unique streams and stream names as far as I know. It’s what the database is designed for. It probably feels a bit alien when thinking of the implications of having hundreds of thousands (or millions) or database tables in a more familiar database system, but EventStore is a different kind of thing.
There’s another big challenge to pre-conceptions that you might watch out for: Aggregate roots aren’t necessarily the “things” in your system. Or I should say, they are often the names of processes. The “things” in a system like this are in fact the major business operations. You might have aggregate roots that reflect the kinds of entities we have with ORM (which is largely the result of database modeling), but it’s largely a coincidence rather than a design objective.
So, rather than InventoryItem, you might in fact have InventoryUpdate. And that’s desirable. Or at least, it’s not out of the ordinary. The service that knows the inventory rules and data for an item might not know it’s price (for example).
Also, it’s normal to not have all the data for an InventoryItem in a single service/aggregate/boundary/component.
All that said, you might in fact have an InventoryItem. Again, I’m not familiar enough with the solution and the problem to be able to make a good judgement from this far away from it.
For the batching issue, you could receive those 12k updates, and break them into smaller batches or individual updates to reduce the risk of the whole batch needing to be re-processed. Once each individual update is processed, the batch would be considered complete/closed/done/etc. So, you might end up with a component that receives those batches from the outside, creates smaller batches (down to the size of 1), and another component that processes the individual updates. Each time an update completes, it records an event in the batch’s stream that the update was completed. Once all the individual updates are accounted for, the sub-batch replies to the larger batch that it was done.
This is more a matter of building in reply semantics, and that’s a bit of work up front if you don’t have a generalized solution for that yet. You’d want to include something like a “replyStreamName” attribute in the metadata, and then use that stream name as the place to record the reply event. Technically, it should be a command that’s recorded and then the original process should record it’s own events, rather than letting anything write into anything else’s event streams (but that’s a finer point)
Given that there are rules (policies) about how big the batch itself can be, this BatchUpdate appears even more to be a concern all on its own.
You can report back on the process by polling an interface (or using some push messaging) that can indicate the state of the process. There are good ways and bad ways to do this though. The most obvious - hanging a JSON API of the service that returns data - is the least best. Updating some other data store with summary information about these BatchUpdate instances might be the better thing to do.
I’m hypothesizing based on patterns and principles though, there are other factors that need to be considered that I don’t have (and again, might not be able to grasp).
If you can find a solution where you don’t need to query event store for the presence of a particular ID (or correlation ID), then you’ll be better off. Otherwise, you’ll end up doing a good deal of sequential scans to figure out whether or not to take some action or whether some action was already taken.
Also, EventStore’s replication might make it appear briefly that an event you just wrote is not readable yet. So, a read-modify-write approach might simply be impossible. That’s a more subtle thing, and it’s an edge case.
I don’t know if the http_eventstore gem supports leader election in EventStore’s cluster (which is when the cluster decides to select a new node to be the master, which it will do as it sees fit). The effect of this is that the master node you thought you were connected to is now a slave node, and it is susceptible to experiencing greater latency.
All of this means that you need to rely more on reply messages than on reading from EventStore and then making a decision as to whether to do (or not do) something. IE: You can’t add to a stock level and then check if there’s sufficient stock, and if so, allocate it to an order (etc). You need to tell the system to reduce stock, and it will in-turn reply to its caller with another message that says whether that stock reduction (allocation, etc) succeeded. And upon that reply, the order can proceed. This is why you might be able to purchase something from Amazon, and then get a message later saying that there was actually insufficient stock at the time that the order went through, and that the order is delayed/refunded/etc.
Finally, I think the metadata fields that start with “$” are for internal use. I was under the impression that these should not be used to store our application-specific stuff (like correlation IDs in our apps). I could have this wrong though.
So, I fear that I might just also be making things more confusing for you. I’m throwing a bunch of stuff into the mix that might make the whole thing feel burdensome. There may be other ways and other suggestions that others have. And others might also have a better picture of your problem/solution than I do.