so,
. i’m reading a file which may contain anywhere between a few hundred and several hundred thousand items.
. each item is small, only a couple of fields.
. when they’ve all been dealt with, the result needs to be reconstituted into the same item set
is it better to store all the items into a single event in eventstore, or handle the extra complexity and have one event per item?
many thanks
100.000’s will probably hit the max eventsize limit.
Single event per item will need a lot of storage, there’s overhead storing very small events. May or may not be a problem, depending on your environment.
I have a similar issue to deal with in the next days, and the options I’m considering are:
Single events, with retention policy.
‘batch sized’ events, say 100 items per event.
(Alternative storage, need the events mainly for reporting.)
Have items in 1000’s rather than 100k’s, but I’m still not very keen on big event, have had issues before…
/Peter
The other question you need to ask is are there any places where a
subscriber might be interested in one thing from the document but not
all the other things..
Another easy thing to look at is how people discuss things. Is a
"file" a global conceptual unit or is it seen as a batch of other
things?
Greg
Tricky one. Currently, the idea is just to treat the file and it’s contents atomically, but going forward who knows 
Mike