Just wondering how people deal with out of order events from different nodes , we may have no comms for a day and then all the events come in and need to be re-merged,. My thoughts
Use a custom projection when seeing an out of order go back to that utc time and replays all later events. (Replay will only occur when events from other nodes are merged)
Aggregates reorder by date when loaded.
Neither are too appealing , but i dont see any other options.
Ben
I'm guessing this is with your own system for moving them around.
Have you looked at vector clocks?
With our current system we use date time which is ok for aggregation ( without the complications introduced by vector clocks) , its not good enough however for moving events ( data in our case) around and here we use sqltiemstamp on each server we get data from . We use a half baked aggregation ( history in sql rows with nullable fields because we have too many sql people but they can see the issues with that now) its cumbersome but does work.
If we move to event store ( which will be a big project) how will we go about this . Especially the aggregation ( the idea of fetching events from the event store and reordering on time to deal with merged events seems wrong but i haven thought deeply enough nor do i know the event store well enough ) , re running projections when we have an out of order is acceptable.
Ben
So in the federated model which is under way the way this works is no
reordering but conflict detection (eg it tells you that things may
have happened you didn't know about) and client level ordering when in
history. Vector clocks are a good way of handling this. In my
experience the problems to watch out for are not so much the
projections having different order of events on multiple nodes (this
can be handled in projection code) but in two nodes having done things
that they wouldn't have been able to do if they knew what the other
was doing.
So in the federated model which is under way the way this works is no
reordering but conflict detection (eg it tells you that things may
have happened you didn’t know about) and client level ordering when in
history.
What do you mean by client level ordering ? Is that reorder the events for processing according to say a datetime field ? Yes that would be fine.
Vector clocks are a good way of handling this.
Utc Datetime should suffice in most cases its minutes to hours between these events ( except for one) but comms could be out for hours to a day. If this becomes non tenable than we will move to Vector clocks.
In my
experience the problems to watch out for are not so much the
projections having different order of events on multiple nodes (this
can be handled in projection code)
Do you mean reprocess projections on a conflict detection ( ie out of order)
but in two nodes having done things
that they wouldn’t have been able to do if they knew what the other
was doing.
Yes that is an issue but the device responsible has human over rides . Our biggest issue is things looking weird on other nodes ( audit / super visor) if things are not processed in the right order ( its pretty much read only at this point) , if some data is missing that is ok as long as it is eventually ok .
Regards,
Ben