reliability of reordering events in projections

EventStoreDB
1

Understanding that the comments say “attempts to feed the projection with events in their natural order” it seems like I may have no luck trying to achieve my current task. I am bulk importing events from an old SQL based version of NEventStore into EventStore, so events are being written to aggregate based streams very quickly. I then created a projection of events I was interested in, emitting links to the events. Projection code here: https://gist.github.com/briandonahue/08dcfacd49ba921e6af6

Unfortunately, the events are not showing up in the projected stream in natural order, so my readmodel updater fails when an EditionCreated event doesn’t happen before another event. Is this just wishful thinking, or is there some way to get this to work? Maybe a better way to achieve what I want? This is a simple test, but I was hoping to have other projections that merged events from multiple aggregates, so I can’t just read the aggregate streams themselves.

Thanks,

Brian

2

Projections get events in exactly the order they were written in. fromStreams and reordering only affects realtime stuff not history. The reason why things are like this is consider what happens when you have streams on different nodes. Another thing you could do here is write your query slightly differently in that it could use a fromAll instead of a fromStreams…

The query fromStreams([’$et-type1’, $ettype2, $ettype3, etc]) can also be expressed as

fromAll().

when({

type1 : function(s,e) { },

type2 : function(s,e) { },

//etc

}

in fact projections will turn the second query into the first automatically if the by-type index is enabled.

In general though for your usecase I think it may be better for us to expose some internal code that can do arbitrary queries joining streams. I have opened an issue for this https://github.com/EventStore/EventStore/issues/162 feel free to comment on the issue itself.

Cheers,

Greg

3

Projections get events in exactly the order they were written in.
fromStreams and reordering only affects realtime stuff not history.

This doesn't seem to be happening in my test. Given these original streams
(written to in bulk on an import, so insert times are likely within
milliseconds of each other for many events):

Event stream 'Edition-256'
2 2@Edition-256 EditionDemographicInfoChanged 2014-07-20 21:34:17
1 1@Edition-256 EditionOfficePermissionsChanged 2014-07-20 21:34:17
0 0@Edition-256 EditionCreated 2014-07-20 21:29:48

Event stream 'Edition-257'
4 4@Edition-257 EditionDemographicInfoChanged 2014-07-20 21:45:11
3 3@Edition-257 EditionDescriptionUpdated 2014-07-20 21:45:11
2 2@Edition-257 EditionDemographicInfoChanged 2014-07-20 21:34:18
1 1@Edition-257 EditionOfficePermissionsChanged 2014-07-20 21:34:18
0 0@Edition-257 EditionCreated 2014-07-20 21:34:18

Event stream 'Edition-2'
3 3@Edition-2 EditionContactInfoChanged 2014-07-20 21:34:19
2 2@Edition-2 EditionDemographicInfoChanged 2014-07-20 21:34:19
1 1@Edition-2 EditionOfficePermissionsChanged 2014-07-20 21:34:19
0 0@Edition-2 EditionCreated 2014-07-20 21:34:19

My projection produces a stream of events in this order:

Event stream 'ReadModelStream'
9 3@Edition-2 EditionContactInfoChanged 2014-07-20 13:21:26
8 1@Edition-2 EditionOfficePermissionsChanged 2014-07-20 13:21:26
7 0@Edition-2 EditionCreated 2014-07-20 13:21:26
6 2@Edition-2 EditionDemographicInfoChanged 2014-07-20 13:21:26
5 1@Edition-257 EditionOfficePermissionsChanged 2014-07-20 13:21:25
4 0@Edition-257 EditionCreated 2014-07-20 13:21:25
3 2@Edition-257 EditionDemographicInfoChanged 2014-07-20 13:21:25
2 1@Edition-256 EditionOfficePermissionsChanged 2014-07-20 13:21:24
1 2@Edition-256 EditionDemographicInfoChanged 2014-07-20 13:21:24
0 0@Edition-256 EditionCreated 2014-07-20 13:21:24

As you see, there are not in the original order, and I'm not sure why. I
was assuming I was doing something wrong, which may be the case, but
thought I'd ask. Happy to provide you with anything you need to help debug
if this is in fact an issue...

4

Can you read from $all and get whats in $all for these? You should never ever get out of order events in the same stream

5

It looks like the events are in the expected order with $all, so I can use that. Thanks for your help!

6

Can you put what is in it?

7

The all stream?

8 2@Edition-2 EditionDemographicInfoChanged 2014-07-20 21:34:19

7 1@Edition-2 EditionOfficePermissionsChanged 2014-07-20 21:34:19

6 0@Edition-2 EditionCreated 2014-07-20 21:34:19

5 2@Edition-257 EditionDemographicInfoChanged 2014-07-20 21:34:18

4 1@Edition-257 EditionOfficePermissionsChanged 2014-07-20 21:34:18

3 0@Edition-257 EditionCreated 2014-07-20 21:34:18

2 2@Edition-256 EditionDemographicInfoChanged 2014-07-20 21:34:17

1 1@Edition-256 EditionOfficePermissionsChanged 2014-07-20 21:34:17

0 0@Edition-256 EditionCreated 2014-07-20 21:29:48

8

Hello Greg,

Hope you are doing well!

A follow up query … in your fromAll() projection example, let’s assume that I am linking to other streams in each function … something like …
function(s,e) {
linkTo(‘stremname1’, e);
linkTo(…);
}

Since we are writing to one or more streams for each event type will it create back pressure leading to events getting lost or going out of order ?