Change position on Projection

EventStoreDB
#1

Take this simple projection:

fromStreams('$et-EventType1', '$et-EventType2')
    .when({
        $init: function(s, e) {
            return {
                count: 0
            };
        },
        $any: function(s, e) {
            s.count++;

            if( s.count > 1000000){ //this could be a date range as well
                //start doing stuff
            }
        }
    );

Assume that $et-EventType2 has millions of events, but we aren’t interested in most of these events.

is it possible to start a projection from the a set position (or event the last position) rather than having to trawl through millions of events?

#2

There is not as of now.

You pick a particularly odd example as well (it has multiple inputs which makes it more annoying API-wise) :slight_smile:

As example I could see relatively easily:

fromStream('foo').StaringWith(1000000)

or even

fromAll().After('1/1/2019')

With multiple streams I would imagine you would want to be able to control it per source stream as opposed to say:

fromStreams('foo', 'bar', 'baz').After('1/1/2019')

How might you see this expressed in terms of writing the filter?

#3

Greg,

Yeah, in my example, $et-EventType2 is the one I want to skip past.

So just now, fromStreams simply lets u pass in a list of strings, could that be altered to be an object?

fromStreams( 
    new { "streamname" : "$et-EventType1" },  
    new { "streamname" : "$et-EventType2", "startPosition" : 1000000 }

I appreciate that looks a little complicated, but I would guess it could be enhanced to start including filters etc.

Another option might be this:

fromStream( "$et-EventType1")
    .fromStream( "$et-EventType2")
    .StaringWith(1000000)
#4

In your second example what does startingwith apply to? The first fromStream the second fromStream? This seems like someone would want to be able to say per stream possibly different values…

#5

It applies to EventType2.

So,

fromStream( “$et-EventType1”).StaringWith(-1).fromStream( “$et-EventType2”).StaringWith(1000000) //the first StartingWith would be the default, but just to show my thinking.

#6

Greg,

Do you want me to create a Github issue for this?

Just for anyone else interested, we have a work around for this problem, where we have created a Persistent Subscription for EventType2, and fed it back into some code.

The StartPosition can be set here, so we can easily skip past the first million events.

#7

Thread necromancy.

This really is a problem for us.
As a trade off, would it be possible to only allow a position start on fromCategory?

fromCategory('SalesAggregate',50000)
.foreachStream()

Right now, we have a category with millions of events (32286455), and we want to create a projection which will create an enriched event once a certain condition is met.

The older events we don’t have to worry about, but I can’t see any way of putting the projection onto the Eventstore without having to iterate through all these streams / events.

Even a “start from last event position” would be great.

Anyone else experience this problem?

#8

Hi Steven,

We do still have this on the roadmap, and intend on adding the feature to start a projection from now shortly after the next major release.
The linked github issues are here and here if anyone would like to track their progress.

We will also be releasing filtered reads soon, which may help in situations like this. You would be able to specify the category as a stream prefix and read from a specific position in the $all stream.

As a workaround for right now, you could write your own checkpoint for the projection.
To do this you would need to do something like :

  1. Stop the projection
  2. Copy an existing checkpoint from the $projections-{projection_name}-checkpoint stream
  3. Update the position or version number in the checkpoint event
  4. Write it back to the checkpoint stream, ensuring you keep the same metadata as the existing checkpoint event
  5. Start the projection again
Impact of updating a user defined projection
Impact of updating a user defined projection
$by_eventtype projection failing to start
#9

Thanks for the reply.
We will give your checkpoint suggestion a go.

We actually just kicked off a crude solution on our Eventstore to get round this problem, which involved us truncating the big category stream. A scavenge is running just now, and once finished, our new projection will start roughly from the last few days worth of events.