Impact of updating a user defined projection

EventStoreDB V5.X
1

Hi, I have a user defined projection that reads from a few categories, performs some logic & conditionally links events together via linkTo, the pseudo code for the projection is as follows:

fromCategories(['A', 'B', 'C'])
    .when({
        $any: function(s, e)
        {
            var streamName = e.streamName;
            var streamId = e.streamId;

            switch(streamName){
                case "stream-a":
                    if(someCondition){
                        linkTo(`New.Stream-${streamId}`, e);
                    }
                    break;
                case "stream-b":
                    if(someOtherCondition){
                        linkTo(`New.Stream-${streamId}`, e);
                    }
                    break;
                case "stream-c":
                    linkTo(`New.Stream-${streamId}`, e);
                    break;
            }
        }
    });

I need to update the projection so that it reads from a further category, D, the projection will be updated as follows

fromCategories(['A', 'B', 'C', 'D'])
    .when({
        $any: function(s, e)
        {
            var streamName = e.streamName;
            var streamId = e.streamId;

            switch(streamName){
                case "stream-a":
                    if(someCondition){
                        linkTo(`New.Stream-${streamId}`, e);
                    }
                    break;
                case "stream-b":
                    if(someOtherCondition){
                        linkTo(`New.Stream-${streamId}`, e);
                    }
                    break;
                case "stream-c":
                    linkTo(`New.Stream-${streamId}`, e);
                    break;
                case "stream-d":
                    linkTo(`New.Stream-${streamId}`, e);
                    break;
                    
            }
        }
    });

From this thread I understand that all of the events in category D would have to be processed by the projection as it isn’t easily to set the start position of a stream within a projection.

Currently D has in the region of 82 million events. ‘B’ averages 20 events/second, A & C are < 1 event/second.

What I’d like to understand is

  • What happens when I add D to the projection
    • Will processing of new events on A, B & C be impacted whilst the projection catches up with events from D?
    • Is it advisable to consume events from $ce-New.Stream whilst the projection catches up?
    • Is it possible to estimate how long it will take for the projection to catch up with events from D?

I’m running event store 5.0.8.0 & currently have 4 persistent subscribers subscribed to $ce-New.Stream

2

Do you need to process all 82 million events from D, or would it be acceptable skip past a number of these (perhaps they are old and not needed)

3

It would be acceptable to skip all of the events from D.

4

The official guys might have a nicer solution, but if it was me, I would introduce the new category stream into the projection, and update the checkpoint on the projection (posting an event onto the stream) to push the position forward.

Something similar:

5

Any pointers on how you’d go about achieving that?

I’ve given it a go without much success so far, here’s a snippet of code

var streamName = "$projections-$projectionName-checkpoint"; 

...

    client.SubscribeToStreamFrom(streamName, null, CatchUpSubscriptionSettings.Default,
                    async (subscription, @event) =>
                    {
                       
                            var eventData = new EventData(Guid.NewGuid(), $"$ProjectionCheckpoint", true,
                                Encoding.Default.GetBytes("[{},{}]"),
                                Encoding.Default.GetBytes(
                                    "{\"$v\":\"<version>\",\"$s\":{\"$ce-A\":1234,\"$ce-B\":1234,\"$ce-C\":1234,\"$ce-D\":82000000}}"));
                            var position = @event.OriginalEventNumber;
                            Console.WriteLine($"Position : {position}");
                            var a = await client.AppendToStreamAsync(streamName, position + 1, eventData);

                            Console.WriteLine($"Log position {a.LogPosition}");
                            Console.WriteLine($"next version {a.NextExpectedVersion}");
                        
                       
                    }, (s) => { });
6

You don’t need to this via source code. The UI makes it pretty easy (or Postman)
If you browse to the checkpoint stream:

$projections-{projection_name}-checkpoint