"CheckpointTag has not been initialized" error after updating projection

All,

I am getting a “CheckpointTag has not been initialized” after updating a projection via the API.

The projection was created using ProjectionManager.CreateContinuous:

public void CreateContinuous(string name, string query);

The projection is updated later via ProjectionManager.UpdateQuery.

public void UpdateQuery(string name, string query);

The UI shows a “Checkpoints Enabled” checkbox when creating a new projection, but I am not seeing any way to control/manage that via the API.

Any ideas?

Thanks,

Ryan

Hi Ryan,

Does the problem persist if you restart the system?

I see the only possible scenario how you can get this exception… it is if you request a projection state immediately after the project has been created/updated. The correct behavior would be to respond with HTTP error code as the projection is not yet available. The small delay before requesting a projection state/result may help as temporary solution.

It is even possible that the projections UI requested a state.

Unfortunately this error causes the projection to stop.

I’ll fix the problem I described and let you now.

-yuriy

I would not recommend using the projections api directly at this
point. It is quite likely that we will break you in the future :slight_smile:

Restarting the service/exe doesn’t resolve the issue.

I’ll try putting in a delay before starting to use it.

Thanks,

Ryan

What direction will the projections API be heading?

Currently I am using the gES projections to create new streams based upon the event types. I have a C# projection that knows which events it cares about, and the C# projection automatically creates/updates a gES projection to create a new stream of those events that it then subscribes to.

As long as I can do the following via an API in some sort of fashion, I should be able to get by:

  • Create a new projection via the API

  • Pull the current javascript (“Query”) for a certain projection

  • Update the javascript (“Query”) for a certain projection

  • Will the “$by_event_type” system projection be sticking around?

  • Enable/Disable projection

Otherwise, it would be nice if the Client API had a way to create new streams based upon event types :slight_smile:

Thanks,

Ryan

The eventtype ones will stay.

For above I think what you really want to do is implement a join
operation (like fromStreams([,])). Its realtively easy though not
trivial to build. You could then do a fromStreams($et-Fooed,
$et-Bared, $et-Bazed) to join the events for your projection instead
of creating a custom stream for each one.

This is possibly something we could put into Client API

Cheers,

Greg

That sounds like what I want. Assuming that the events would be ordered properly and I could use a Subscription to get the events. The only thing I can’t determine is when I would need to do a full rebuild (i.e. restart the subscription at the 0 event). The way I am doing it currently, I can check to if the “query” script is different then a newly generated script. If it is, I know I need to do a rebuild (i.e. restart the subscription at the 0 event & do clean up in the C# projection).

The template that looks like this when generated:

options({ useEventIndexes: true });
function linkIt(s, e) {
linkTo(‘someStream’, e);
}
fromAll()
.when({
Event1: linkIt,
Event2: linkIt,
Event3: linkIt,
Event4: linkIt,
Event5: linkIt,
});

Where I fill in the stream name & the event names based upon what events I have in C#.

Thanks,

Ryan

Yuriy,

I tried putting in a delay (500ms) prior to using the projection, but it still fails:

Failure Reason:

```

CheckpointTag has not been initialized


It seems to only happen when I update the projection's "Query" script.

Ryan

You can do this from C# as well instead of doing a custom projection every time.

We will talk about pushing that functionality into the client API
tomorrow. Basically what you do is read from N streams and reorder the
results (projections does this internally for your query!)

Cheers,

Greg

Let me know if you need any of my input. :slight_smile:

Ryan

Do you see anything in the -err log?

Yuriy,

OK after working on this a little more, I am not always getting the previous error “CheckpointTag has not been initialized”, but I do see the gES Projection freeze up and stop working (even though it shows Running still).

It only seems to happen when you update the Query/script… In the cases I have been seeing this happen, I am usually adding additional events that I want to project, or I am removing events I want to follow.

The last time this happened, I captured the error log, here are the error messages:

[PID:18340 2013.04.23 18:59:55.976 Error QueuedHandlerMRES 19]
Error while processing message EventStore.Projections.Core.Messages.ProjectionSubscriptionManagement+Subscribe in queued handler ‘Projection Core #2’.
System.ArgumentException: Number of streams does not match
Parameter name: fromPositions
at EventStore.Projections.Core.Services.Processing.MultiStreamEventReaderBase1.ValidateTag(CheckpointTag fromPositions) in c:\BuildAgent\work\c5e2638da829398b\src\EventStore\EventStore.Projections.Core\Services\Processing\MultiStreamEventReaderBase.cs:line 87 at EventStore.Projections.Core.Services.Processing.MultiStreamEventReaderBase1…ctor(IPublisher publisher, Guid distibutionPointCorrelationId, String[] streams, Dictionary2 fromPositions, Boolean resolveLinkTos, ITimeProvider timeProvider, Boolean stopOnEof, Boolean skipStreamCreated) in c:\BuildAgent\work\c5e2638da829398b\src\EventStore\EventStore.Projections.Core\Services\Processing\MultiStreamEventReaderBase.cs:line 74 at EventStore.Projections.Core.Services.Processing.CheckpointStrategy.CreatePausedEventIndexEventReader(Guid eventReaderId, IPublisher publisher, CheckpointTag checkpointTag, Boolean stopOnEof, Boolean resolveLinkTos, IEnumerable1 streams) in c:\BuildAgent\work\c5e2638da829398b\src\EventStore\EventStore.Projections.Core\Services\Processing\CheckpointStrategy.cs:line 155
at EventStore.Projections.Core.Services.Processing.CheckpointStrategy.CreatePausedEventReader(Guid eventReaderId, IPublisher publisher, CheckpointTag checkpointTag, Boolean stopOnEof) in c:\BuildAgent\work\c5e2638da829398b\src\EventStore\EventStore.Projections.Core\Services\Processing\CheckpointStrategy.cs:line 105
at EventStore.Projections.Core.Services.Processing.ProjectionSubscriptionBase.CreatePausedEventReader(IPublisher publisher, Guid eventReaderId) in c:\BuildAgent\work\c5e2638da829398b\src\EventStore\EventStore.Projections.Core\Services\Processing\ProjectionSubscriptionBase.cs:line 134
at EventStore.Projections.Core.Services.Processing.ProjectionCoreService.Handle(Subscribe message) in c:\BuildAgent\work\c5e2638da829398b\src\EventStore\EventStore.Projections.Core\Services\Processing\ProjectionCoreService.cs:line 222
at EventStore.Core.Bus.MessageHandler1.TryHandle(Message message) in c:\BuildAgent\work\c5e2638da829398b\src\EventStore\EventStore.Core\Bus\MessageHandler.cs:line 60 at EventStore.Core.Bus.InMemoryBus.PublishByType(Message message, Type type) in c:\BuildAgent\work\c5e2638da829398b\src\EventStore\EventStore.Core\Bus\InMemoryBus.cs:line 127 at EventStore.Core.Bus.InMemoryBus.DispatchByType(Message message) in c:\BuildAgent\work\c5e2638da829398b\src\EventStore\EventStore.Core\Bus\InMemoryBus.cs:line 105 at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o) in c:\BuildAgent\work\c5e2638da829398b\src\EventStore\EventStore.Core\Bus\QueuedHandlerMRES.cs:line 136 [PID:18340 2013.04.23 19:11:21.506 Error QueuedHandlerMRES 19] Error while processing message EventStore.Projections.Core.Messages.ProjectionSubscriptionManagement+Unsubscribe in queued handler 'Projection Core #2'. System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary. at System.Collections.Generic.Dictionary2.get_Item(TKey key)
at EventStore.Projections.Core.Services.Processing.ProjectionCoreService.Handle(Pause message) in c:\BuildAgent\work\c5e2638da829398b\src\EventStore\EventStore.Projections.Core\Services\Processing\ProjectionCoreService.cs:line 183
at EventStore.Projections.Core.Services.Processing.ProjectionCoreService.Handle(Unsubscribe message) in c:\BuildAgent\work\c5e2638da829398b\src\EventStore\EventStore.Projections.Core\Services\Processing\ProjectionCoreService.cs:line 238
at EventStore.Core.Bus.MessageHandler1.TryHandle(Message message) in c:\BuildAgent\work\c5e2638da829398b\src\EventStore\EventStore.Core\Bus\MessageHandler.cs:line 60 at EventStore.Core.Bus.InMemoryBus.PublishByType(Message message, Type type) in c:\BuildAgent\work\c5e2638da829398b\src\EventStore\EventStore.Core\Bus\InMemoryBus.cs:line 127 at EventStore.Core.Bus.InMemoryBus.DispatchByType(Message message) in c:\BuildAgent\work\c5e2638da829398b\src\EventStore\EventStore.Core\Bus\InMemoryBus.cs:line 105 at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o) in c:\BuildAgent\work\c5e2638da829398b\src\EventStore\EventStore.Core\Bus\QueuedHandlerMRES.cs:line 136 [PID:18340 2013.04.23 19:11:21.537 Error QueuedHandlerMRES 19] Error while processing message EventStore.Projections.Core.Messages.ProjectionSubscriptionManagement+Subscribe in queued handler 'Projection Core #2'. System.ArgumentException: An item with the same key has already been added. at System.Collections.Generic.Dictionary2.Insert(TKey key, TValue value, Boolean add)
at EventStore.Projections.Core.Services.Processing.ProjectionCoreService.Handle(Subscribe message) in c:\BuildAgent\work\c5e2638da829398b\src\EventStore\EventStore.Projections.Core\Services\Processing\ProjectionCoreService.cs:line 222
at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message) in c:\BuildAgent\work\c5e2638da829398b\src\EventStore\EventStore.Core\Bus\MessageHandler.cs:line 60
at EventStore.Core.Bus.InMemoryBus.PublishByType(Message message, Type type) in c:\BuildAgent\work\c5e2638da829398b\src\EventStore\EventStore.Core\Bus\InMemoryBus.cs:line 127
at EventStore.Core.Bus.InMemoryBus.DispatchByType(Message message) in c:\BuildAgent\work\c5e2638da829398b\src\EventStore\EventStore.Core\Bus\InMemoryBus.cs:line 105
at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o) in c:\BuildAgent\work\c5e2638da829398b\src\EventStore\EventStore.Core\Bus\QueuedHandlerMRES.cs:line 136

Thanks,

Ryan

Ryan,

I understand the problem now. You are right that it is related to the changes in the number of events handled by the projection. It changes the structure of the projection position tag.

Changes to the projection definition are not always correctly recognized as safe or unsafe. If you remove an event type from the projection using event-type-index you will likely get a problem.

In the nearest release, we will allow developers to restart projections. It will solve similar problems while you are developing projections. Any unsafe changes to the projection definition will be allowed only with the projection restart.

In production, however, you need to plan projection upgrades, so they stay compatible with their previous versions.

regards,
yuriy