Projection stuck

EventStoreDB
1

Hi,
We have a projection stuck at some point, we cannot get the projection’s state or result (error code 408).

The status of the projection is “running”, here is the stats we have:
Events/sec
0.0
Buffered events
2739
Events processed
0
Partitions cached
1064
Here is the error backtrace:
[PID:20973:013 2015.04.20 13:47:41.935 ERROR QueuedHandlerAutoRes] Error while processing message EventStore.Core.Messages.ClientMessage+ReadStreamEventsBackwardCompleted in queued handler ‘Projection Core #1’.
System.Exception: StartObject or EndArray expected (At: 1, 2)
at EventStore.Projections.Core.Services.Processing.PartitionState.Error (Newtonsoft.Json.JsonTextReader reader, System.String message) [0x00000] in :0
at EventStore.Projections.Core.Services.Processing.PartitionState.Deserialize (System.String serializedState, EventStore.Projections.Core.Services.Processing.CheckpointTag causedBy) [0x00000] in :0
at EventStore.Projections.Core.Services.Processing.DefaultCheckpointManager.OnLoadPartitionStateReadStreamEventsBackwardCompleted (EventStore.Core.Messages.ReadStreamEventsBackwardCompleted message, EventStore.Projections.Core.Services.Processing.CheckpointTag requestedStateCheckpointTag, System.Action1 loadCompleted, System.String partitionStreamName, System.String stateEventType) [0x00000] in <filename unknown>:0 at EventStore.Projections.Core.Services.Processing.DefaultCheckpointManager+<BeginLoadPartitionStateAt>c__AnonStorey0.<>m__0 (EventStore.Core.Messages.ReadStreamEventsBackwardCompleted m) [0x00000] in <filename unknown>:0 at EventStore.Core.Messaging.RequestResponseDispatcher2[EventStore.Core.Messages.ClientMessage+ReadStreamEventsBackward,EventStore.Core.Messages.ClientMessage+ReadStreamEventsBackwardCompleted].Handle (EventStore.Core.Messages.ReadStreamEventsBackwardCompleted message) [0x00000] in :0
at EventStore.Core.Messaging.RequestResponseDispatcher2[EventStore.Core.Messages.ClientMessage+ReadStreamEventsBackward,EventStore.Core.Messages.ClientMessage+ReadStreamEventsBackwardCompleted].EventStore.Core.Bus.IHandle<TResponse>.Handle (EventStore.Core.Messages.ReadStreamEventsBackwardCompleted message) [0x00000] in <filename unknown>:0 at EventStore.Core.Bus.MessageHandler1[EventStore.Core.Messages.ClientMessage+ReadStreamEventsBackwardCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0
at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0
at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0
at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0

``

I understand the projection state is faulty here but I don’t see a way to actually debug it: the projection debug page is stuck at “Loading definition…”.

We are running Ubuntu GNU/Linux 14.04.2 LTS.

Thoughts anyone?
Thanks

2

I think it’s malformed javascript. Can you post your entire projection here please?

3

" at EventStore.Projections.Core.Services.Processing.PartitionState.Deserialize (System.String serializedState, EventStore.Projections.Core.Services.Processing.CheckpointTag causedBy) [0x00000] in:0"

Looks like its the projections state not the projection itself

4

Sure, here is the projection code:

/* globals fromAll, fromCategory, foreachStream, emit, linkTo */

var indexDuel = function (duels, duel_id) {
if (duels.indexOf(duel_id) === -1) {
duels.push(duel_id);
}
};

var forEachDuel = function (duels, iter) {
duels = duels || [];
var i, n = duels.length, duel_id;

for (i = 0; i < n; i++) {
duel_id = duels[i];
iter(duel_id);
}
};

fromAll().
partitionBy(function (e) { return e.data.hail_id; }).
when({
“$init”: function () { return []; },

"DuelCreated": function (duels, message) {
  indexDuel(duels, message.data.duel_id);
},

"HailQuestionsFinished": function (duels, message) {
  forEachDuel(duels, function (duel_id) {
    emit("hail_to_duels_dispatch", "DuelQuestionsFinished", { duel_id: duel_id, source: message.data });
  });
},

"GuessesScoringWaitTimedout": function (duels, message) {
  forEachDuel(duels, function (duel_id) {
    emit("hail_to_duels_dispatch", "DuelScoringWaitTimedout", { duel_id: duel_id, source: message.data });
  });
}

});

``

5

Hi Greg,
Is there any known restriction we should be aware in projections?
Do you see a way to figure out this issue?

6

No this is an issue that happens occasionally on recovery (IIRC it was
found and is either fixed or is in process and was an off by one
error). Until then you can either manually edit the check point to
point to the appropriate event or you can reset the projection by hand
which will cause it to re-run over data but will result in correct
data at the end

7

No this is an issue that happens occasionally on recovery (IIRC it was found and is either fixed or is in process and was an off by one error).
Until then you can either manually edit the check point to point to the appropriate event or you can reset the projection by hand which will cause it to re-run over data but will result in correct data at the end

I assume reset is not safe when using linkTo, only when projecting to state? In my tests I get duplicates in the resulting streams...

/Peter

8

Thanks, that helps.
Could you point me how to manually edit checkpoints? I’m unable to find documentation related to that.

9

You do not get duplicates. Look again. What happens is the original
streams get deleted and then recreated (all events have new positions)

10

The checkpoint stream is $projections-{projection}-checkpoint