Projection faulted due to '"Record is too big Parameter name: data"

EventStoreDB V5.X
1

Hi! I’ve been running ES 5.0.9 for ages without problems.
But now one of my projections suddenly started failing.
Even if I create a new projection from scratch, it fails at exactly the same place. Here is excerpt from the logs:

[00001,26,06:35:43.567] SLOW QUEUE MSG ["Projection Core #1"]: "CoreTick" - 861ms. Q: 6/9.
[00001,26,06:35:44.512] SLOW BUS MSG ["bus"]: "CoreTick" - 944ms. Handler: "ProjectionCoreService".
[00001,26,06:35:44.512] SLOW QUEUE MSG ["Projection Core #1"]: "CoreTick" - 944ms. Q: 5/9.
[00001,26,06:35:45.570] SLOW BUS MSG ["bus"]: "CoreTick" - 1058ms. Handler: "ProjectionCoreService".
[00001,26,06:35:45.570] SLOW QUEUE MSG ["Projection Core #1"]: "CoreTick" - 1058ms. Q: 4/6.
[00001,26,06:35:46.677] SLOW BUS MSG ["bus"]: "CoreTick" - 1105ms. Handler: "ProjectionCoreService".
[00001,26,06:35:46.677] SLOW QUEUE MSG ["Projection Core #1"]: "CoreTick" - 1106ms. Q: 7/13.
[00001,26,06:35:46.677] PROJECTIONS: Scheduling the writing of "$faulted" to "$projections-$master". Current status of Writer: Busy: true
[00001,26,06:35:46.697] PROJECTIONS: Finished writing events to "$projections-$master": "$faulted"
[00001,24,06:35:46.697] PROJECTIONS: Response received: 1095873@"$faulted"
[00001,24,06:35:46.698] The '"lk-payments-2"' projection faulted due to '"Record is too big
Parameter name: data
  at EventStore.Core.Data.Event..ctor (System.Guid eventId, System.String eventType, System.Boolean isJson, System.Byte[] data, System.Byte[] metadata) [0x0009f] in <53d7a0232f974186a44900c67be2f4b8>:0 
  at EventStore.Projections.Core.Services.Processing.CoreProjectionCheckpointWriter.BeginWriteCheckpoint (EventStore.Core.Messaging.IEnvelope envelope, EventStore.Projections.Core.Services.Processing.CheckpointTag requestedCheckpointPosition, System.String requestedCheckpointState) [0x0003f] in <14e7a544419e498182e450f39a3aee52>:0 
  at EventStore.Projections.Core.Services.Processing.DefaultCheckpointManager.BeginWriteCheckpoint (EventStore.Projections.Core.Services.Processing.CheckpointTag requestedCheckpointPosition, System.String requestedCheckpointState) [0x00013] in <14e7a544419e498182e450f39a3aee52>:0 
  at EventStore.Projections.Core.Services.Processing.CoreProjectionCheckpointManager.Handle (CoreProjectionProcessingMessage+ReadyForCheckpoint message) [0x00040] in <14e7a544419e498182e450f39a3aee52>:0 
  at EventStore.Projections.Core.Services.Processing.ProjectionCheckpoint.OnCheckpointCompleted () [0x00014] in <14e7a544419e498182e450f39a3aee52>:0 
  at EventStore.Projections.Core.Services.Processing.ProjectionCheckpoint.Prepare (EventStore.Projections.Core.Services.Processing.CheckpointTag position) [0x0007c] in <14e7a544419e498182e450f39a3aee52>:0 
  at EventStore.Projections.Core.Services.Processing.CoreProjectionCheckpointManager.StartCheckpoint (EventStore.Projections.Core.Services.Processing.PositionTracker lastProcessedEventPosition, EventStore.Projections.Core.Services.Processing.PartitionState projectionState) [0x00061] in <14e7a544419e498182e450f39a3aee52>:0 
  at EventStore.Projections.Core.Services.Processing.CoreProjectionCheckpointManager.RequestCheckpoint (EventStore.Projections.Core.Services.Processing.PositionTracker lastProcessedEventPosition, System.Boolean forcePrepareCheckpoint) [0x00029] in <14e7a544419e498182e450f39a3aee52>:0 
  at EventStore.Projections.Core.Services.Processing.CoreProjectionCheckpointManager.CheckpointSuggested (EventStore.Projections.Core.Services.Processing.CheckpointTag checkpointTag, System.Single progress) [0x00051] in <14e7a544419e498182e450f39a3aee52>:0 
  at EventStore.Projections.Core.Services.Processing.CheckpointSuggestedWorkItem.WriteOutput () [0x00028] in <14e7a544419e498182e450f39a3aee52>:0 
  at EventStore.Projections.Core.Services.Processing.WorkItem.Process (System.Int32 onStage, System.Action`2[T1,T2] readyForStage) [0x00086] in <14e7a544419e498182e450f39a3aee52>:0 
  at EventStore.Projections.Core.Services.Processing.StagedProcessingQueue.ProcessEntry (EventStore.Projections.Core.Services.Processing.StagedProcessingQueue+TaskEntry entry) [0x00037] in <14e7a544419e498182e450f39a3aee52>:0 
  at EventStore.Projections.Core.Services.Processing.StagedProcessingQueue.Process (System.Int32 max) [0x0001c] in <14e7a544419e498182e450f39a3aee52>:0 
  at EventStore.Projections.Core.Services.Processing.CoreProjectionQueue.ProcessOneEventBatch () [0x00019] in <14e7a544419e498182e450f39a3aee52>:0 
  at EventStore.Projections.Core.Services.Processing.CoreProjectionQueue.ProcessEvent () [0x00010] in <14e7a544419e498182e450f39a3aee52>:0 
  at EventStore.Projections.Core.Services.Processing.EventSubscriptionBasedProjectionProcessingPhase.ProcessEvent () [0x00000] in <14e7a544419e498182e450f39a3aee52>:0 
  at EventStore.Projections.Core.Services.Processing.CoreProjection.Tick () [0x0002c] in <14e7a544419e498182e450f39a3aee52>:0 "'
[00001,24,06:39:48.116] Disabling '"lk-payments-2"' projection
[00001,24,06:39:48.148] '"lk-payments-2"' projection source has been written

Does it mean that data in the source stream is corrupt?
How do I fix it?
When viewing the data in the stream around the place where it fails, it looks completely normal, all events are around 1k in size.

2

It means that the size of the written projection checkpoint exceeds what is supported in terms of event size on disk (data length + metadata length + event type length). It has nothing to do with the source stream. This is fairly hard to diagnose without more details of what your projection is doing. It may be worth looking at the size of the last good checkpoint of that projection to get a clue. If I have to take a wild guess, I’d look at the size of the state that you are producing in your projection.

1 Like
3

Thank you for the quick response!
The projection state could indeed be quite large, I was trying to accumulate events of certain type in the projection.
What is the limit for the projection state size?

4

That would be +/- 16MB.