Hi,
the system projections $by_category and $stream_by_category projections in our eventstore (4.1.1) faulted with “Exception has been thrown by the target of an invocation”.
What might be the reason for this? Do you have any pointers/ideas what we could look at or try to recover from that?
Looking at the system logs would be a good starting point.
I had a look into the err.log and there is a few very concerning messages:
“Specified cast is not valid.” && “Object reference not set to an instance of an object”.
{ “PID”: “57207”, “ThreadID”: “29”, “Date”: “2019-06-01T08:06:21.566446Z”, “Level”: “Error”, “Logger”: “ProjectionManager”, “Message”: “The ‘{projection}’ projection faulted due to ‘{e}’”, “EventProperties”: { “projection”: “$by_category”, “e”: “Exception has been thrown by the target of an invocation.” } }
{ “PID”: “57207”, “ThreadID”: “29”, “Date”: “2019-06-01T08:06:21.567016Z”, “Level”: “Error”, “Logger”: “ProjectionManager”, “Message”: “The ‘{projection}’ projection faulted due to ‘{e}’”, “EventProperties”: { “projection”: “$stream_by_category”, “e”: “Exception has been thrown by the target of an invocation.” } }
{ “PID”: “57207”, “ThreadID”: “32”, “Date”: “2019-06-01T08:06:31.396959Z”, “Level”: “Error”, “Logger”: “QueuedHandlerMRES”, “Message”: “Error while processing message {message} in queued handler ‘{queue}’.”, “Exception”: {“Message”:“Specified cast is not valid.”, “Data”:{}, “TargetSite”:“EventStore.Core.Data.StreamMetadata FromJsonReader(Newtonsoft.Json.JsonTextReader)”, “StackTrace”:" at EventStore.Core.Data.StreamMetadata.FromJsonReader (Newtonsoft.Json.JsonTextReader reader) [0x00145] in :0 \n at EventStore.Core.Data.StreamMetadata.FromJson (System.String json) [0x0000c] in :0 \n at EventStore.Projections.Core.Standard.StreamDeletedHelper.IsStreamDeletedEvent (System.String streamOrMetaStreamId, System.String eventType, System.String eventData, System.String& deletedPartitionStreamId) [0x00042] in <2ba14f5cefd5409b8b7a717e729ae5a0>:0 \n at EventStore.Projections.Core.Standard.StreamDeletedHelper.IsStreamDeletedEventOrLinkToStreamDeletedEvent (EventStore.Projections.Core.Services.Processing.ResolvedEvent resolvedEvent, System.String& deletedPartitionStreamId) [0x0001a] in <2ba14f5cefd5409b8b7a717e729ae5a0>:0 \n at EventStore.Projections.Core.Services.Processing.TransactionFileEventReader.DeliverEvent (EventStore.Core.Data.ResolvedEvent event, System.Int64 lastCommitPosition, EventStore.Core.Data.TFPos currentFrom) [0x0006b] in <2ba14f5cefd5409b8b7a717e729ae5a0>:0 \n at EventStore.Projections.Core.Services.Processing.TransactionFileEventReader.Handle (EventStore.Core.Messages.ClientMessage+ReadAllEventsForwardCompleted message) [0x000d0] in <2ba14f5cefd5409b8b7a717e729ae5a0>:0 \n at EventStore.Core.Messaging.SendToThisEnvelope.ReplyWith[T] (T message) [0x0000f] in :0 \n at EventStore.Projections.Core.Messaging.PublishToWrapEnvelop+<>c__DisplayClass3_01[T].<ReplyWith>b__0 () [0x00000] in <2ba14f5cefd5409b8b7a717e729ae5a0>:0 \n at EventStore.Projections.Core.Messaging.UnwrapEnvelopeHandler.Handle (EventStore.Projections.Core.Messaging.UnwrapEnvelopeMessage message) [0x00006] in <2ba14f5cefd5409b8b7a717e729ae5a0>:0 \n at EventStore.Core.Bus.MessageHandler1[T].TryHandle (EventStore.Core.Messaging.Message message) [0x00014] in :0 \n at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00036] in :0 \n at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 \n at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue (System.Object o) [0x0009b] in :0 ", “Source”:“EventStore.Core”, “HResult”:-2147467262}, “EventProperties”: { “message”: “EventStore.Projections.Core.Messaging.UnwrapEnvelopeMessage”, “queue”: “Projection Core #2” } }
{ “PID”: “57386”, “ThreadID”: “21”, “Date”: “2019-06-01T08:27:36.048970Z”, “Level”: “Error”, “Logger”: “QueuedHandlerMRES”, “Message”: “Error while processing message {message} in queued handler ‘{queue}’.”, “Exception”: {“Message”:“Object reference not set to an instance of an object”, “Data”:{}, “TargetSite”:“Void OnSecurityAccessGranted(System.Guid)”, “StackTrace”:" at EventStore.Core.Services.RequestManager.Managers.WriteStreamTwoPhaseRequestManager.OnSecurityAccessGranted (System.Guid internalCorrId) [0x00000] in :0 \n at EventStore.Core.Services.RequestManager.Managers.TwoPhaseRequestManagerBase.Handle (EventStore.Core.Messages.StorageMessage+CheckStreamAccessCompleted message) [0x0000d] in :0 \n at EventStore.Core.Services.RequestManager.RequestManagementService.DispatchInternal[T] (System.Guid correlationId, T message) [0x0001a] in :0 \n at EventStore.Core.Services.RequestManager.RequestManagementService.Handle (EventStore.Core.Messages.StorageMessage+CheckStreamAccessCompleted message) [0x00000] in :0 \n at EventStore.Core.Bus.MessageHandler1[T].TryHandle (EventStore.Core.Messaging.Message message) [0x00014] in <d80f9476b42347baaae00b7def8335e1>:0 \n at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00036] in <d80f9476b42347baaae00b7def8335e1>:0 \n at EventStore.Core.Services.VNode.VNodeFSMHandling1+<>c__DisplayClass4_0[TMessage].b__0 (EventStore.Core.Data.VNodeState s, EventStore.Core.Messaging.Message m) [0x00000] in :0 \n at EventStore.Core.Services.VNode.VNodeFSM.Handle (EventStore.Core.Messaging.Message message) [0x0004f] in :0 \n at EventStore.Core.Services.VNode.ClusterVNodeController.EventStore.Core.Bus.IHandle<EventStore.Core.Messaging.Message>.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 \n at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue (System.Object o) [0x0009b] in :0 ", “Source”:“EventStore.Core”, “HResult”:-2147467261}, “EventProperties”: { “message”: “EventStore.Core.Messages.StorageMessage+CheckStreamAccessCompleted”, “queue”: “MainQueue” } }
{ “PID”: “57386”, “ThreadID”: “21”, “Date”: “2019-06-01T08:27:36.076663Z”, “Level”: “Error”, “Logger”: “QueuedHandlerMRES”, “Message”: “Error while processing message {message} in queued handler ‘{queue}’.”, “Exception”: {“Message”:“Object reference not set to an instance of an object”, “Data”:{}, “TargetSite”:“Void OnSecurityAccessGranted(System.Guid)”, “StackTrace”:" at EventStore.Core.Services.RequestManager.Managers.WriteStreamTwoPhaseRequestManager.OnSecurityAccessGranted (System.Guid internalCorrId) [0x00000] in :0 \n at EventStore.Core.Services.RequestManager.Managers.TwoPhaseRequestManagerBase.Handle (EventStore.Core.Messages.StorageMessage+CheckStreamAccessCompleted message) [0x0000d] in :0 \n at EventStore.Core.Services.RequestManager.RequestManagementService.DispatchInternal[T] (System.Guid correlationId, T message) [0x0001a] in :0 \n at EventStore.Core.Services.RequestManager.RequestManagementService.Handle (EventStore.Core.Messages.StorageMessage+CheckStreamAccessCompleted message) [0x00000] in :0 \n at EventStore.Core.Bus.MessageHandler1[T].TryHandle (EventStore.Core.Messaging.Message message) [0x00014] in <d80f9476b42347baaae00b7def8335e1>:0 \n at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00036] in <d80f9476b42347baaae00b7def8335e1>:0 \n at EventStore.Core.Services.VNode.VNodeFSMHandling1+<>c__DisplayClass4_0[TMessage].b__0 (EventStore.Core.Data.VNodeState s, EventStore.Core.Messaging.Message m) [0x00000] in :0 \n at EventStore.Core.Services.VNode.VNodeFSM.Handle (EventStore.Core.Messaging.Message message) [0x0004f] in :0 \n at EventStore.Core.Services.VNode.ClusterVNodeController.EventStore.Core.Bus.IHandle<EventStore.Core.Messaging.Message>.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 \n at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue (System.Object o) [0x0009b] in :0 ", “Source”:“EventStore.Core”, “HResult”:-2147467261}, “EventProperties”: { “message”: “EventStore.Core.Messages.StorageMessage+CheckStreamAccessCompleted”, “queue”: “MainQueue” } }