Crash eventstore

EventStoreDB
1

Hi,

I’m facing an issue on the database, and I’m not able to start again the service:

I still investigating

Below the error log:

[PID:00733:088 2018.09.22 10:51:13.615 ERROR TcpDispatcher ] Error while wrapping message EventStore.Core.Messages.ClientMessage+PersistentSubscriptionStreamEventAppeared.

System.NullReferenceException: Object reference not set to an instance of an object

at ProtoBuf.ProtoWriter.WriteString (System.String value, ProtoBuf.ProtoWriter writer) [0x0001e] in :0

at (wrapper dynamic-method) EventStore.Core.Messages.TcpClientMessageDto+EventRecord:proto_15 (object,ProtoBuf.ProtoWriter)

at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Write (System.Object value, ProtoBuf.ProtoWriter dest) [0x00000] in :0

at ProtoBuf.Meta.RuntimeTypeModel.Serialize (System.Int32 key, System.Object value, ProtoBuf.ProtoWriter dest) [0x00016] in :0

at ProtoBuf.ProtoWriter.WriteObject (System.Object value, System.Int32 key, ProtoBuf.ProtoWriter writer) [0x0001f] in :0

at (wrapper dynamic-method) EventStore.Core.Messages.TcpClientMessageDto+ResolvedIndexedEvent:proto_13 (object,ProtoBuf.ProtoWriter)

at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Write (System.Object value, ProtoBuf.ProtoWriter dest) [0x00000] in :0

at ProtoBuf.Meta.RuntimeTypeModel.Serialize (System.Int32 key, System.Object value, ProtoBuf.ProtoWriter dest) [0x00016] in :0

at ProtoBuf.ProtoWriter.WriteObject (System.Object value, System.Int32 key, ProtoBuf.ProtoWriter writer) [0x0001f] in :0

at (wrapper dynamic-method) EventStore.Core.Messages.TcpClientMessageDto+PersistentSubscriptionStreamEventAppeared:proto_11 (object,ProtoBuf.ProtoWriter)

at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Write (System.Object value, ProtoBuf.ProtoWriter dest) [0x00000] in :0

at ProtoBuf.Meta.RuntimeTypeModel.Serialize (System.Int32 key, System.Object value, ProtoBuf.ProtoWriter dest) [0x00016] in :0

at ProtoBuf.Meta.TypeModel.SerializeCore (ProtoBuf.ProtoWriter writer, System.Object value) [0x00022] in :0

at ProtoBuf.Meta.TypeModel.Serialize (System.IO.Stream dest, System.Object value, ProtoBuf.SerializationContext context) [0x00010] in :0

at ProtoBuf.Meta.TypeModel.Serialize (System.IO.Stream dest, System.Object value) [0x00000] in :0

at ProtoBuf.Serializer.Serialize[T] (System.IO.Stream destination, T instance) [0x0000d] in :0

at EventStore.Core.Services.Transport.Tcp.ProtobufExtensions.Serialize[T] (T protoContract) [0x00008] in <1dff67fb68564269afea7f7d0c2d2519>:0

at EventStore.Core.Services.Transport.Tcp.ClientTcpDispatcher.WrapPersistentSubscriptionStreamEventAppeared (EventStore.Core.Messages.ClientMessage+PersistentSubscriptionStreamEventAppeared msg) [0x00032] in <1dff67fb68564269afea7f7d0c2d2519>:0

at (wrapper delegate-invoke) System.Func`2[EventStore.Core.Messages.ClientMessage+PersistentSubscriptionStreamEventAppeared,EventStore.Core.Services.Transport.Tcp.TcpPackage]:invoke_TResult_T (EventStore.Core.Messages.ClientMessage/PersistentSubscriptionStreamEventAppeared)

at EventStore.Core.Services.Transport.Tcp.TcpDispatcher+c__AnonStorey0`1[T].<>m__0 (EventStore.Core.Messaging.Message x) [0x00000] in <1dff67fb68564269afea7f7d0c2d2519>:0

at (wrapper delegate-invoke) System.Func`2[EventStore.Core.Messaging.Message,EventStore.Core.Services.Transport.Tcp.TcpPackage]:invoke_TResult_T (EventStore.Core.Messaging.Message)

at EventStore.Core.Services.Transport.Tcp.TcpDispatcher.WrapMessage (EventStore.Core.Messaging.Message message, System.Byte version) [0x0002b] in <1dff67fb68564269afea7f7d0c2d2519>:0

[PID:00733:007 2018.09.22 10:51:23.671 ERROR TcpDispatcher ] Error while wrapping message EventStore.Core.Messages.ClientMessage+PersistentSubscriptionStreamEventAppeared.

System.NullReferenceException: Object reference not set to an instance of an object

at ProtoBuf.ProtoWriter.WriteString (System.String value, ProtoBuf.ProtoWriter writer) [0x0001e] in :0

at (wrapper dynamic-method) EventStore.Core.Messages.TcpClientMessageDto+EventRecord:proto_15 (object,ProtoBuf.ProtoWriter)

at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Write (System.Object value, ProtoBuf.ProtoWriter dest) [0x00000] in :0

at ProtoBuf.Meta.RuntimeTypeModel.Serialize (System.Int32 key, System.Object value, ProtoBuf.ProtoWriter dest) [0x00016] in :0

at ProtoBuf.ProtoWriter.WriteObject (System.Object value, System.Int32 key, ProtoBuf.ProtoWriter writer) [0x0001f] in :0

at (wrapper dynamic-method) EventStore.Core.Messages.TcpClientMessageDto+ResolvedIndexedEvent:proto_13 (object,ProtoBuf.ProtoWriter)

at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Write (System.Object value, ProtoBuf.ProtoWriter dest) [0x00000] in :0

at ProtoBuf.Meta.RuntimeTypeModel.Serialize (System.Int32 key, System.Object value, ProtoBuf.ProtoWriter dest) [0x00016] in :0

at ProtoBuf.ProtoWriter.WriteObject (System.Object value, System.Int32 key, ProtoBuf.ProtoWriter writer) [0x0001f] in :0

at (wrapper dynamic-method) EventStore.Core.Messages.TcpClientMessageDto+PersistentSubscriptionStreamEventAppeared:proto_11 (object,ProtoBuf.ProtoWriter)

at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Write (System.Object value, ProtoBuf.ProtoWriter dest) [0x00000] in :0

at ProtoBuf.Meta.RuntimeTypeModel.Serialize (System.Int32 key, System.Object value, ProtoBuf.ProtoWriter dest) [0x00016] in :0

at ProtoBuf.Meta.TypeModel.SerializeCore (ProtoBuf.ProtoWriter writer, System.Object value) [0x00022] in :0

at ProtoBuf.Meta.TypeModel.Serialize (System.IO.Stream dest, System.Object value, ProtoBuf.SerializationContext context) [0x00010] in :0

at ProtoBuf.Meta.TypeModel.Serialize (System.IO.Stream dest, System.Object value) [0x00000] in :0

at ProtoBuf.Serializer.Serialize[T] (System.IO.Stream destination, T instance) [0x0000d] in :0

at EventStore.Core.Services.Transport.Tcp.ProtobufExtensions.Serialize[T] (T protoContract) [0x00008] in <1dff67fb68564269afea7f7d0c2d2519>:0

at EventStore.Core.Services.Transport.Tcp.ClientTcpDispatcher.WrapPersistentSubscriptionStreamEventAppeared (EventStore.Core.Messages.ClientMessage+PersistentSubscriptionStreamEventAppeared msg) [0x00032] in <1dff67fb68564269afea7f7d0c2d2519>:0

at (wrapper delegate-invoke) System.Func`2[EventStore.Core.Messages.ClientMessage+PersistentSubscriptionStreamEventAppeared,EventStore.Core.Services.Transport.Tcp.TcpPackage]:invoke_TResult_T (EventStore.Core.Messages.ClientMessage/PersistentSubscriptionStreamEventAppeared)

at EventStore.Core.Services.Transport.Tcp.TcpDispatcher+c__AnonStorey0`1[T].<>m__0 (EventStore.Core.Messaging.Message x) [0x00000] in <1dff67fb68564269afea7f7d0c2d2519>:0

at (wrapper delegate-invoke) System.Func`2[EventStore.Core.Messaging.Message,EventStore.Core.Services.Transport.Tcp.TcpPackage]:invoke_TResult_T (EventStore.Core.Messaging.Message)

at EventStore.Core.Services.Transport.Tcp.TcpDispatcher.WrapMessage (EventStore.Core.Messaging.Message message, System.Byte version) [0x0002b] in <1dff67fb68564269afea7f7d0c2d2519>:0

[PID:00733:058 2018.09.22 10:51:33.721 ERROR TcpDispatcher ] Error while wrapping message EventStore.Core.Messages.ClientMessage+PersistentSubscriptionStreamEventAppeared.

System.NullReferenceException: Object reference not set to an instance of an object

at ProtoBuf.ProtoWriter.WriteString (System.String value, ProtoBuf.ProtoWriter writer) [0x0001e] in :0

at (wrapper dynamic-method) EventStore.Core.Messages.TcpClientMessageDto+EventRecord:proto_15 (object,ProtoBuf.ProtoWriter)

at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Write (System.Object value, ProtoBuf.ProtoWriter dest) [0x00000] in :0

at ProtoBuf.Meta.RuntimeTypeModel.Serialize (System.Int32 key, System.Object value, ProtoBuf.ProtoWriter dest) [0x00016] in :0

at ProtoBuf.ProtoWriter.WriteObject (System.Object value, System.Int32 key, ProtoBuf.ProtoWriter writer) [0x0001f] in :0

at (wrapper dynamic-method) EventStore.Core.Messages.TcpClientMessageDto+ResolvedIndexedEvent:proto_13 (object,ProtoBuf.ProtoWriter)

at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Write (System.Object value, ProtoBuf.ProtoWriter dest) [0x00000] in :0

at ProtoBuf.Meta.RuntimeTypeModel.Serialize (System.Int32 key, System.Object value, ProtoBuf.ProtoWriter dest) [0x00016] in :0

at ProtoBuf.ProtoWriter.WriteObject (System.Object value, System.Int32 key, ProtoBuf.ProtoWriter writer) [0x0001f] in :0

at (wrapper dynamic-method) EventStore.Core.Messages.TcpClientMessageDto+PersistentSubscriptionStreamEventAppeared:proto_11 (object,ProtoBuf.ProtoWriter)

at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Write (System.Object value, ProtoBuf.ProtoWriter dest) [0x00000] in :0

at ProtoBuf.Meta.RuntimeTypeModel.Serialize (System.Int32 key, System.Object value, ProtoBuf.ProtoWriter dest) [0x00016] in :0

at ProtoBuf.Meta.TypeModel.SerializeCore (ProtoBuf.ProtoWriter writer, System.Object value) [0x00022] in :0

at ProtoBuf.Meta.TypeModel.Serialize (System.IO.Stream dest, System.Object value, ProtoBuf.SerializationContext context) [0x00010] in :0

at ProtoBuf.Meta.TypeModel.Serialize (System.IO.Stream dest, System.Object value) [0x00000] in :0

at ProtoBuf.Serializer.Serialize[T] (System.IO.Stream destination, T instance) [0x0000d] in :0

at EventStore.Core.Services.Transport.Tcp.ProtobufExtensions.Serialize[T] (T protoContract) [0x00008] in <1dff67fb68564269afea7f7d0c2d2519>:0

at EventStore.Core.Services.Transport.Tcp.ClientTcpDispatcher.WrapPersistentSubscriptionStreamEventAppeared (EventStore.Core.Messages.ClientMessage+PersistentSubscriptionStreamEventAppeared msg) [0x00032] in <1dff67fb68564269afea7f7d0c2d2519>:0

at (wrapper delegate-invoke) System.Func`2[EventStore.Core.Messages.ClientMessage+PersistentSubscriptionStreamEventAppeared,EventStore.Core.Services.Transport.Tcp.TcpPackage]:invoke_TResult_T (EventStore.Core.Messages.ClientMessage/PersistentSubscriptionStreamEventAppeared)

at EventStore.Core.Services.Transport.Tcp.TcpDispatcher+c__AnonStorey0`1[T].<>m__0 (EventStore.Core.Messaging.Message x) [0x00000] in <1dff67fb68564269afea7f7d0c2d2519>:0

at (wrapper delegate-invoke) System.Func`2[EventStore.Core.Messaging.Message,EventStore.Core.Services.Transport.Tcp.TcpPackage]:invoke_TResult_T (EventStore.Core.Messaging.Message)

at EventStore.Core.Services.Transport.Tcp.TcpDispatcher.WrapMessage (EventStore.Core.Messaging.Message message, System.Byte version) [0x0002b] in <1dff67fb68564269afea7f7d0c2d2519>:0

[PID:00733:020 2018.09.22 10:51:43.792 ERROR TcpDispatcher ] Error while wrapping message EventStore.Core.Messages.ClientMessage+PersistentSubscriptionStreamEventAppeared.

System.NullReferenceException: Object reference not set to an instance of an object

at ProtoBuf.ProtoWriter.WriteString (System.String value, ProtoBuf.ProtoWriter writer) [0x0001e] in :0

at (wrapper dynamic-method) EventStore.Core.Messages.TcpClientMessageDto+EventRecord:proto_15 (object,ProtoBuf.ProtoWriter)

at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Write (System.Object value, ProtoBuf.ProtoWriter dest) [0x00000] in :0

at ProtoBuf.Meta.RuntimeTypeModel.Serialize (System.Int32 key, System.Object value, ProtoBuf.ProtoWriter dest) [0x00016] in :0

at ProtoBuf.ProtoWriter.WriteObject (System.Object value, System.Int32 key, ProtoBuf.ProtoWriter writer) [0x0001f] in :0

at (wrapper dynamic-method) EventStore.Core.Messages.TcpClientMessageDto+ResolvedIndexedEvent:proto_13 (object,ProtoBuf.ProtoWriter)

at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Write (System.Object value, ProtoBuf.ProtoWriter dest) [0x00000] in :0

at ProtoBuf.Meta.RuntimeTypeModel.Serialize (System.Int32 key, System.Object value, ProtoBuf.ProtoWriter dest) [0x00016] in :0

at ProtoBuf.ProtoWriter.WriteObject (System.Object value, System.Int32 key, ProtoBuf.ProtoWriter writer) [0x0001f] in :0

at (wrapper dynamic-method) EventStore.Core.Messages.TcpClientMessageDto+PersistentSubscriptionStreamEventAppeared:proto_11 (object,ProtoBuf.ProtoWriter)

at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Write (System.Object value, ProtoBuf.ProtoWriter dest) [0x00000] in :0

at ProtoBuf.Meta.RuntimeTypeModel.Serialize (System.Int32 key, System.Object value, ProtoBuf.ProtoWriter dest) [0x00016] in :0

at ProtoBuf.Meta.TypeModel.SerializeCore (ProtoBuf.ProtoWriter writer, System.Object value) [0x00022] in :0

at ProtoBuf.Meta.TypeModel.Serialize (System.IO.Stream dest, System.Object value, ProtoBuf.SerializationContext context) [0x00010] in :0

at ProtoBuf.Meta.TypeModel.Serialize (System.IO.Stream dest, System.Object value) [0x00000] in :0

at ProtoBuf.Serializer.Serialize[T] (System.IO.Stream destination, T instance) [0x0000d] in :0

at EventStore.Core.Services.Transport.Tcp.ProtobufExtensions.Serialize[T] (T protoContract) [0x00008] in <1dff67fb68564269afea7f7d0c2d2519>:0

at EventStore.Core.Services.Transport.Tcp.ClientTcpDispatcher.WrapPersistentSubscriptionStreamEventAppeared (EventStore.Core.Messages.ClientMessage+PersistentSubscriptionStreamEventAppeared msg) [0x00032] in <1dff67fb68564269afea7f7d0c2d2519>:0

at (wrapper delegate-invoke) System.Func`2[EventStore.Core.Messages.ClientMessage+PersistentSubscriptionStreamEventAppeared,EventStore.Core.Services.Transport.Tcp.TcpPackage]:invoke_TResult_T (EventStore.Core.Messages.ClientMessage/PersistentSubscriptionStreamEventAppeared)

at EventStore.Core.Services.Transport.Tcp.TcpDispatcher+c__AnonStorey0`1[T].<>m__0 (EventStore.Core.Messaging.Message x) [0x00000] in <1dff67fb68564269afea7f7d0c2d2519>:0

at (wrapper delegate-invoke) System.Func`2[EventStore.Core.Messaging.Message,EventStore.Core.Services.Transport.Tcp.TcpPackage]:invoke_TResult_T (EventStore.Core.Messaging.Message)

at EventStore.Core.Services.Transport.Tcp.TcpDispatcher.WrapMessage (EventStore.Core.Messaging.Message message, System.Byte version) [0x0002b] in <1dff67fb68564269afea7f7d0c2d2519>:0

[PID:00733:087 2018.09.22 10:51:53.857 ERROR TcpDispatcher ] Error while wrapping message EventStore.Core.Messages.ClientMessage+PersistentSubscriptionStreamEventAppeared.

System.NullReferenceException: Object reference not set to an instance of an object

at ProtoBuf.ProtoWriter.WriteString (System.String value, ProtoBuf.ProtoWriter writer) [0x0001e] in :0

at (wrapper dynamic-method) EventStore.Core.Messages.TcpClientMessageDto+EventRecord:proto_15 (object,ProtoBuf.ProtoWriter)

at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Write (System.Object value, ProtoBuf.ProtoWriter dest) [0x00000] in :0

at ProtoBuf.Meta.RuntimeTypeModel.Serialize (System.Int32 key, System.Object value, ProtoBuf.ProtoWriter dest) [0x00016] in :0

at ProtoBuf.ProtoWriter.WriteObject (System.Object value, System.Int32 key, ProtoBuf.ProtoWriter writer) [0x0001f] in :0

at (wrapper dynamic-method) EventStore.Core.Messages.TcpClientMessageDto+ResolvedIndexedEvent:proto_13 (object,ProtoBuf.ProtoWriter)

at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Write (System.Object value, ProtoBuf.ProtoWriter dest) [0x00000] in :0

at ProtoBuf.Meta.RuntimeTypeModel.Serialize (System.Int32 key, System.Object value, ProtoBuf.ProtoWriter dest) [0x00016] in :0

at ProtoBuf.ProtoWriter.WriteObject (System.Object value, System.Int32 key, ProtoBuf.ProtoWriter writer) [0x0001f] in :0

at (wrapper dynamic-method) EventStore.Core.Messages.TcpClientMessageDto+PersistentSubscriptionStreamEventAppeared:proto_11 (object,ProtoBuf.ProtoWriter)

at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Write (System.Object value, ProtoBuf.ProtoWriter dest) [0x00000] in :0

at ProtoBuf.Meta.RuntimeTypeModel.Serialize (System.Int32 key, System.Object value, ProtoBuf.ProtoWriter dest) [0x00016] in :0

at ProtoBuf.Meta.TypeModel.SerializeCore (ProtoBuf.ProtoWriter writer, System.Object value) [0x00022] in :0

at ProtoBuf.Meta.TypeModel.Serialize (System.IO.Stream dest, System.Object value, ProtoBuf.SerializationContext context) [0x00010] in :0

at ProtoBuf.Meta.TypeModel.Serialize (System.IO.Stream dest, System.Object value) [0x00000] in :0

at ProtoBuf.Serializer.Serialize[T] (System.IO.Stream destination, T instance) [0x0000d] in :0

at EventStore.Core.Services.Transport.Tcp.ProtobufExtensions.Serialize[T] (T protoContract) [0x00008] in <1dff67fb68564269afea7f7d0c2d2519>:0

at EventStore.Core.Services.Transport.Tcp.ClientTcpDispatcher.WrapPersistentSubscriptionStreamEventAppeared (EventStore.Core.Messages.ClientMessage+PersistentSubscriptionStreamEventAppeared msg) [0x00032] in <1dff67fb68564269afea7f7d0c2d2519>:0

at (wrapper delegate-invoke) System.Func`2[EventStore.Core.Messages.ClientMessage+PersistentSubscriptionStreamEventAppeared,EventStore.Core.Services.Transport.Tcp.TcpPackage]:invoke_TResult_T (EventStore.Core.Messages.ClientMessage/PersistentSubscriptionStreamEventAppeared)

at EventStore.Core.Services.Transport.Tcp.TcpDispatcher+c__AnonStorey0`1[T].<>m__0 (EventStore.Core.Messaging.Message x) [0x00000] in <1dff67fb68564269afea7f7d0c2d2519>:0

at (wrapper delegate-invoke) System.Func`2[EventStore.Core.Messaging.Message,EventStore.Core.Services.Transport.Tcp.TcpPackage]:invoke_TResult_T (EventStore.Core.Messaging.Message)

at EventStore.Core.Services.Transport.Tcp.TcpDispatcher.WrapMessage (EventStore.Core.Messaging.Message message, System.Byte version) [0x0002b] in <1dff67fb68564269afea7f7d0c2d2519>:0

[PID:00733:056 2018.09.22 10:52:03.901 ERROR TcpDispatcher ] Error while wrapping message EventStore.Core.Messages.ClientMessage+PersistentSubscriptionStreamEventAppeared.

System.NullReferenceException: Object reference not set to an instance of an object

at ProtoBuf.ProtoWriter.WriteString (System.String value, ProtoBuf.ProtoWriter writer) [0x0001e] in :0

at (wrapper dynamic-method) EventStore.Core.Messages.TcpClientMessageDto+EventRecord:proto_15 (object,ProtoBuf.ProtoWriter)

at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Write (System.Object value, ProtoBuf.ProtoWriter dest) [0x00000] in :0

at ProtoBuf.Meta.RuntimeTypeModel.Serialize (System.Int32 key, System.Object value, ProtoBuf.ProtoWriter dest) [0x00016] in :0

at ProtoBuf.ProtoWriter.WriteObject (System.Object value, System.Int32 key, ProtoBuf.ProtoWriter writer) [0x0001f] in :0

at (wrapper dynamic-method) EventStore.Core.Messages.TcpClientMessageDto+ResolvedIndexedEvent:proto_13 (object,ProtoBuf.ProtoWriter)

at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Write (System.Object value, ProtoBuf.ProtoWriter dest) [0x00000] in :0

at ProtoBuf.Meta.RuntimeTypeModel.Serialize (System.Int32 key, System.Object value, ProtoBuf.ProtoWriter dest) [0x00016] in :0

at ProtoBuf.ProtoWriter.WriteObject (System.Object value, System.Int32 key, ProtoBuf.ProtoWriter writer) [0x0001f] in :0

at (wrapper dynamic-method) EventStore.Core.Messages.TcpClientMessageDto+PersistentSubscriptionStreamEventAppeared:proto_11 (object,ProtoBuf.ProtoWriter)

at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Write (System.Object value, ProtoBuf.ProtoWriter dest) [0x00000] in :0

at ProtoBuf.Meta.RuntimeTypeModel.Serialize (System.Int32 key, System.Object value, ProtoBuf.ProtoWriter dest) [0x00016] in :0

at ProtoBuf.Meta.TypeModel.SerializeCore (ProtoBuf.ProtoWriter writer, System.Object value) [0x00022] in :0

at ProtoBuf.Meta.TypeModel.Serialize (System.IO.Stream dest, System.Object value, ProtoBuf.SerializationContext context) [0x00010] in :0

at ProtoBuf.Meta.TypeModel.Serialize (System.IO.Stream dest, System.Object value) [0x00000] in :0

at ProtoBuf.Serializer.Serialize[T] (System.IO.Stream destination, T instance) [0x0000d] in :0

at EventStore.Core.Services.Transport.Tcp.ProtobufExtensions.Serialize[T] (T protoContract) [0x00008] in <1dff67fb68564269afea7f7d0c2d2519>:0

at EventStore.Core.Services.Transport.Tcp.ClientTcpDispatcher.WrapPersistentSubscriptionStreamEventAppeared (EventStore.Core.Messages.ClientMessage+PersistentSubscriptionStreamEventAppeared msg) [0x00032] in <1dff67fb68564269afea7f7d0c2d2519>:0

at (wrapper delegate-invoke) System.Func`2[EventStore.Core.Messages.ClientMessage+PersistentSubscriptionStreamEventAppeared,EventStore.Core.Services.Transport.Tcp.TcpPackage]:invoke_TResult_T (EventStore.Core.Messages.ClientMessage/PersistentSubscriptionStreamEventAppeared)

at EventStore.Core.Services.Transport.Tcp.TcpDispatcher+c__AnonStorey0`1[T].<>m__0 (EventStore.Core.Messaging.Message x) [0x00000] in <1dff67fb68564269afea7f7d0c2d2519>:0

at (wrapper delegate-invoke) System.Func`2[EventStore.Core.Messaging.Message,EventStore.Core.Services.Transport.Tcp.TcpPackage]:invoke_TResult_T (EventStore.Core.Messaging.Message)

at EventStore.Core.Services.Transport.Tcp.TcpDispatcher.WrapMessage (EventStore.Core.Messaging.Message message, System.Byte version) [0x0002b] in <1dff67fb68564269afea7f7d0c2d2519>:0

[PID:00733:043 2018.09.22 10:52:13.964 ERROR TcpDispatcher ] Error while wrapping message EventStore.Core.Messages.ClientMessage+PersistentSubscriptionStreamEventAppeared.

System.NullReferenceException: Object reference not set to an instance of an object

at ProtoBuf.ProtoWriter.WriteString (System.String value, ProtoBuf.ProtoWriter writer) [0x0001e] in :0

at (wrapper dynamic-method) EventStore.Core.Messages.TcpClientMessageDto+EventRecord:proto_15 (object,ProtoBuf.ProtoWriter)

at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Write (System.Object value, ProtoBuf.ProtoWriter dest) [0x00000] in :0

at ProtoBuf.Meta.RuntimeTypeModel.Serialize (System.Int32 key, System.Object value, ProtoBuf.ProtoWriter dest) [0x00016] in :0

at ProtoBuf.ProtoWriter.WriteObject (System.Object value, System.Int32 key, ProtoBuf.ProtoWriter writer) [0x0001f] in :0

at (wrapper dynamic-method) EventStore.Core.Messages.TcpClientMessageDto+ResolvedIndexedEvent:proto_13 (object,ProtoBuf.ProtoWriter)

at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Write (System.Object value, ProtoBuf.ProtoWriter dest) [0x00000] in :0

at ProtoBuf.Meta.RuntimeTypeModel.Serialize (System.Int32 key, System.Object value, ProtoBuf.ProtoWriter dest) [0x00016] in :0

at ProtoBuf.ProtoWriter.WriteObject (System.Object value, System.Int32 key, ProtoBuf.ProtoWriter writer) [0x0001f] in :0

at (wrapper dynamic-method) EventStore.Core.Messages.TcpClientMessageDto+PersistentSubscriptionStreamEventAppeared:proto_11 (object,ProtoBuf.ProtoWriter)

at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Write (System.Object value, ProtoBuf.ProtoWriter dest) [0x00000] in :0

at ProtoBuf.Meta.RuntimeTypeModel.Serialize (System.Int32 key, System.Object value, ProtoBuf.ProtoWriter dest) [0x00016] in :0

at ProtoBuf.Meta.TypeModel.SerializeCore (ProtoBuf.ProtoWriter writer, System.Object value) [0x00022] in :0

at ProtoBuf.Meta.TypeModel.Serialize (System.IO.Stream dest, System.Object value, ProtoBuf.SerializationContext context) [0x00010] in :0

at ProtoBuf.Meta.TypeModel.Serialize (System.IO.Stream dest, System.Object value) [0x00000] in :0

at ProtoBuf.Serializer.Serialize[T] (System.IO.Stream destination, T instance) [0x0000d] in :0

at EventStore.Core.Services.Transport.Tcp.ProtobufExtensions.Serialize[T] (T protoContract) [0x00008] in <1dff67fb68564269afea7f7d0c2d2519>:0

at EventStore.Core.Services.Transport.Tcp.ClientTcpDispatcher.WrapPersistentSubscriptionStreamEventAppeared (EventStore.Core.Messages.ClientMessage+PersistentSubscriptionStreamEventAppeared msg) [0x00032] in <1dff67fb68564269afea7f7d0c2d2519>:0

at (wrapper delegate-invoke) System.Func`2[EventStore.Core.Messages.ClientMessage+PersistentSubscriptionStreamEventAppeared,EventStore.Core.Services.Transport.Tcp.TcpPackage]:invoke_TResult_T (EventStore.Core.Messages.ClientMessage/PersistentSubscriptionStreamEventAppeared)

at EventStore.Core.Services.Transport.Tcp.TcpDispatcher+c__AnonStorey0`1[T].<>m__0 (EventStore.Core.Messaging.Message x) [0x00000] in <1dff67fb68564269afea7f7d0c2d2519>:0

at (wrapper delegate-invoke) System.Func`2[EventStore.Core.Messaging.Message,EventStore.Core.Services.Transport.Tcp.TcpPackage]:invoke_TResult_T (EventStore.Core.Messaging.Message)

at EventStore.Core.Services.Transport.Tcp.TcpDispatcher.WrapMessage (EventStore.Core.Messaging.Message message, System.Byte version) [0x0002b] in <1dff67fb68564269afea7f7d0c2d2519>:0

2
  • I’m wrong database restarts
3

Here is my configuration:

Ubuntu 18.04.1 x64 LTS with the kernel: 4.15.0-1019-gcp

EventStore version 4.1.1.0 (HEAD/cd0eafec71baffbcc0b9a4bb7c58e996cf290d34, Wed, 16 May 2018 11:08:07 +0400)

The persistent subscription still crash sometimes with the following logs:

[00562,16,13:04:55.920] Checkpoint write successful for $persistentsubscription-$ce-Message::PostOffice-checkpoint

[00562,10,13:04:56.915] Publishing checkpoint for $persistentsubscription-$ce-Message::PostOffice-checkpoint: 36618466

[00562,16,13:04:56.922] Checkpoint write successful for $persistentsubscription-$ce-Message::PostOffice-checkpoint

[00562,19,13:04:57.921] Publishing checkpoint for $persistentsubscription-$ce-Message::PostOffice-checkpoint: 36618496

[00562,16,13:04:57.941] Checkpoint write successful for $persistentsubscription-$ce-Message::PostOffice-checkpoint

[00562,47,13:04:58.925] Publishing checkpoint for $persistentsubscription-$ce-Message::PostOffice-checkpoint: 36618538

[00562,16,13:04:58.938] Checkpoint write successful for $persistentsubscription-$ce-Message::PostOffice-checkpoint

[00562,20,13:04:59.936] Publishing checkpoint for $persistentsubscription-$ce-Message::PostOffice-checkpoint: 36618572

[00562,16,13:04:59.945] Checkpoint write successful for $persistentsubscription-$ce-Message::PostOffice-checkpoint

[00562,28,13:05:00.314] Switching MemTable, currently: 2 awaiting tables.

[00562,62,13:05:00.314] Awaiting tables queue size is: 2.

[00562,61,13:05:00.944] Publishing checkpoint for $persistentsubscription-$ce-Message::PostOffice-checkpoint: 36618623

[00562,16,13:05:00.964] Checkpoint write successful for $persistentsubscription-$ce-Message::PostOffice-checkpoint

[00562,61,13:05:00.985] Error while processing message EventStore.Core.Messages.ClientMessage+PersistentSubscriptionAckEvents in queued handler ‘PersistentSubscriptions’.

Object reference not set to an instance of an object

[00562,13,13:05:01.063] Error while wrapping message EventStore.Core.Messages.ClientMessage+PersistentSubscriptionStreamEventAppeared.

Object reference not set to an instance of an object

[00562,62,13:05:01.928] Cached 65536 index midpoints to PTable

[00562,62,13:05:01.987] Dumped MemTable [7d0fa6b7-ff3d-487f-aee3-6df488b1efff, 1000000 entries] in 00:00:01.6725506.

[00562,62,13:05:01.988] Loading and Verification of PTable ‘4c16e3fc-bf09-4b4f-a1ea-2da593d7826e’ started…

[00562,62,13:05:02.169] Loading PTable (Version: 4) ‘4c16e3fc-bf09-4b4f-a1ea-2da593d7826e’ (1000000 entries, cache depth 16) done in 00:00:00.1811131.

[00562,62,13:05:02.174] PTables merge started (specialized for <= 2 tables).

[00562,14,13:05:03.071] SLOW QUEUE MSG [StorageWriterQueue]: WritePrepares - 684ms. Q: 0/1.

[00562,16,13:05:03.165] SLOW BUS MSG [Core Output]: ReadAllEventsForward - 64ms. Handler: F`1.

[00562,16,13:05:03.165] SLOW BUS MSG [MainBus]: EventCommitted - 65ms. Handler: AwakeService.

[00562,16,13:05:03.165] SLOW QUEUE MSG [MainQueue]: EventCommitted - 65ms. Q: 1/4.

[00562,21,13:05:03.296] SLOW QUEUE MSG [StorageReaderQueue #4]: ReadStreamEventsBackward - 218ms. Q: 0/15.

[00562,62,13:05:04.583] Cached 65536 index midpoints to PTable

[00562,62,13:05:04.707] PTables merge finished in 00:00:02.5315241 ([1000001, 1000000] entries merged into 2000001).

[00562,62,13:05:04.708] Loading and Verification of PTable ‘5215e7ea-fbd2-4884-8797-5584a99ffd66’ started…

[00562,62,13:05:05.057] Loading PTable (Version: 4) ‘5215e7ea-fbd2-4884-8797-5584a99ffd66’ (2000001 entries, cache depth 16) done in 00:00:00.3494377.

[00562,62,13:05:05.061] There are now 1 awaiting tables.

[00562,62,13:05:05.073] Awaiting tables queue size is: 1.

Have you any hint to solve this problem ?

4

Havn’t seen this one before but will look through possible causes today

5

So a few things.

  1. This is a message failing to be wrapped. This should prevent this message from being sent but other things in the db should still be working such as reads/writes. What behaviour are you seeing?

  2. Is it possible that your actual event data is null?

6

The first crash broke others services : as I saw web admin has frozen, and no more read/write (strace -cfp). But following crash was only on the persistent subscription.

About null data, it should never append. I’ll check if it was the case.

Is there a way to jump to a deterministic position when browsing $all ?