Describe the bug
Under load only, we experience fatal crashes due to what looks like “Exception in writer” bugs.
“Message”:“array too small. numBytes/offset wrong.\nParameter name: array”, “ParamName”:“array”, “Data”:{}
Stack traces posted below, but link to ES code and not application code.
To Reproduce
Steps to reproduce the behavior:
- Unfortunately we are not able to reproduce this in any other environment when not under load.
EventStore details
- EventStore server version:
Release 5.0.8 - Operating system:
Linux, hosted in Azure Container instance - EventStore client version (if applicable):
EventFlow v0.81.4483.0
–
Logs
{ “PID”: “22172”, “ThreadID”: “13”, “Date”: “2021-07-05T10:36:57.090345Z”, “Level”: “Error”, “Logger”: “StorageWriterService”, “Message”: “Exception in writer.”, “Exception”: {“Message”:“array too small. numBytes/offset wrong.\nParameter name: array”, “ParamName”:“array”, “Data”:{}, “StackTrace”:" at (wrapper managed-to-native) System.IO.MonoIO.Write(intptr,byte[],int,int,System.IO.MonoIOError&)\n at System.IO.MonoIO.Write (System.Runtime.InteropServices.SafeHandle safeHandle, System.Byte[] src, System.Int32 src_offset, System.Int32 count, System.IO.MonoIOError& error) [0x00010] in <0f8aeac9d63d4b8aa575761bb4e65b79>:0 \n at System.IO.FileStream.FlushBuffer () [0x00051] in <0f8aeac9d63d4b8aa575761bb4e65b79>:0 \n at System.IO.FileStream.WriteInternal (System.Byte[] src, System.Int32 offset, System.Int32 count) [0x000b4] in <0f8aeac9d63d4b8aa575761bb4e65b79>:0 \n at System.IO.FileStream.Write (System.Byte[] array, System.Int32 offset, System.Int32 count) [0x00090] in <0f8aeac9d63d4b8aa575761bb4e65b79>:0 \n at EventStore.Core.TransactionLog.Chunks.TFChunk.WriterWorkItem.AppendData (System.Byte[] buf, System.Int32 offset, System.Int32 len) [0x00008] in :0 \n at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.WriteRawData (EventStore.Core.TransactionLog.Chunks.TFChunk.WriterWorkItem workItem, System.Byte[] buf, System.Int32 len) [0x00016] in :0 \n at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.WriteRawData (EventStore.Core.TransactionLog.Chunks.TFChunk.WriterWorkItem workItem, System.IO.MemoryStream buffer) [0x0000f] in :0 \n at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.TryAppend (EventStore.Core.TransactionLog.LogRecords.LogRecord record) [0x0008d] in :0 \n at EventStore.Core.TransactionLog.Chunks.TFChunkWriter.Write (EventStore.Core.TransactionLog.LogRecords.LogRecord record, System.Int64& newPos) [0x00000] in :0 \n at EventStore.Core.Services.Storage.StorageWriterService.WritePrepareWithRetry (EventStore.Core.TransactionLog.LogRecords.PrepareLogRecord prepare) [0x00009] in :0 \n at EventStore.Core.Services.Storage.StorageWriterService.EventStore.Core.Bus.IHandle<EventStore.Core.Messages.StorageMessage.WritePrepares>.Handle (EventStore.Core.Messages.StorageMessage+WritePrepares msg) [0x0013c] in :0 ", “HResult”:-2147024809}, “EventProperties”: { } }
{ “PID”: “22172”, “ThreadID”: “13”, “Date”: “2021-07-05T10:36:57.186202Z”, “Level”: “Fatal”, “Logger”: “StorageWriterService”, “Message”: “Unexpected error in StorageWriterService. Terminating the process…”, “Exception”:
{“Message”:“array too small. numBytes/offset wrong.\nParameter name: array”, “ParamName”:“array”, “Data”:{}, “StackTrace”:" at (wrapper managed-to-native) System.IO.MonoIO.Write(intptr,byte[],int,int,System.IO.MonoIOError&)\n at System.IO.MonoIO.Write (System.Runtime.InteropServices.SafeHandle safeHandle, System.Byte[] src, System.Int32 src_offset, System.Int32 count, System.IO.MonoIOError& error) [0x00010] in <0f8aeac9d63d4b8aa575761bb4e65b79>:0 \n at System.IO.FileStream.FlushBuffer () [0x00051] in <0f8aeac9d63d4b8aa575761bb4e65b79>:0 \n at System.IO.FileStream.WriteInternal (System.Byte[] src, System.Int32 offset, System.Int32 count) [0x000b4] in <0f8aeac9d63d4b8aa575761bb4e65b79>:0 \n at System.IO.FileStream.Write (System.Byte[] array, System.Int32 offset, System.Int32 count) [0x00090] in <0f8aeac9d63d4b8aa575761bb4e65b79>:0 \n at EventStore.Core.TransactionLog.Chunks.TFChunk.WriterWorkItem.AppendData (System.Byte[] buf, System.Int32 offset, System.Int32 len) [0x00008] in :0 \n at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.WriteRawData (EventStore.Core.TransactionLog.Chunks.TFChunk.WriterWorkItem workItem, System.Byte[] buf, System.Int32 len) [0x00016] in :0 \n at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.WriteRawData (EventStore.Core.TransactionLog.Chunks.TFChunk.WriterWorkItem workItem, System.IO.MemoryStream buffer) [0x0000f] in :0 \n at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.TryAppend (EventStore.Core.TransactionLog.LogRecords.LogRecord record) [0x0008d] in :0 \n at EventStore.Core.TransactionLog.Chunks.TFChunkWriter.Write (EventStore.Core.TransactionLog.LogRecords.LogRecord record, System.Int64& newPos) [0x00000] in :0 \n at EventStore.Core.Services.Storage.StorageWriterService.WritePrepareWithRetry (EventStore.Core.TransactionLog.LogRecords.PrepareLogRecord prepare) [0x00009] in :0 \n at EventStore.Core.Services.Storage.StorageWriterService.EventStore.Core.Bus.IHandle<EventStore.Core.Messages.StorageMessage.WritePrepares>.Handle (EventStore.Core.Messages.StorageMessage+WritePrepares msg) [0x00217] in :0 \n at EventStore.Core.Bus.MessageHandler`1[T].TryHandle (EventStore.Core.Messaging.Message message) [0x00014] in :0 \n at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00100] in :0 \n at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0 \n at EventStore.Core.Services.Storage.StorageWriterService.CommonHandle (EventStore.Core.Messaging.Message message) [0x000b9] in :0 ", “HResult”:-2147024809}, “EventProperties”: { } }
{ “PID”: “22172”, “ThreadID”: “13”, “Date”: “2021-07-05T10:36:57.186202Z”, “Level”: “Error”, “Logger”: “Application”, “Message”: “Exiting with exit code: {exitCode}.\nExit reason: {e}”, “EventProperties”: { “exitCode”: 1, “e”: “Unexpected error in StorageWriterService: array too small. numBytes/offset wrong.\nParameter name: array” } }