Intermittent OOM Exception on Test Runs

EventStoreDB
1

I get this in many different places, either from mighty moose or the Cedar test runner on my machine. Never seems to happen on our build server.

Example Code:

https://github.com/damianh/Cedar/blob/e07e149c965f62d099214ecd99e0cfccce947fe3/src/Cedar.Tests/Handlers/ResolvedEventDispatcherTests.cs

Stacktrace:

Assembly: E:\projects\Cedar\src\Cedar.Tests\bin\AutoTest.Net\Cedar.Tests.dll

Test: Cedar.Handlers.ResolvedEventtDispatcherTests.When_handler_throws_Then_invoke_exception_callback

Duration: 33 milliseconds

Message:

System.OutOfMemoryException : Insufficient memory to continue the execution of the program.

Stack trace

at System.Runtime.InteropServices.Marshal.AllocHGlobal(IntPtr cb) in

at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.CreateInMemChunk(ChunkHeader chunkHeader, Int32 fileSize) in

at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.InitNew(ChunkHeader chunkHeader, Int32 fileSize) in

at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.CreateWithHeader(String filename, ChunkHeader header, Int32 fileSize, Boolean inMem) in

at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.CreateNew(String filename, Int32 chunkSize, Int32 chunkStartNumber, Int32 chunkEndNumber, Boolean isScavenged, Boolean inMem) in

at EventStore.Core.TransactionLog.Chunks.TFChunkManager.AddNewChunk() in

at EventStore.Core.TransactionLog.Chunks.TFChunkDb.Open(Boolean verifyHash, Boolean readOnly) in

at EventStore.Core.ClusterVNode…ctor(TFChunkDb db, ClusterVNodeSettings vNodeSettings, IGossipSeedSource gossipSeedSource, InfoController infoController, ISubsystem[] subsystems) in

at EventStore.ClientAPI.Embedded.EmbeddedVNodeBuilder.Build() in

at EventStore.ClientAPI.Embedded.EmbeddedVNodeBuilder.op_Implicit(EmbeddedVNodeBuilder builder) in

at Cedar.Handlers.ResolvedEventtDispatcherTests…ctor() in e:\projects\Cedar\src\Cedar.Tests\Handlers\ResolvedEventDispatcherTests.cs:line 34

2

Are you running tests in a 32 bit process by chance?

Also we should probably not allow the unmanaged memory in tests (set -c 0 equivalent)

3

It’s visual studio (which is still 32 bit) calling mightymoose, so maybe?

Am I correct when assuming this would not happen in in production, as we would always be writing to disk there?

4

Yes.

But you can check easily. There is an auto test.testrunner process just check if it’s 32 or 64 bit. You can also lower the memory usage by changing the chunk size for the tests (you likely don’t need 256mb chunks

5

Doh! It’s the 32 bit process for mighty moose.

Even so, when I run tests with our own test runner from a 64 bit powershell process I occasionally get this error message. I will look into changing the chunk size!

6

Ive never had this error with plain nunit console

7

I also get this error. It’s intermittent for me too.

I’m using the NCrunch runner and it seems the test host is x86/32bit.

Here is a partially redacted stack trace:

System.AggregateException: One or more errors occurred.

System.OutOfMemoryException: Insufficient memory to continue the execution of the program.

at System.Runtime.InteropServices.Marshal.AllocHGlobal(IntPtr cb)

at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.CreateInMemChunk(ChunkHeader chunkHeader, Int32 fileSize)

at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.InitNew(ChunkHeader chunkHeader, Int32 fileSize)

at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.CreateWithHeader(String filename, ChunkHeader header, Int32 fileSize, Boolean inMem)

at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.CreateNew(String filename, Int32 chunkSize, Int32 chunkStartNumber, Int32 chunkEndNumber, Boolean isScavenged, Boolean inMem)

at EventStore.Core.TransactionLog.Chunks.TFChunkManager.AddNewChunk()

at EventStore.Core.TransactionLog.Chunks.TFChunkDb.Open(Boolean verifyHash, Boolean readOnly)

at EventStore.Core.ClusterVNode…ctor(TFChunkDb db, ClusterVNodeSettings vNodeSettings, IGossipSeedSource gossipSeedSource, InfoController infoController, ISubsystem[] subsystems)

at EventStore.ClientAPI.Embedded.EmbeddedVNodeBuilder.Build()

at --.MicroServices.Common.General.TestingHelpers.InMemoryEventStore…ctor() in C:\dev–\microservice-poc\MicroService-Common–.MicroServices.Common.General\TestingHelpers\EmbeddedEventStore.cs:line 19

at --.–.Service.UnitTests.Itineraries.TheItinerariesController.<.ctor>b__3_0(ContainerBuilder builder) in C:\dev–\microservice-poc—Service–.--.Service.UnitTests\Itineraries\ItinerariesControllerTests.cs:line 438

at System.Action`1.Invoke(T obj)

at Autofac.Core.Lifetime.LifetimeScope.CreateScopeRestrictedRegistry(Object tag, Action`1 configurationAction)

at Autofac.Core.Lifetime.LifetimeScope.BeginLifetimeScope(Object tag, Action`1 configurationAction)

at Autofac.Core.Lifetime.LifetimeScope.BeginLifetimeScope(Action`1 configurationAction)

at Autofac.Core.Container.BeginLifetimeScope(Action`1 configurationAction)

at ControllerTests.ApiControllerTestBase`1.<>c__DisplayClass2.b__1()

at System.Lazy`1.CreateValue()

— End of stack trace from previous location where exception was thrown —

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

at System.Lazy`1.get_Value()

at ControllerTests.ApiControllerTestBase`1.Dispose(Boolean disposing)

at ControllerTests.ApiControllerTestBase`1.Dispose()

at Xunit.Sdk.ExecutionTimer.Aggregate(Action action)

at ReflectionAbstractionExtensions.DisposeTestClass(ITest test, Object testClass, IMessageBus messageBus, ExecutionTimer timer, CancellationTokenSource cancellationTokenSource)

at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass45_0.b__1()

at Xunit.Sdk.ExceptionAggregator.Run(Action code)

System.OutOfMemoryException: Insufficient memory to continue the execution of the program.

at System.Runtime.InteropServices.Marshal.AllocHGlobal(IntPtr cb)

at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.CreateInMemChunk(ChunkHeader chunkHeader, Int32 fileSize)

at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.InitNew(ChunkHeader chunkHeader, Int32 fileSize)

at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.CreateWithHeader(String filename, ChunkHeader header, Int32 fileSize, Boolean inMem)

at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.CreateNew(String filename, Int32 chunkSize, Int32 chunkStartNumber, Int32 chunkEndNumber, Boolean isScavenged, Boolean inMem)

at EventStore.Core.TransactionLog.Chunks.TFChunkManager.AddNewChunk()

at EventStore.Core.TransactionLog.Chunks.TFChunkDb.Open(Boolean verifyHash, Boolean readOnly)

at EventStore.Core.ClusterVNode…ctor(TFChunkDb db, ClusterVNodeSettings vNodeSettings, IGossipSeedSource gossipSeedSource, InfoController infoController, ISubsystem[] subsystems)

at EventStore.ClientAPI.Embedded.EmbeddedVNodeBuilder.Build()

at --.MicroServices.Common.General.TestingHelpers.InMemoryEventStore…ctor() in C:\dev–\microservice-poc\MicroService-Common–.MicroServices.Common.General\TestingHelpers\EmbeddedEventStore.cs:line 19

at --.–.Service.UnitTests.Itineraries.TheItinerariesController.<.ctor>b__3_0(ContainerBuilder builder) in C:\dev–\microservice-poc—Service–.--.Service.UnitTests\Itineraries\ItinerariesControllerTests.cs:line 438

at System.Action`1.Invoke(T obj)

at Autofac.Core.Lifetime.LifetimeScope.CreateScopeRestrictedRegistry(Object tag, Action`1 configurationAction)

at Autofac.Core.Lifetime.LifetimeScope.BeginLifetimeScope(Object tag, Action`1 configurationAction)

at Autofac.Core.Lifetime.LifetimeScope.BeginLifetimeScope(Action`1 configurationAction)

at Autofac.Core.Container.BeginLifetimeScope(Action`1 configurationAction)

at ControllerTests.ApiControllerTestBase`1.<>c__DisplayClass2.b__1()

at System.Lazy`1.CreateValue()

— End of stack trace from previous location where exception was thrown —

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

at System.Lazy`1.get_Value()

at ControllerTests.ApiControllerTestBase`1.Dispose(Boolean disposing)

at ControllerTests.ApiControllerTestBase`1.Dispose()

at Xunit.Sdk.ExecutionTimer.Aggregate(Action action)

at ReflectionAbstractionExtensions.DisposeTestClass(ITest test, Object testClass, IMessageBus messageBus, ExecutionTimer timer, CancellationTokenSource cancellationTokenSource)

at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass45_0.b__1()

at Xunit.Sdk.ExceptionAggregator.Run(Action code)

And in case you’re wondering what EmbeddedEventStore.cs is, here you go: Line 19 is “node = EmbeddedVNodeBuilder.”

public class InMemoryEventStore : IDisposable

{

    // thanks https://github.com/yreynhout/AggregateSource/blob/master/src/EventStore/AggregateSource.EventStore.IntegratedTests/Framework/EmbeddedEventStore.cs

    private bool disposed;

    private readonly ClusterVNode node;

    public InMemoryEventStore()

    {

        node = EmbeddedVNodeBuilder.

            AsSingleNode().

            OnDefaultEndpoints().

            RunInMemory().

            Build();

        node.Start();

        var tcs = new TaskCompletionSource<object>();

        node.NodeStatusChanged += (sender, args) =>

        {

            if (args.NewVNodeState == VNodeState.Master)

                tcs.SetResult(null);

        };

        tcs.Task.Wait();

        var connection = EmbeddedEventStoreConnection.Create(node);

        connection.ConnectAsync().Wait();

        Connection = connection;

    }

    public IEventStoreConnection Connection { get; private set; }

    public void Dispose()

    {

        Dispose(true);

        GC.SuppressFinalize(this);

    }

    ~InMemoryEventStore()

    {

        Dispose(false);

    }

    protected virtual void Dispose(bool disposing)

    {

        if (disposed)

            return;

        if (disposing)

        {

            // free other managed objects that implement

            // IDisposable only

            Connection?.Close();

            Connection?.Dispose();

            node?.Stop();

        }

        // release any unmanaged objects

        // set the object references to null

        disposed = true;

    }

}

Anyone got any idea why this might be happening? memory leak perhaps?

8

Try setting WithTfChunkSize(1024 * 1024) on the vnode builder. By default GES will allocate 256mb for the chunk but you really don’t need that when you are testing. Also I would not start the node in the constructor, but rather as a separate Task Start() method. I believe xunit’s test runner will construct all your test cases first, then run them.

9

"and it seems the test host is x86/32bit"

TFChunks are 256mb by default. Likely you are building many instances
and running out of 2gb

10

Thanks guys! WithTfChunkSize(1024 * 1024) made all the difference.

The results were really easy to see with all tests running in parallel. I could reliably run all tests and have 2-5 fail repeatedly.

With WithTfChunkSize(1024 * 1024) the whole suite (300+) passes every time and sometimes up to >2x faster. Nice!