EventStore Client Hangs when mixing Async and Blocking calls

EventStoreDB
1

Hi,

I have run into an issue where our application was hanging on some operations.

When hanging we would see something similar to this from EventStore.SingleNode.exe:

[11928,11,09:10:01.937] [09:10:01.937: N127.0.0.1:51427, L127.0.0.1:1113, {8563e2e4-7cbe-421e-afc4-0687bd4ab5eb}]:

Received bytes: 211, Sent bytes: 124

Send calls: 4, callbacks: 4

Receive calls: 4, callbacks: 3

Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 3

[11928,11,09:10:01.937] Connection ‘external-normal’ [127.0.0.1:51427, {8563e2e4-7cbe-421e-afc4-0687bd4ab5eb}] closed: Success.

After some experimentation I have come up with a minimal test case that exhibits this behavior. The key to the error seems to be synchronous calls inside of an async block mixed with await calls. The test below will hang on this line:

connection.ReadStreamEventsForward(“NonExistantStream1”, 0, 100, false);

I have replicated this using EventStore 2.0.1 for .Net.

My test was .Net 4.5 using the 2.0.2 EventStore.Client package.

[TestFixture]

public class EventStoreClientLockup

{

private readonly EventData _eventData =

new EventData(eventId: Guid.NewGuid(),

type: “ignore”,

isJson: false,

data: null,

metadata: null);

[Test]

public void EventstoreLockupTest()

{

var userCredentials = new UserCredentials(“admin”, “changeit”);

ConnectionSettings connectionSettings =

ConnectionSettings.Create()

.SetDefaultUserCredentials(userCredentials)

.KeepReconnecting()

.KeepRetrying();

var ipAddress = IPAddress.Parse(“127.0.0.1”);

var ipEndPoint = new IPEndPoint(ipAddress, 1113);

var connection = EventStoreConnection.Create(connectionSettings, ipEndPoint);

connection.Connect();

Lockup(connection).Wait();

}

public async Task Lockup(IEventStoreConnection connection)

{

// await the first read

await connection.ReadStreamEventsForwardAsync(“NonExistantStream1”, 0, 100, false);

string randomStreamIdToWriteTo = “LockTestStream-” + Guid.NewGuid().ToString();

// await a random write

await connection.AppendToStreamAsync(randomStreamIdToWriteTo, -1, _eventData);

// this read would succeed if awaited but not if called synchronously

// instead the code will hang

connection.ReadStreamEventsForward(“NonExistantStream1”, 0, 100, false);

}

}

Now that I understand the issue we should be able to avoid it in the future but from my understanding of the way async and await works this should not cause problems even if the code is a sub-optimal.

cheers

Andrew

2

Hmm, I’ll look into this. The blocking calls are all implemented as NonblockingVersion.Wait() if that narrows it down in the meantime.

3

Thanks James,

Yes I found that while investigating. The problem presents itself the same way if you change the last call to the async version with a .Wait()

The problem does not happen if the last call uses await like the others.

Cheers Andrew

4

Does it “hang” if its calling read on randomStreamIdToWriteTo? Or is it only nonexistentstream1?

5

OK I have been reproducing this and there is some pretty weird stuff going on in here somewhere:

public async Task Lockup(IEventStoreConnection connection)

{

// await the first read

Console.WriteLine(“reading non-existent stream”);

await connection.ReadStreamEventsForwardAsync(“NonExistantStream1”, 0, 100, false);

string randomStreamIdToWriteTo = “LockTestStream-” + Guid.NewGuid().ToString();

// await a random write

Console.WriteLine(“reading " + randomStreamIdToWriteTo + " stream”);

await connection.AppendToStreamAsync(randomStreamIdToWriteTo, -1, _eventData);

// this read would succeed if awaited but not if called synchronously

// instead the code will hang

Console.WriteLine(“reading non-existent stream”);

await connection.ReadStreamEventsForwardAsync(“NonExistantStream1”, 0, 100, false);

Console.WriteLine(“done reading non-existent stream”);

}

Does not hang

public async Task Lockup(IEventStoreConnection connection)

{

// await the first read

Console.WriteLine(“reading non-existent stream”);

await connection.ReadStreamEventsForwardAsync(“NonExistantStream1”, 0, 100, false);

string randomStreamIdToWriteTo = “LockTestStream-” + Guid.NewGuid().ToString();

// await a random write

Console.WriteLine(“reading " + randomStreamIdToWriteTo + " stream”);

await connection.AppendToStreamAsync(randomStreamIdToWriteTo, -1, _eventData);

// this read would succeed if awaited but not if called synchronously

// instead the code will hang

Console.WriteLine(“reading non-existent stream”);

var x = connection.ReadStreamEventsForwardAsync(“NonExistantStream1”, 0, 100, false).Result;

Console.WriteLine(“done reading non-existent stream”);

}

Does hang. Currently we are internally using .Result not await. I think we should change these to await as this appears to be described here as a possible deadlock circumstance due to the use of .Result http://msdn.microsoft.com/en-us/magazine/jj991977.aspx

Cheers,

Greg

6

We could change to this in time for the next RC easily enough, it’s not a lot of work.

My personal preference would be to take away the synchronous versions and make people acknowledge what they’re doing - though this would be a fairly major departure in terms of API at this stage.

I’ve opened up an issue on GitHub to change over to using await: https://github.com/EventStore/EventStore/issues/118

James

7

its not just using await (then all the methods have to be async as well)

8

to be more clear async methods must return void/task/task so its just as much a breaking change.

9

Please don’t implement async void! :wink: I’ve been bitten by that so many times.