Client looses messages when bringing down 1 cluster node from 3

EventStoreDB
#1

Hello,

I’m experimenting with a very basic EventStoreDB cluster. I have setup a cluster with 3 nodes using docker containers on the same host as described in ’ Highly-available cluster’

Every 500 ms I’m creating an event but while doing so I stop the container of the leader node. By doing so either the client hangs or the client continues but looses messages. I’m using the C# client. I will share the client code that I’m currently using so you can have a look at what I’m doing wrong.

Thanks in advance

using EventStore.Client;
using System.Text.Json;


EventStoreClient GetConnection()
{
    string connectionString = new string("esdb://127.0.0.1:2113,127.0.0.1:2112,127.0.0.1:2111?tls=false&keepAliveTimeout=10000&keepAliveInterval=10000");

    var settings = EventStoreClientSettings.Create(connectionString);
    return new EventStoreClient(settings);
}


int i = 0;

while (i < 100)
{
    var evt = new TestEvent(EntityId: Guid.NewGuid(), WhatsUp: "I wrote my first event!");
    var eventData = new EventData(
        Uuid.NewUuid(),
        "TestEvent",
        JsonSerializer.SerializeToUtf8Bytes(evt)
    );


    GetConnection().AppendToStreamAsync("my-first-stream", StreamState.Any, new[] { eventData });

    Thread.Sleep(500);
    Console.WriteLine($"Successfully written {i}");
    i++;
}

public record TestEvent(Guid EntityId, String WhatsUp);
#2

This code is a bit off.

GetConnection().AppendToStreamAsync(“my-first-stream”, StreamState.Any, new[] { eventData });

You are making an async call and not awaiting/checking/etc the result?

#3

Hi Greg,

Thanks for that. Indeed on Friday I tested a bit further and when calling the AppendToStream bit in a try catch block things seem to go better. It was not clear to me from the documentation that I had to do these checks. I thought I read somewhere in the documentation that the C# client would take care of errors and reconnecting. I must have misunderstood that. On top of that I’m new to C# (I’m used to C++) so I’m missing some of the language features like async and wait. Anyway I changed my code as shown here below. Could you confirm this is the correct way for the client to do this.

Thanks

bool committed = false;
do
{
committed = false;
try
{
var result = evdbClient.AppendToStreamAsync(“my-first-stream”, clientRevision, new[] { eventData });
await result;
committed = result.IsCompletedSuccessfully;
}
catch (Exception _)
{
Console.WriteLine(“Failed to write event, retry !!!”);
Thread.Sleep(20);
}
} while (committed != true);

    try
    {
        var result = evdbClient.AppendToStreamAsync("my-first-stream", clientRevision, new[] { eventData });
        await result;
        committed = result.IsCompletedSuccessfully;
    }
    catch (Exception _)
    {
        Console.WriteLine("Failed to write event, retry !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!");
        Thread.Sleep(20);
    }
#4

The first thing, it’s about an async call that is not awaited. This code is not idiomatic async C#:

var result = evdbClient.AppendToStreamAsync(“my-first-stream”, clientRevision, new[] { eventData });
await result;
committed = result.IsCompletedSuccessfully;

There are plenty of resources about async programming with .NET, you’d want to check those out.

Also, Read and Append operations are RPC calls and there’s no such thing as “reconnect” as there’s no “connect”. The client makes a call and if it can’t do it, the call fails. I don’t think it’s a good idea for any code that does an important database call (whatever the database would be, ESDB, MongoDB, Postgres) and doesn’t handle exceptions. The notion of “transient exception” is well-known, and tools like Polly allow you to define retry policies in your code, which will take care of handling transient exceptions.