DeadlineExceeded errors

Hello, just getting started with ESDB. Following the the ideas and principles of your webcasts and the book by @alexey.zimarev I have the following code to check the stream is existing.

public async Task<bool> Exists<T, TId>(TId aggregateId)
{
    var stream = GetStreamName<T, TId>(aggregateId);
    var result = _esClient.ReadStreamAsync(Direction.Forwards, stream, StreamPosition.Start);
    
    return await result.ReadState != ReadState.StreamNotFound;
}

This method works as expected. It connects to the ESDB and returns the correct state. But straight after this call I get the following messages in the console

[08:07:17 WRN] gRPC call deadline exceeded.
[08:07:17 ERR] Call failed with gRPC error status. Status code: 'DeadlineExceeded', Message: ''.

This error message is directly linked to the TimeoutAfter property. It appears after the timeout configured in the AddEventStoreClient

    services.AddEventStoreClient(settings =>
    {
        settings.ConnectivitySettings.Address = new Uri("https://localhost:2113");
        settings.DefaultCredentials           = new UserCredentials("admin", "changeit");
        settings.OperationOptions = new EventStoreClientOperationOptions
        {
            TimeoutAfter = TimeSpan.FromSeconds(1)
        };
        settings.CreateHttpMessageHandler = () =>
            new SocketsHttpHandler
            {
                SslOptions =
                {
                    RemoteCertificateValidationCallback = delegate { return true; }
                }
            };
    });

Am I missing something here or what could cause these errors I see?

I’m running ESDB via docker by eventstore/eventstore:20.6.0-buster-slim. This leads to another question. Currently I’m a bit confused what to use, as this image is about 3 months old. The 5.x line seems to be more recent.

As I’m currently just prototyping and evaluating ES for new ideas I think the 20.x line might the way to go as it seems to be the future. But really not sure about this. Any input about this greatly appreciated as well.

The error might be connected to the fact that ReadyStreamAsync returns the async enumerable of events and it probably expects you to read the stream but you don’t. So, I’d suggest trying to add the count parameter to the call, which is right after the stream position and set it to one.

About Docker images, we retagged v5 image so they will have the same tag style, that’s why they look “new”. 20.6.0 is the latest public release and 20.6.1 is the upcoming release.

Unfortunately this alone does not change anything. Iterating over the event(s) solves it for the case the stream exists. But if the stream does not exist, and thus gets created in succession, this call for check still generates the same error after the timeout.

public async Task<bool> Exists<T, TId>(TId aggregateId)
{
    var stream = GetStreamName<T, TId>(aggregateId);
    var result = _esClient.ReadStreamAsync(Direction.Forwards, stream, StreamPosition.Start, 1);

    if (await result.ReadState == ReadState.StreamNotFound)
    {
        // This still generates an DeadlineExceeded error
        return false;
    }
    
    await foreach (var e in result) {
        // If the stream exists, iterate over the event(s) to not create the DeadlineExceeded error
    }

    return true;
} 

Thanks for the clarification.

I opened an issue here. It seems to be more related to the underlying gRPC client. The error doesn’t effect the execution but logs aren’t helpful https://github.com/EventStore/EventStore-Client-Dotnet/issues/72

Note for other who might hit this using the 20.x & 21.2 / 21.6 grpc client
the default timeout for ReadStreamAsync is 5 sec :
and can be controlled on a per request basis like this
we do not recommend to set it to infinite …
var result = client.ReadStreamAsync(Direction.Forwards, “thestream”, StreamPosition.Start,
configureOperationOptions:
o => o.TimeoutAfter = TimeSpan.FromSeconds(30));

1 Like

Sorry, this does not work for me. Set the timeout to 1 Minute. Still “Deadline Exceeded”.

need more info , what are you doing ?

Currently trying to get last Eventnumber for my Projections with this code:

  public static async Task<long> GetLastEventNumberAsync(this EventStoreClient connection, string projectionName)
        {
	        try
	        {
		        var response  =  connection.ReadStreamAsync(Direction.Backwards, projectionName, StreamPosition.End, 1,
			        options => options.TimeoutAfter = TimeSpan.FromMinutes(1), true);

		        var lastEvent = await response.SingleOrDefaultAsync().ConfigureAwait(false);
		        return lastEvent.OriginalEventNumber.ToInt64();
	        }
	        catch (StreamNotFoundException e)
	        {
		        return 0;
	        }
        }

Works about 80 times, than crashes.
None of the Streams exists.

Testing it a separate Solution:

  public static async Task Main(string[] args)
        {
            var settings = EventStoreClientSettings.Create("esdb://docker:2113?tls=false&keepAliveTimeout=10000&keepAliveInterval=10000");
            var client = new EventStoreClient(settings);
            var i = 0;
            while (i < 100000)
            {
                var projectionName = $"Test{i}";
                var result =  await client.GetLastEventNumberAsync(projectionName);
                Console.WriteLine($"{projectionName}:{result}");
                i++;
            }
        }

Works fine… must be something in my bigger Solution…

Parallel works fine, too…

public static async Task Main(string[] args)
{
    var settings = EventStoreClientSettings.Create("esdb://docker:2113?tls=false&keepAliveTimeout=10000&keepAliveInterval=10000");
    var client = new EventStoreClient(settings);
    var items = Enumerable.Range(1, 100000).ToList();

    ParallelOptions parallelOptions = new()
    {
        MaxDegreeOfParallelism = 32
    };
    
    await Parallel.ForEachAsync( items, parallelOptions, async  (i, _)=>
    {
        var projectionName = $"Test{i}";
        var result = await client.GetLastEventNumberAsync(projectionName).ConfigureAwait(false);
        Console.WriteLine($"{projectionName}:{result}");
    });

}

I just enabled the logging. Detected somthing weired.
At the beginning of the Log everthing seems to be normal.
There are Starting Grpc-Calls like this :
Starting gRPC call. Method type: 'ServerStreaming', URI: 'http://docker:2113/event_store.client.streams.Streams/Read'.

There are finished Grpc-Calls like this:
Finished gRPC call. <s:Grpc.Net.Client.Internal.GrpcCall>

After a while (about 2 Minutes on my Machine) I still get Starting Grpc-Calls.
But I don’t get Finishing calls. My Timeout is currently 10 Seconds.
Guess what happens after these 10 Seconds…
[WRN] gRPC call deadline exceeded. <s:Grpc.Net.Client.Internal.GrpcCall>

This happens when I start up the Application and it makes sure that all Projections are created and all Readmodels are up to date.

Any Idea what I am doing wrong ?

You need to dispose the response. If you don’t then the underlying gRPC call will remain open.