Grpc.Core.RpcException DeadlineExceeded in AWS Lambda

EventStoreDB V20.X
1

I am using Grpc.Net.Client 2.35.0 and the latest eventstore in an AWS ECS

I have a AWS Lambda developed with C# for netcoreapp3.1 (using also Amazon.Lambda.AspNetCoreServer 5.3.1)

When running “locally” (the logic as a standalone aspnet core app) everything goes well, the grpc client can access event store and write events.

I’m using tls=false in my connection string.

When running in AWS the Lambda fails with the following exception

[Error] Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware: An unhandled exception has occurred while executing the request. 
An unhandled exception has occurred while executing the request.
Grpc.Core.RpcException: Status(StatusCode="DeadlineExceeded", Detail="")
at EventStore.Client.Interceptors.TypedExceptionInterceptor.<AsyncClientStreamingCall>b__5_0[TRequest,TResponse](Task`1 t)
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__274_0(Object obj)
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at EventStore.Client.EventStoreClient.AppendToStreamInternal(AppendReq header, IEnumerable`1 eventData, EventStoreClientOperationOptions operationOptions, UserCredentials userCredentials, CancellationToken cancellationToken)
at Rubiko.EventStore.EventStoreDb.EventStore.Append(String streamName, Int32 aggregateVersion, IEnumerable`1 events, CancellationToken cancellationToken)

It seems something internal in the grpc client but I can’t figure out what or why running it from a AWS lambda makes any difference.

Happy to share more details if needed.

2

I’ve created a repo to reproduce the issue. Instructions on how to deploy/run with AWS Console are in the README. Notice you must adapt the script to your own needs:

  • the ARN for the role created to execute lambda
  • the --profile diegosasw is my own profile. You’ll need your own AWS profile (or --profile default or empty)
3

I suspect it must be some missing permission in AWS Lambda to access external EventStoreDB with gRPC, but unsure. Probably some VPC or similar needed and the error is due to being unable to reach EventStoreDB?

4

15.236.204.20 is not exposed publicly, at least the port 2113. I guess your assumption is correct, you need to ensure your function has access to the database machine.

5

Thanks for your reply Alexey. I’ve stopped the VM so that IP doesn’t belong to me anymore, but that IP was exposed on port 2113 and I could access the UI and the EventStoreDb from any localhost console application without any issue.

When the code runs within AWS Lambda it probably needs additional network settings to allow lambda external connections (VPC or similar, not too familiar yet with AWS) and probably that’s the reason my lambdas cannot access it.

It doesn’t look like a client gRPC issue anyway :slightly_smiling_face:

Thanks.

6

I think so too. As soon as the client can access the server (routing, firewalls, etc) - it works. It’s usually a networking issue.