AWS Lambda cannot connect with "esdb+discover"

Hey,

I have an issue with the connection between my lambda function and EventStoreDB. The EventStoreDB is deployed on EC2 with three node clusters - works fine. I also created a separate VPC for this EventStore and my lambda function also exists in the same VPC as EventStore.

The Lambda doesn’t give me any errors I can only see in GQL Playground “Status 504” after ~30 sec of thinking.

I have tested this on local and AWS.

ON LOCAL:

we can use the DNS in two variants with, and without +discover

  • esdb://nodes.eventstore.myDomain:2113?tls=false
  • esdb+discover://nodes.eventstore.myDomain:2113?tls=false

This DNS is a Multivalue Record A which stores Public IP Addresses from three instances (3 clusters of EventStore)

These two variants work fine which means we can write events to the EventStoreDB

ON AWS:

we can use the DNS in two variants with, and without +discover

  • esdb://cluster.eventstore.myDomain:2113?tls=false
  • esdb+discover://cluster.eventstore.myDomain:2113?tls=false <---- THIS ONE DOESN’T WORK

Only the first one works fine.

We can also use the IP address that would look like this

  • esdb://PRIVATE_IP_ADDRESS:2113?tls=false or other of clusters
  • esdb+discover://PRIVATE_IP_ADDRESS:2113?tls=false <---- THIS ONE DOESN’T WORK

We can also use the DNS address of each IP address:

  • esdb://node1private.eventstore.myDomain:2113?tls=false
  • esdb+discover://node1private.eventstore.myDomain:2113?tls=false <---- THIS ONE DOESN’T WORK

On balance, it looks like every time we want to use +discover the connection doesn’t work and we’re unable to write events.

There shouldn’t be any DNS issue because as I tested we can use IP, and DNS to connect. The issue is only when we get try to use +discover keyword.

The DNS entry cluster.eventstore.myDomain should resolve to IP addresses of all the nodes. Also, the nodes need to be configured to advertised externally as nodeXprivate.eventstore.myDomain where X is the number, assuming these are the hostnames. The nodes also need to use ClusterDns gossip.

I am quote sure it has nothing to do with Lambda. You can run a similar app on an EC2 instance in the same VPC, and you’d probably get the same issue.