Eventstore with AWS lambda

Anyone had an experience with using aws lambda’s and Eventstore?
We were playing with the idea of making some of our system serverless, but my concern is the connection to the Eventstore (share connection which is always established)

1 Like

While I don’t have personal experience, I have talked people doing something similar. Here are the key takeaways:

  • Forget about running esdb subscrptions on a lambda
  • Instead, your command handling lives in a lambda, then sticks the position of the write result into something like sns fanout + sqs
  • your projection living in lambda then reads the position off its queue and compares it against the last position it stored. if seen position > stored position it reads from stored position (while also skipping it as reading is inclusive WRT position) -> end and applies to the read model

So where does the write to ES actually happen?

From my perspective - you should write the event in the Lambda and then publish to SNS or SQS (or Kinesis, Kafka, whatever queue you prefer) position of the write result. Then another lambda (or set of lambdas - I suggest lambda per projection), will be triggered by queue, and you can do logic as @Joao_Braganca described in the last bullet point.

oskar,

When you say “write the event in the Lambda” I assume you mean create a connection to the Eventstore, and AppendToStream?

@steven.blair - yes, exactly. If you have a concern about the time needed to open the connection, you may try to memorize it. See, e.g. https://technology.amis.nl/2019/10/31/state-management-in-serverless-functions-connection-pooling-in-aws-lambda-leveraging-memoized-functions/.

Thanks for the help guys.
So, I would look at replacing my ES javascript projections and subscriptions and use a Function instead that is being fed from SQS (the published event?)

Yes, that’s what I’d suggest.

You can also run a mixed approach and run the subscription client as a container on EC2 or Fargate task. Such could be used to subscribe and handle projections or be just thin pass through subscribing and pushing the event to SNS or SQS. Using SNS has the benefit that you could push all the events there and do the routing to SQS that will trigger your lambda with projections.

I also saw that AWS recently added Docker support for lambda - that can probably open some options.

Ok, we are going to have a crack up building a prototype using AWS Lambda and see how we get on.
we heavy rely on projections for building enriched events, so it’s hard to see how we could replace that at the moment.

@steven.blair then maybe you could start by making the write model serverless and keep the subscriptions at EC2 or Fargate? That would give you an option to take this step by step.

Could you elaborate more on the “building enriched events” part? Does it mean that you’re doing the event transformations?

Oskar,

Yeah this approach sounds like a good start point for us (writing serverless and a permanent subscription)

As for the event transformations, we have projections which subscribe to various categories, and using a partition, holds some useful state from various different events.
We then emit a nice chunky event that makes updating the read model a chore.
It’s a really import part of our system, and not something I want to change.

There might be other technologies out there that could be used to get the same result (we had a quick look at Kafka but it was Linux only).

we found our read model was having to perform the heavy lifting when an event arrived, and with this approach it’s pretty slick and minimal code.

Ok, understood. Thank you for an explanation. I agree that putting the queue as the man in the middle will cut off a lot of benefits of using subscriptions.

If you have a constant load, then EC2 would be even cheaper than AWS Lambda, if not then you can consider Fargate Tasks as they’re a middle ground between Lambda and EC2. You could publish events to SQS to trigger the Fargate Tasks and start processing subscription. Having that, you could cut some costs if, e.g. the majority of traffic happens during the day, but not so much at night.

Keep us posted on your progress :slight_smile:

For our proof of concept we have settled on:

  1. AWS Lambda for writing to a self hosted Eventstore (EC2)
  2. Self hosted Eventstore continues to run projections
  3. Self hosted service for persistent subscriptions which will call AWS Lambda for writing to our RDS*

*do you think there would be in value writing the enriched events to SQS, then using SQS - Lambda or is this just another component for the sake of it?

I see two places where SQS (or other queues) may help:

  • trigger Fargate Task (with subscriptions) after storing an event,
  • decoupling the RDS writing from the subscription/projections. You could benefit from the built-in retries, DLQ etc. If you add SNS, then you could also get notifications (if, e.g. something went wrong), etc.

I think that you can start without them and add them later if needed. However, I think that it would be good to consider and at least do some PoC to verify what’s the better approach for you.

1 Like

Btw, the gRPC client doesn’t really have an always-open connection, it is stateless for writes. The only part that we can consider “stateful” are continuous reads and subscriptions.

As it comes to EC2, I don’t remember when I had to deal with VMs in production last time. We moved to containers a while ago and it solved most of the deployment and scaling issues.

That’s interesting Alexey.
Our perceived connection sharing problem could be replaced if you started using gRPC.
Currently, we are still running 5.0.8 using the tcp client.