snapshot size

EventStoreDB
#1

Hi
I’m using Akka persistence.

My data consists of a fairly large and complex model used for logistics planning and optimisation.

In the context it’s used, it makes sense for me to use event sourcing on the whole model and some form of snapshot of the whole model

In json I guess the model will take up about 0.5mb and I’d probably snapshot every 15 minutes or so.

All the examples I’ve seen for eventstore seem to write only small things to the database.

Are larger objects appropriate? Would you expect any problems (especially with respect to the Akka persistence drivers)?

Thanks

Tim

#2

I don't see any issues with it (we test up to a few mb). There is an
upper limit on it though. The other issue to watch out for is
obviously such a write could cause latency on other writes if you
needed consistent latency etc.

#3

The absolute maximum record size is 16MB of which some is our internal structure. However, I strongly recommend sticking to under 1MB of payload in events to avoid throughput issues.

#4

I’d consider offloading large payloads to a secondary data store more suited to that purpose. S3 buckets, for example. You then embed links to the payload in the event data. Obviously, the secondary store needs to be logically append-only so that the events remain deterministic.

This works well for large payloads that aren’t directly processed during projection. The classic example: big binary attachments such as pictures.

I don’t know your use-case, but everything else being equal, I view very large events as a possible design smell. Is something preventing you from using fine-grained events?

#5

In this case it sounds like there are fine grained events, but it’s also desirable to use the same storage for snapshot storage. Not necessarily a design problem, though it could be.

#6

@ Sam - “optimisation” produces a solution to a complex problem. The solution itself is a complex object and may completely replace the previous solution. The new solution would only be decomposed if it were artificially split into many units - but since the whole is an interlinked structure, that would just be make-work.
Manual edits to the solution create smaller events, as does partial optimisations and these would fit naturally into the event system.