Archiving eventstore data

EventStoreDB
#1

Hello,

I was wandering what’s the eventstore’s story with data archiving. For example, say we were running eventstore for 4 years and now want to archive the events for the first year and keep the events for the last 3 years.

I was thinking about a migration tool which would pump the events with timestamps up to (now - 3 years) through a projection which would produce a snapshot for each stream with all of the remaining events appended on top (in a second eventstore instance). Then the application would be switched to the new eventstore instance.

Maybe the application could produce historical snapshots (as opposed to technical snapshots which would be produced more frequently) by itself and then trigger eventstore’s GC (I vaguely remember reading about something like $maxEventCount on stream metadata).

What would you recommend?

#2

Hi, Дед Мороз

I'm using a second approach.

I believe it's more logical since only the application (aggregates) knows which internal state it derives from events.
So dumping it once in a while (every n events) via a special event is easy.

Yes, using $maxCount is preferable as it is more deterministic than $maxAge.

Yevhen

#3

How do you handle events that readmodels depend upon? Keep forever, och do you have “permanent” readmodels? (I consider the read-database to be disposable, it should always be recreatable)

/Peter

#4

Projection handlers should support those special snapshot events for
sure.

Yep, projection store is indeed disposable and could be recreated

from events.

#5

What about backups? I’d want to keep all of the events somewhere AND keep the current operational state with maxCount = N.
Doing simple copies of the operational state db would make restoring all of the historical events very time consuming, as you’d have to go through all of the backups.

Probably a separate projection which would pump events into another eventstore instance (skipping snapshots)?

Any ideas?

#6

Ah, I was confused by your use of the word snapshot, I imagined it as a traditional snapshot, wich would open up a lot of versioning issues. But a “collecting” event might not be a bad idea.

/Peter

#7

Ye, sorry. It’s really a “collecting” event :slight_smile:

#8

Ye, you can have a js projection in event store which will just copy
all events in archive stream …