Update of stored Events after code updates

Vlad_Andronache · December 28, 2019, 6:30pm

Hi,

I have already posted this as an “Issue” on Github but I’ve got no response, so I thought to share it here as well. Pasting the post :

We are currently using EventStore.org as our (shockingly) event store. So far we’ve gone on with the following approach:

Each aggregate has it’s own Stream, with the naming convention: “AggregateName-AggregateId”
The Event Type is set to the Assembly Fully Qualified name , so later when we get the event’s back we know the Type and can deserialize the json Data and reconstitute.

However we recently decided for other reasons, to change our Event Type in from the fully qualified name to the simple type name… since the EventType property is in our BaseEvent class, the change got propagated everywhere, including all the way to EventStore, so now our new Streams have the new Event Type. This leaves us with thousands of “old” aggregates / streams, which have the EventType in the old form.

So how should we go about updating (migrating) these old Events ? The article above states that the only option would be to create a new Stream and delete the old one. I’ve tried to do that using the .NET API, it works, but the following thing happens below: (screenshot is of a really old migrated aggregate, please ignore the fact that there are 5 “PartnerRegistered” events, I was doing debugging, and this is the QA environment )

You can see the Event’s numbering has not been reset, it goes on from 5, whereas previously it was 0… also the more worrying thing for me is the Created Date, as you see it’s been reset (of course, as this is a new Stream, with new events…). What are the implications to this approach ? I.e. having all the streams reset like this, with the Creation Date probably identical for all Events in them … it doesn’t seem like a good idea to me, to practice this kind of migration on all aggregates. For us internally, we use a “CreationDateUtc” that we store in the Data payload, so not a problem for us, but I am not sure what the implications are in Event Store (if we were to decide later to use the Projections functionality, or anything else).

Here is a screenshot of a not-migrated aggregate, so you see what I am talking about, the Type column:

So how would you get around this, should should we continue on with deleting and creating of new Streams ? But also, thinking in the future… I understand the theortical arguments of not letting Update operations happen… but maybe we change our classes’ structure (that we deseralize the event’s JSON into… so then we would need the event’s JSON migrated too, how would you get around that ?)

Looking forward to your opinion,
Vlad

Thoughts ?

Greg_Young1 · December 28, 2019, 6:48pm

“So how would you get around this, should should we continue on with deleting and creating of new Streams ? But also, thinking in the future… I understand the theortical arguments of not letting Update operations happen… but maybe we change our classes’ structure (that we deseralize the event’s JSON into… so then we would need the event’s JSON migrated too, how would you get around that ?)”

I actually wrote a book on this sorry for punting on it but I got these questions a lot therefore wrote a much longer answer https://leanpub.com/esversioning

You can see the Event’s numbering has not been reset, it goes on from 5, whereas previously it was 0… also the more worrying thing for me is the Created Date, as you see it’s been reset (of course, as this is a new Stream, with new events…). What are the implications to this approach ? I.e. having all the streams reset like this, with the Creation Date probably identical for all Events in them … it doesn’t seem like a good idea to me, to practice this kind of migration on all aggregates. For us internally, we use a “CreationDateUtc” that we store in the Data payload, so not a problem for us, but I am not sure what the implications are in Event Store (if we were to decide later to use the Projections functionality, or anything else).

CreatedDate is only used by things that want to use it. It is written at the time of the actual event write. Most do as you do and have their own concept of a CreationDate as obviously if doing a migration etc the time something was written might not be the time that it was effective.

Now the next bit might be “why does it start from 5”… We could in fact restart it from zero however this would make a whole pile of other issues as an example what would happen to cacheability of any subscriber who wanted to use the position? This applies especially to subscribers over say http where this is built in but can also apply to many other cases.

Vlad_Andronache · December 29, 2019, 10:04am

Hi Greg,

Thanks for your answer… I will get the book and see your thoughts on the versioning… looking forward to learn some new approaches myself.

As for the migration process, first thing I did was to do a backup of our EventStore database (essentially made a copy of the “data” folder) which has about ~10 GB. After (of course) I broke some streams on QA in the migration process, I wanted to just restore it. So I stopped the event store service, switched the name of the folders to have the backup folder as the new data… however upon starting the service I noticed the login is not working … I then stopped the service and started the eventstore console, to see what’s happening, and I see that Event Store it’s rebuilding it’s indexes:

index rebuild.png

After it finished rebuilding the indexes (it’s been about 15 minutes) I see it’s now reading all events:

All in all it’s already been 20 minutes or so, and the eventstore has not finished starting up - is this “normal” ? And in the meanwhile the login in the web-dashboard is not working, supposedly because it’s busy starting up. Can I start it up without it having to go trough all this rebuilding ? Even if I didn’t do a backup & restore of the database, the service could be restarted for other reasons.

Thank you,

Vlad