Changing something in the past ;-S

Thomas_Pierrain · July 18, 2017, 4:37pm

Hello,

One of my customer wants me to implement a Fund Referential system (asset
management business). They need various capabilities such as Audit Trails,
various kind of projections but also tons of temporal stuffs like: “I want
to know how this Fund X looked like last February…”. I initially thought that Event
Sourcing would be the perfect fit and started to write them a tiny demo of what
an event sourced system looks like (with an in-memory data store). Demo went
well and we’ve been decided then to evaluate GetEventStore and to spike the
read model and projections capabilities of the event sourcing paradigm. So far
so good.

Then… they came with another key-requirements for them: the ability to
patch a Fund in the past (e.g.: to be able to generate reports with data such as they should have been input initially). Basically I understand that the book of record for
Funds won’t be that new system but somehow the real world with other teams and other corps updating them
(other BCs). What they need is to catch-up sometimes with changes that has
happened various weeks/months (like the rename of a Fund) which they didn’t had
time yet to apply on their own System. In that case, they want the answer of the “I want to know how this Fund looked like last February…” request gives them the name of the fund as they should have been input initially and not as their system had it before.

In other word => they want me to get rid of the event sourcing
idempotency (…scary music here…)

I am now trying to think about all the options I have:

either doing something else
than event sourcing for their need

or
to adjust the event sourcing
approach to their specific need with one or many technical strategies

In particular, I’m wondering what may be the consequences of introducing
a kind of “patch event” targeting an applicable timestamp in the past (stored
as meta-data). With such “patch event” we could imagine an implementation like
LoadFromHistoryIncludingPatchEvents(maxTimestamp) on our EventSourcedAggregates
instances that will replay all events from the past until that specific date +
all the following “patch events” that would be relevant (i.e. without erasing same
event types applied after the applicable timestamp of the patch for instance).

To be honest, I have the gut feeling that we’re gonna create a monster here following that lead…
Maybe there could be other alternatives such as “Copy and Replace” etc. Anyway, it’s the first time I heard about that need of patching the past in a system and some hints would be much appreciated here.

Cheers.

Andrew_Badera · July 18, 2017, 4:41pm

Truncate the stream at event number last, and rewrite as stream with corrected data.

Andrew_Badera · July 18, 2017, 4:42pm

Actually, truncateBefore, last+1.

Greg_Young1 · July 18, 2017, 4:42pm

Have you read this? http://codebetter.com/gregyoung/2014/03/02/event-sourcing-and-postpre-dated-transactions/

Thomas_Pierrain · July 18, 2017, 5:49pm

Thanks for your
answers.

@Greg: actually I think there might be an issue if I
simply follow what your blog post is explaining. Let me clarify it with an example:

January 1^st of 2016: I
create a Fund with the name : “Lame name for a Fund”
January 1^st of 2017: I
rename the same Fund raising a FundNameChanged event with the name: “More serious name for a
fund”
June 8^th of 2017: I raise
a retroactive FundNameChanged event for the same Fund but with the name: “My joking Fund”. This retroactive event has
the “applies” metadata set to April 1^st of 2016

Following your
explanations on the blog post, If I make an “As of” request to get the state of
my Fund for the date of: July 1^st
2017, I’ll retrieve “My joking Fund” as the name of the Fund whereas I’m
expecting to have “More serious name for a fund” (actually,
I’m expecting to retrieve the “My joking Fund” answer only for “As of” requests with date
between April 1^st of 2016 and December 31 of 2016).

To
workaround this issue, I was thinking about storing the “applies” metadata for
every event, and to cache the last timestamp for every event type applied in my
EventSourcedAggregate base class so that I would then be able to apply
retroactive events ONLY IF no more recent event of this type (for this
Aggregate) has been already applied. But while I was
thinking about it I was wondering: am I twisting that event sourcing paradigm
to death or what?

Because fixing
the name of my Fund aggregate is somehow easy if you compare with the more
problematic situation of numeric figures of my aggregate that I would need to
increment/decrement over the time. Am I really want to follow that path introducing retroactive events? Is Event
Sourcing with such retroactive events recommended in that case?

Thomas

PS: your book about ES versioning is really interesting

Thomas_Pierrain · July 26, 2017, 8:31am

Just to let you know that we’ve implemented a naive but successful implementation for those 2 kinds of projections.

I say naive because it doesn’t handle yet the case where a change in the past must have side-effects on every occuring events afterwards.

In other word, it works when we need to patch something that can be easily overridden with other event afterwards (e.g. “FundRenamed”). But not when the result of a retroactive event must “patch” other events afterwards (e.g. a numerical value that would be the starting point for other computations afterwards).

Anyway, here is what it looks like so far:

///

/// Inspired by Greg Young typical implementation + our version of bi-temporality.

///

public class EventStoreBackedRepository : IRepository

where T : EventSourcedAggregateRoot, new()

{

private readonly IEventStore _storage;

public EventStoreBackedRepository(IEventStore storage)

{

_storage = storage;

}

public void Save(EventSourcedAggregateRoot eventSourcedAggregate, long expectedLastEventVersionForThisAggregate)

{

_storage.SaveEvents(eventSourcedAggregate.Id, eventSourcedAggregate.GetUncommittedChanges(), expectedLastEventVersionForThisAggregate);

}

///

/// Projection playing all the events as we were aware of at the time (“Tel qu’on le savait à l’époque”)

/// If an event is created after its validFrom date, we call it a “retrospective event” and we will see

/// its impact on the Aggregate only since the createdAt date of the retrospective event at best

/// (i.e. if it hasn’t been overidden afterwards by an event of the same type for this aggregate).

///

/// The identifier of the Aggregate to restore.

/// The date at which we want to restore the Aggregate at.

/// The aggregate as we were aware of it at the .

public T GetByIdAsAt(Guid aggregateId, Timestamp viewPointTimestamp)

{

var obj = new T();

var history = _storage.GetEventsForAggregate(aggregateId);

var eventSourcedAggregate = (ICanApplyChanges) obj;

foreach (var evt in history)

if (evt.CreatedAt <= viewPointTimestamp && !eventSourcedAggregate.HasAlreadyAppliedEventOfSameTypeWithLaterValidFrom(evt))

((ICanApplyChanges) obj).ApplyChange(evt, false);

else

break;

return obj;

}

///

/// Projection playing all the events as we should have known at the time (“Tel qu’on aurait du le savoir à l’époque”)

/// If an event is created after its validFrom date, we call it a “retrospective event” and we will see

/// its impact on the Aggregate since its validFrom date at best

/// (i.e. if it hasn’t been overidden afterwards by an event of the same type for this aggregate with a validFrom date > ).

///

/// The aggregate as we should have known at the (i.e.applying retroactive patches).

public T GetByIdAsOf(Guid aggregateId, Timestamp viewPointTimestamp)

{

var obj = new T();

var history = _storage.GetEventsForAggregate(aggregateId);

var eventSourcedAggregate = (ICanApplyChanges) obj;

foreach (var evt in history)

{

if (evt.ValidFrom <= viewPointTimestamp && !eventSourcedAggregate.HasAlreadyAppliedEventOfSameTypeWithLaterValidFrom(evt))

{

eventSourcedAggregate.ApplyChange(evt, false);

}

return obj;

}

public T GetByIdAsOfNow(Guid aggregateId)

{

return GetByIdAsOf(aggregateId, DateTime.Now.GetTimestamp());

}

[Obsolete]

public T GetById(Guid aggregateId)

{

var obj = new T(); //lots of ways to do this

var history = _storage.GetEventsForAggregate(aggregateId);

foreach (var e in history)

((ICanApplyChanges)obj).ApplyChange(e, false);

return obj;

}

[Obsolete]

public T GetByIdAtAGivenVersion(Guid aggregateId, long versionNumberWeWantToGet)

{

var obj = new T(); //lots of ways to do this

var events = _storage.GetEventsForAggregate(aggregateId);

foreach (var e in events)

{

((ICanApplyChanges)obj).ApplyChange(e, false);

if (e.Version >= versionNumberWeWantToGet)

return obj;

}

return obj;

}

Rickard_Oberg · July 27, 2017, 4:42am

Hi,

We patch our events ALL THE TIME, and it is DEAD SIMPLE!

Whenever we want to either change existing events or insert/delete
events, we create a new cluster with the new version of the app, empty
EventStore. Then we have a subscriber get data from existing system
and write to the new one, with a transform step in the middle. This
lets us do whatever we want with the events to fix things up. Once the
new system is reading live events we switch loadbalancer to new
servers, and off we go. We do this all the time, change events
massively pretty routinely, and also delete/rebuild our production
databases as well if our schema changes (no migration, just rebuild).
If your volumes are such that this is practical, I would seriously
consider it.

regards, Rickard

Greg_Young1 · July 27, 2017, 8:08am

This strategy is discussed in http://leanpub.com/esversioning in the chapter called “cheating” in more detail.

Thomas_Pierrain · July 28, 2017, 6:59am

Thanks for your answers. Actually, retrospective changes (i.e. changing/patching something that I didn’t knew at the time) seems to be the bread and butter of the middle office here. Thus, I would like to find a solution that would be as easy as possible for the business to be autonomous. Instead of using streams cheating techniques (I’ve read your book on that too), I think that the bi-temporality (i.e. having for every event a createdAt timestamp and a validFrom one and 2 projections strategies/filters like AsOf and AsAt) would better fit our constraints (including the culture of the dev team here).

Anyway. We’re continuing our POC. I’ll let you know how it will work.

Cheers.