Proper approach to handle "upgrades" or bug fixes in projections?

EventStoreDB
1

We’re trying to figure out what the best approach will be for us to handle modifying projections once our system has gone live for the cases of handling bug fixes or enhancements.

We currently have a stream for events (source code commit messages and metadata), let’s call it Commits.

If a commit messages contains a pattern like S-12345 or D-98765 then a projection named CommitsLink uses linkTo(‘workitem-S-12345’, event) or linkTo(‘workitem-D-98765’). Then, another system, which has the context of S-12345 or D-98765 already, simply queries these streams. That part is working just fine.

We made a mistake in not at first accounting for messages like “S-12345 and D-98765 and T-77886”. That is, a single commit message that mentions 3 distinct workitems.

We corrected the logic in our development branch and everything is fine. We’ve already got a semi-production deployment though that we’ve pumped data into, and thus all commits that have more than one item mentioned, are just having the 0th linkTo()ed now.

Does the following sound like a legitimate approach to handle “upgrading” the system?

  • Before modifying the existing CommitsLink projection, create a new projection, CommitsLinkHistoricalFixUp, which pays attention to the sequenceNumber value that events have, and passes over all the existing events, and for ones that fall into a range for the sequenceNumber, then simply links the 2nd - N mentions, ignoring the 1st, since we know that’s already been handled by the existing projection
  • Stop the CommitsLinkHistoricalFixUp projection now that it has handled all historically missed events
  • Modify the CommitsLink projection to have the new code from the corrected version in our development branch
  • Now, I suppose both could co-exist forever, but that makes the system more difficult to understand later for other developers
  • Check whether any events came in after we stopped the CommitsLinkHistoricalFixUp and rectified the CommitsLink projection
  • If so, modify the range inside this projection and re-run it to handle those missed events.
  • I suppose there might still be a change of missing events, but it seems pretty low in our case and current throughput
  • And, it seems like inspecting the Position value in the projection meta-data will help mitigate this anyway.
    Example event:

data: {message: “Implemented styling for the apply button. close:T-77886 fix:D-98765 ready:S-12345” }

branch: “teamRoomUX2_S-12345”

eventType: “Commit”

isJson: true

linkMetadataRaw: “”

metadataRaw: “”

metadata_: null

partition: “”

sequenceNumber: “2194”

streamId: “Commits”

Here’s something we tried which does NOT seem like the best way to do this!

  • Stopped the CommitsLink projection that does the linkTo and modified it to have the new logic to handle 0 -> N.
  • Result: no problem, saved fine. Events Processed count set back to 0, pointer at the end of the Commits stream. That’s fine.
  • Reset the CommitsLink projection
  • Result: duplicates the events into the various workitem- streams.
  • I think this the expected and proper EventStore behavior, but not the correct approach for us

Thanks!

Josh

2

Here's something we tried which does NOT seem like the best way to do this!

Stopped the CommitsLink projection that does the linkTo and modified
it to have the new logic to handle 0 -> N.

Result: no problem, saved fine. Events Processed count set back to 0,
pointer at the end of the Commits stream. That's fine.

Reset the CommitsLink projection

Result: duplicates the events into the various workitem-<pattern> streams.

I think this the expected and proper EventStore behavior, but not the
correct approach for us

This is more in general projection questions (not ES projections).
Technically speaking there is no such thing as editing a projection
you can create a new projection but can't "edit" one. ES projections
does support edit in place this is running with scissors. In general
the correct behaviour is to replay from the beginning on a change (as
you may have changed the past). You talk about "duplicating events",
are they really duplicates? How would we know this?

Cheers,

Greg

3

Thanks Greg,

If we replayed from the beginning, we'd have to first delete all the other streams that were linked to from the original projection. I don't necessarily mind doing that...but I'm not sure of an easy way other than querYing for the names that start with 'workitem-'.

Although, I don't quite get the changed the past comment. The way I see it we did not change the past. We realized that events already stored should be linked to more places. In our case, we don't have any subscriptions to the workitem- streams, we just read them from top.

4

Pretty sure it will do that have you tried? :slight_smile:

Although, I don’t quite get the changed the past comment. The way I see it we did not change the past. We realized that events already stored should be linked to more places. In our case, we don’t have any subscriptions to the workitem- streams, we just read them from top.

Es projections support either the concept of a projection (in which case it does change the past) or of a state machine in which case it doesn’t. This is why you can change them in place. However changing in place is a dangerous operation if you rerun it at some point it will give historically different results.

5

Pretty sure it will do that have you tried? :slight_smile:

Are you referring to deleting multiple streams by category “workitem-”? If so, I haven’t see info on how to do that (other than manually after querying /streams/$streams and manually scripting the pattern match in code)

If there’s a way to pattern-match delete that ES internally takes care of that would be much easier, and then it would be easy to just re-run the projections against the original events and project out the linked streams that we actually do want now.

Although, I don’t quite get the changed the past comment. The way I see it we did not change the past. We realized that events already stored should be linked to more places. In our case, we don’t have any subscriptions to the workitem- streams, we just read them from top.

Es projections support either the concept of a projection (in which case it does change the past) or of a state machine in which case it doesn’t. This is why you can change them in place. However changing in place is a dangerous operation if you rerun it at some point it will give historically different results.

This is why I was thinking we’d want to create a new non-continuous projection that looks at the previous events that had already been processed by the “v1” version of the projection in order to linkTo the additional streams that were missed.

thanks,

Josh

6

Resetting a projection deletes the previously emitted data.

7

OK, we’ll try that…I think an issue is we have a couple of projections, so maybe we have to reset them in a specific order

Thanks

8

So what happens is it issues a soft delete (the streams stay the same).

eg if I had a stream myStream which was having say linktos in it
currently there are 5 events. When I restart the first event will
start at 6 (0-5 are soft deleted)

9

OK, excellent!

We also realized we only needed to reset one projection, because the other one wasn’t changed.

Here’s what we did:

  1. Stopped the projection, at which point it was stopped on position 9874.

  2. Edited the projection so that it would linkTo 2 - N instead of just the first matched item.

  3. Started the projection (Verified that it still had position 9874).

  4. Reset the projection

  5. Verified that it deleted the old stream data and projected all the others out as before AND ALSO the new ones. (Event # increased within the linked to streams, which is fine because we do not depend on those numbers for anything)

Worked great!

Thanks for your help,

Josh

10

So, we ran into one more possible glitch.

Early on, we started using the HTTP api to create our projections, but they got named “projection.js”, but later we moved to just naming them “projection”.

Is it possible to rename a projection?

We tried to stop, then reset, then modify the existing projection to essentially do nothing…thinking that it might still result in the previously emitted events getting soft-deleted, but that doesn’t appear to be the case.

We also tried posting a “dummy event” into the same stream names, and that does appear to work, but we’d prefer not to have those dummy events.

What we’re trying now is:

fromAll().whenAny(function(state, ev) {

if (!state.streams) state.streams = { ids: {}, commands: [] };

if (!state.streams.ids[ev.streamId] && ev.streamId.indexOf(“workitem-”) === 0) {

state.streams.ids[ev.streamId] = true;

state.streams.commands.push(‘curl -X DELETE https://localhost:2113/streams/’ + ev.streamId);

}

});

It produces output like:

{
  "streams": {
    "ids": {
      "workitem-S-11111": true,
      "workitem-AT-01001": true,
      "workitem-S-01001": true,

      etc.......
    },
    "commands": [
      "curl -X DELETE https://localhost:2113/streams/workitem-S-11111",
      "curl -X DELETE https://localhost:2113/streams/workitem-AT-01001",
      "curl -X DELETE https://localhost:2113/streams/workitem-S-01001",

      etc......
    ]
  }
}

We’d then remove quotes and make a shell script out of that to execute soft-deletes on all those streams before we delete the existing projections and recreate them fresh.

Is there an any cleaner way to do this?

Thanks,

Josh

11

We ended up doing the curl based soft deletes that way. That appeared to work, but then we ran into strange issues.

We deleted the projections named with a .js extension, and then pointed the app at the ES. Our app reads the all non transient projections end point and when it doesn't find our projections, it posts to create them. So, it created the properly named ones.

At this point the only stream we had left was our "commits" stream, which itself is not the result of a projection's linkTo, copyTo, or emit. It is posted to by http clients only.

When our first projection ran which bisects commits by analyzing the events and linkTo()ing events with the right pattern into commits-with-mention and those without it into commits-without-mention, we got an error about WrongExpectedVersion and the projection faulted.

Next we tried to copyTo all the events from commits into commits-001, soft delete commits, and then recopy the events back into commits to simulate them coming in fresh into the same stream name. We read from "commits" in the app, so worst case now is we have to change the stream name, but we'd rather keep it the same.

We were able to do the first copyTo, into commits-001, but not able to copy them back into commits. We were also able to successfully copy them from commits-001 into commits-002.

So, again: I think our worst case scenario now is we have to copy the events and then rename the stream that the other part of our app reads from. Or, perhaps we should just create a clean instance of ES, and then POST the same events into a stream of the same name.

But, I'm wondering if we're still doing something odd that would cause the brand new projection to fault with that WrongExpectedVersion when attempting to linkTo over that original commits stream. I don't really understand why a new projection thinks there should be any particular expected version anyway.

12

There didn't appear to be any error when we tried to copy from commits-001 back into commits...it just stuck on position: commits -1.

Maybe we should do it as an emit instead?

13

I tried emit instead. Here’s what the log shows (the stream name is actually github-events…I just named it “commits” in the previous emails for simplicity)

14

The projection WILL emit to some brand new stream names, but not to other brand new stream names. Just don’t understand what it’s going on with it.

We’ll probably just create a clean instance and post the original events into the stream via HTTP.

15

One interesting thing we noticed was that we had to use embed=tryharder to fetch the contents of the events via curl.

curl https://localhost:2113/streams/commits-001?embed=tryharder

When we tried embed=content or embed=prettybody there was no “data” property that came back. The property was not present at all in the response.

16

We did the POST approach on a clean instance and it worked great!