Duplicated Events in Category Stream after linkTo Projection reset

EventStoreDB
#1

I have encountered a strange scenario in my use of projections:

  1. I write a FloatDepositMade Event to my Account Aggregate Stream - AccountAggregate-4fd7a2c1fccf46dda72661d668486639 (I have a stream per account)

  2. I have a projection pick this event up and link it to a new stream using linkTo(). The new stream is called AccountBalanceAggregate-4fd7a2c1fccf46dda72661d668486639 (I have a balance stream per account)

  3. I have a second projection which partitions the AccountBalanceAggregate streams using the $ce-AccountBalanceAggregate stream.

This has been working perfectly fine until I reset the projection is step 2 by accident and I now have the following results

  • The AccountAggregate-4fd7a2c1fccf46dda72661d668486639 stream has the original event as written in step 1
  • The AccountBalanceAggregate-4fd7a2c1fccf46dda72661d668486639 has only one event but its a new event with the old one being truncated and the metadata updated to $tb: 1 accordingly
  • The $ce-AccountBalanceAggregate catgegory stream now has 2 events (exactly the same)
    The result for me is I am adding 2 x the deposited money into the partition as this is running of the category stream.

I have done multiple resets now and every time the reset is done another copy of the event is written to the $ce-AccountBalanceAggregate stream :frowning:

Can someone confirm if this is expected behaviour when resetting projections and if there is away I can work around the duplication of the events in my projection from the $ce-AccountBalanceAggregate stream.

Regards,

Stuart Ferguson

#2

We are seeing a similar issue.

I guess the question is, does linkTo write indempotently?

When we reset our projection which has a linkTo we seem to see a “duplicate” event written in the category, but also the stream we linkTo is truncated.

Anyone else experienced this?

Perhaps it’s more a design flaw than a technical issue.

#3

Just for reference, here is a short example of what I have been doing:

var processEvent = function(s, e) {

if (e.data) {

        var merchantStreamName = 'MerchantAggregate-' +  e.data.merchantId.replace(/-/gi, "");

        //We we reset this projection event 0 is soft deleted, and our event from
        //$ce-Bucket becomes #1
        //Also, in $ce-MerchantAggregate we now have two links to the original event
        linkTo(merchantStreamName, e);
}

};

fromStreams(’$ce-Bucket’)
.when({
$init: function(s, e) {
return {
count: 0
}
},
‘BalanceDepositMadeEvent’: processEvent
});

``

#4

It’s by design. Resetting a projection makes it ignore all previous output. This is done by incrementing part of the projection version that is tracked and stored with emited events.

#5

Laurence,

Thanks for the reply.

it’s really puzzling me this.

I am guessing by the lack of other people having this problem, that I am doing something fundamentally wrong with the projections.

Could you suggest an alternative approach?

#6

I have done a bit more testing, and when I query the category stream, the UI gives me a 404 for the event that has been soft deleted, as does postman, but projections subscribed to this stream does get soft deleted events?

Is this expected?

There doesn’t appear to be a way of checking the event to see if it’s position has been soft deleted or not.

I would be happy enough someone telling me my approach is what the issue, but this seems like a basic thing, creating a new stream in a projection using linkTo.

if we reset the projection, we end up with duplicates in the newly created stream (even though postman and the UI “know” it’s a duplicate)

#7

Sorry I missed the nuance of your question. The $all stream doesn’t check stream ACLs or deletion. If the projection is written such that it uses the $all stream (FromAll) then it will pick up deleted events.

The work around is to scavenge the events marked as deleted, but that only works on complete chunks so not perfect.

#8

Laurence,

Thanks again for responding.

Having thought this through a bit more, my real problem is having to use a projection to essentially maintain my Aggregate.

In my Domain code, I need to write to Save to two difference streams on the back of a single Command, but with EventStore, I can’t group these writes together like a SQL Database.

Therefore, I took the decision to write to a single stream (I call it a bucket) and then let my projections (via linkTo) add the events to the correct streams.

Using the example above, I have a Merchant Aggregate, which keeps a balance and I have a Transaction aggregate.

As part of a Transaction being performed, we update the Transaction with the response code, and also update the Merchants Balance.

So I have a Transaction stream to write to and a Merchant stream to write to.

It would be a non issue using traditional persistence, but this seems to be tricky in EventStore.

Using the projections was more of a workaround for this problem. I would prefer to always be updating my Aggregates directly and saving them directly to their own streams.

Have you had any experience with similar issues?