Safely adding new projections

EventStoreDB
1

Every time we add a new projection, especially one that processes a lot of events, all of our projections go from “100%” done to something lower. It takes quite a while to go back to 100%, and while this happens, we tend to have poor performance or downtime. Can somebody explain this (coudn’t find anything in the docs), and how to mitigate it.

2

I’m new to event store myself, but i’m curious here as well about the performance of projections. I’m guessing the answers to the following questions will be helpful in diagnosing:

  • How many continuous projections do you have?

  • From what streams are they projecting? Are they generally using the same streams?

3

We have 10 continuous projections, 5 of these are the built in projections

Some of them are from the same streams, but not all.

4

Hi,

When you add a new projection it will by design project new events into the Event Store.

Those new events will then need to be processed by at least the 5 standard projections you have running.

This processing of the continually added events during catch-up is why you see the % “drop” from 100.

The projections are processing all of the new events being added.

Then depending entirely on your design the new projection may need to process anything the existing projection emit as well until everyone is caught up.

The performance load on the systems will be directly related to the amount of processing the interaction projections produce.

From what you’ve said I’d look at redesigning or refactoring interactions between your projections, also check for write amplification.

(i.e. if N events are written to a stream and result in N x X events emitted by projections then the amplified load is N + N x X )

Thanks,

Chris

5

Just to be clear:

if I did something like this:

fromStreams([....])
.when({
    $any: (state, event) => {
        linkTo('newStream', event)
    }
})

so this is writing a new event each time?

I found an old blog post which seems to say contrary, so I’m a bit confused:

In order to get into indexing we will need to introduce a new basic operation linkTo. linkTo works very similarly to emit except when you linkTo(“stream”, event) it does not actually write the event to the stream. It instead emits a pointer to the event to the stream.

https://eventstore.com/blog/projections-5-indexing/

6

Hi,
The pointer event is generally much smaller than the original event, but it it still a write operation to place it in the log.
- Chris

7

Is there any way to deal with this or speed it up? Every time we create a new stream like this, it brings our whole system down. The last time we tried it took 24 hours (although I think it may have been exacerbated by a process that was writing new events at the same time)

8

That time frame does seem excessive. Without a deep dive into your implementation, the only options would be to scavenge the db or increase the hardware/network provisioning.
- Chris