Hey ESDB Team,
I have seen Greg Young post about updating projections and generally saying that you should consider projections immutable. So I’d like to know, what kind of workflow do you recommend when you need to replace a previous projection with a new one (or effectively update the old)?
Two options I’ve considered:
- Call UpdateProjection which is built into ESDB
- What are potential consequences of this and how is it working internally? I noticed that checkpoints are created for each stream. When the projection is updated, are these checkpoints restored sequentially or are they only restored when a new event comes in related to the particular stream? I’m wondering about risks of downtime etc.
- Deploy a new projection in parallel. Read all events from the existing input streams through the new projection. Consume the new output events to a new collection in the read model. Once the new projection has caught up to the old, use a feature flag to switch which subscriber/read model is used. Then delete the old projection.
If there is some alternative recommendation or a detailed guide somewhere on how to do this, can you provide this information?
Thank you for your time.
You can use either, depending on what you want to achieve and what the update is.
For the first option, the new projection code will start running immediately after the projection update. I don’t exactly follow the question about checkpoints. Mostly, confused by the word “restore”. Checkpoint is the position of the last event processed by the projection. When you update a projection, it will stop running for a brief moment, then the new code will execute from the last checkpoint. There’s less than a second downtime, usually it is unnoticeable.
The second option is better when you substantially change the projection’s output. In that case, it’s hardly possible to execute an update as you’d also need to change all consumers that use the projection output. If you want to run old and new consumers side by side, deploying a brand new projection allows you to support both old and new projection output. You can then kill the old projection when your consumers are updated. The downside there is that the new projection will start from scratch and it will take time to catch up.
Thank you for the response!
Sorry for the confusion. When I say restore, I mean restoring the state of the projection. If my projection holds some $ balances for accounts for example. Event 1,2,3 have some deposits that leave account 1 with a balance of 1000. Then the projection is updated. How does the projection restore that value? I’m assuming it’s not just held in memory originally since there may be millions of accounts & associated streams.
But if there is only one checkpoint value representing the last processed event, then I’m wondering how state is maintained for all of these accounts through projection updates.
Projection state is held in projection output streams. It’s always persisted.
1 Like