5.0.1 Upgrade Procedure

EventStoreDB
#1

Hi,

the upgrade procedure described in your blog post has left me a bit confused and worried:

Step 1: Stop all projections. This will force the projections to write a checkpoint and when the node is restarted, events up to the checkpoint will not be processed again.

Note following Step 1 may cause your projections to fault with the following error and you will need to either reset the projection or manually adjust the checkpoint

Does this mean a zero-downtime upgrade is not possible?

What are the implications of not following Step 1?

How do you even stop system projections? Trying to stop them via the es-admin just leads to “Failure: Projection could not be stopped”.

If stopping them leads to a fault as warned, how do you manually adjust the checkpoint?

#2

Hi Mihael,

Thanks for the feedback.

It seems you misread the sentence: “Not following Step 1” as “Note: Following Step 1”. We will change the sentence on the blog post to make it clearer.

If you gracefully stop the projections from the UI before shutting down the node, you should not run into any issue during the upgrade to 5.0.1.

If you’re having any issue stopping the projections, please try to restart the node with the same old version, then stop the projections again.

After successfully stopping the projections, you can now proceed to the upgrade.

If you need further assistance please feel free to let us know.

Thanks

Shaan

#3

Hi Shaan,

I also wanted some clarification on this. Assuming we’re running 4.1.0 on 3 nodes, if I stop the projections and update one node to 5.0.1, then only 1 of the 3 nodes will be 5.0.1 and that one is a slave. Do I have to wait until all 3 are 5.0.1 before starting back up the projections?

#4

I also find the instructions to be unclear.

  1. Do we have to stop all projections, then do the rolling upgrade on all 3 nodes and only then start all projections?
  2. Or can we stop all projections, do the upgrade on one node, then start all projections, and do this for each node?
  3. Or can we even only do a stop/start cycle for all projections and then do the rolling upgrade on all 3 nodes with projections being live all the time during the upgrade process?

Our concern is of course projections downtime.
Options 1 and 2 would result in our Projections being down for at least 1 hour (it takes at least 20min for a node to initialize), with Option 2 splitting this into 3 time slots. And Option 3 would result in just 1 minute of downtime.

#5

Apologies, our scenario is upgrading from 4.0.3 to 4.1.4, so my post doesn’t belong to this Topic (5.0.1 Upgrade Procedure) .
Still I’d be happy if someone points some light on this.

#6

For safety reasons we did the whole cluster upgrade while Projections were stopped, so:

  • stop all projections

  • do the rolling upgrade on all 3 nodes sequentially

  • start all projections

There’s quite some Projections downtime with this procedure, for us it was around 90min as it takes a node quite a while to initialize.

Bash commands for stopping and starting projections, might come useful to someone:

PROJECTIONS=’

$by_category

# disable
for PROJECTION in $(echo "${PROJECTIONS}") ; do \
  echo ; echo ; \
  echo "PROJECTION: ${PROJECTION}" ; \
  curl -XPOST -d "" -u admin:password http://localhost:2113/projection/${PROJECTION}/command/disable ; \
done

# enable
for PROJECTION in $(echo "${PROJECTIONS}") ; do \
  echo ; echo ; \
  echo "PROJECTION: ${PROJECTION}" ; \
  curl -XPOST -d "" -u admin:password http://localhost:2113/projection/${PROJECTION}/command/enable ; \
done