Verifying hash vs Initializing

EventStoreDB
#1

Hi,

I’ve noticed when restarting an EventStore node, it first gets listed as Initializing in the cluster status (which makes sense), while the log output is running through “Verifying hash for TFChunk” and counting down.

However, the node appears to become an active Slave node before all hashes have been verified.

What is the relationship between “Initializing” and “Verifying hash”? And specifically, is there a way to get it to become a full member of the cluster quicker (say after a fixed number of chunks being verified), while still running the hash check in the background to detect the unlikely event of data corruption?

Cheers,

Kristian

#2

Verifying hashes is scanning all the data to see if data may have changed on disk during a shutdown etc (power outages or faulty disks can cause this). It will be optimistic on start-up by default but still verify.

#3

Thank you. Could you explain what happens between a node starting, initializing and finally becoming a full member of the cluster? I’m hoping to speed this up in our cluster to better deal with rolling upgrades etc and had thought it was waiting on the has verification to complete.

#4

There is an option to disable the verification of chunks which is --skip-db-verify (check docs or run with --help to be sure but I believe its that).

#5

OK, now I’m confused. Will skipping chunk verification make it join the cluster as a full member sooner? Because I see it join the cluster now after some, but not all, chunks have been verified.

#6

chunk verification is asynchronous index verification is currently synchronous (also preloads index at the same time). There is a ticket to make that asynchronous as well but not sure the status of it.

#7

I asked Shaan who is working on this to update.

#8

Hi Kristian,

As Greg mentioned, index verification and loading is done synchronously.

We have just completed an enhancement that should speed up loading of the index data at the expense of disabling the index verification (md5 hash will not be verified). It’s being tested and should be available in next release if everything goes well.

Ideally, we should also be able to move the index verification to the background just like it’s done for the database chunk files. It’s something we are considering to also work on.

Thanks

Shaan

#9

Just as an addition with the background loading the node start up time will be faster but the behaviour will be less predictable during that period of time.