Cluster instances not completing scavenge

EventStoreDB
1

When running a local scavenge i get this log:

Notably with the “completed line” like this:
2024-06-12 15:04:30 [ 1,48,13:04:30.976,DBG] SCAVENGING: Scavenge Completed. Total time taken: 00:00:05.1476400. Total space saved: 3112960.

However, when we run a scavenge on our cluster it seems to stop half way without any error logs:

Where the last line is:
SCAVENGING: Released the chunks lock
And then just nothing more. No errors or more logs regarding scavenging.

The /metrics reports the pods as being idle (eventstore_statuses / scavenge)

The scavenge command is executed via a cronjob with a two minute delay between each pod

Any ideas about what is going on?

Evenstore version:
eventstore/eventstore:24.2.0-jammy

Thanks in advance

2

Hi @bj.wiegell,

It may be that the log level in the cluster is set to information and locally it is set to debug.

Granted quite a few of those messages should probably be at the Information level (we intend to revise these).

In the mean time we recommend leaving the log levels as their defaults. They’re controlled in logconfig.json and the defaults are available here EventStore/src/EventStore.Common/Log/logconfig.json at release/oss-v23.10 · EventStore/EventStore · GitHub

Hope this helps
Tim

3

Hi Tim, you were absolutely right, we had it set to Information on the cluster. We would prefer to keep it at Information in our production environment while still being able to confirm that scavenges are completed (log screened continually in grafana). Do you have an open issue about this, or do you want me to create one? Thanks for the assistance.

4

Hi @bj.wiegell

I’ve created this public ‘idea’ that you can track/comment/vote on Adjust the Log levels | EventStoreDB Ideas Portal

Just so you know, there are other useful logs at the Debug level apart from scavenge. Usually we recommend people to keep the level at Debug (as counter intuitive as this is).

You can also view completed scavenges in the UI on the Admin tab

Thanks
Tim

5

Ok i put in a vote :slight_smile: Never knew you had that idea platform too.

6

It’s pretty new :slight_smile:
<adding more characters because apparently you need at least 20 to send a reply>