Corrupt database detected. Unexpected files

julian.heeb · July 17, 2017, 12:21pm

Hi

After restarting Event Store 4 I got this error message:
[00001,01,14:20:28.615] Unhandled exception while starting application:
Corrupt database detected.
[00001,01,14:20:28.621] Corrupt database detected.
Unexpected files: /var/lib/eventstore/chunk-000001.000000, /var/lib/eventstore/chunk-000002.000000, /var/lib/eventstore/chunk-000003.000000, /var/lib/eventstore/chunk-000004.000000.
Corrupt database detected.

``

Do you know what may be the problem and how we can fix it?

Thanks
Julian

pieter.germishuys · July 17, 2017, 12:33pm

We would need to see the complete log, from the point to where the node has started up until it throws this exception.
For example the absense of the appropriate checkpoint files might cause this and this would be spotted by providing the logs.

julian.heeb · July 17, 2017, 1:50pm

Do the logs below help you any further?

I must say that we started the event store in a docker container on our development machine. After stopping and starting the container this problem occured.

exchange.eventstore | 2017-07-14T14:26:56.691406097Z exchange.eventstore | 2017-07-14T14:26:56.691432610Z ES VERSION: exchange.eventstore | 2017-07-14T14:26:56.742324010Z exchange.eventstore | 2017-07-14T14:26:56.742342241Z exchange.eventstore | 2017-07-14T14:26:56.742346505Z exchange.eventstore | 2017-07-14T14:26:56.742349766Z exchange.eventstore | 2017-07-14T14:26:56.742352898Z exchange.eventstore | 2017-07-14T14:26:56.742356013Z exchange.eventstore | 2017-07-14T14:26:56.742358737Z exchange.eventstore | 2017-07-14T14:26:56.742361782Z exchange.eventstore | 2017-07-14T14:26:56.742364674Z exchange.eventstore | 2017-07-14T14:26:56.742367687Z exchange.eventstore | 2017-07-14T14:26:56.742370636Z exchange.eventstore | 2017-07-14T14:26:56.742373647Z exchange.eventstore | 2017-07-14T14:26:56.742376588Z exchange.eventstore | 2017-07-14T14:26:56.742379373Z exchange.eventstore | 2017-07-14T14:26:56.742382040Z exchange.eventstore | 2017-07-14T14:26:56.742389867Z exchange.eventstore | 2017-07-14T14:26:56.742393272Z exchange.eventstore | 2017-07-14T14:26:56.742396607Z HELP: exchange.eventstore | 2017-07-14T14:26:56.742399758Z exchange.eventstore | 2017-07-14T14:26:56.742402809Z LOG: exchange.eventstore | 2017-07-14T14:26:56.742405926Z exchange.eventstore | 2017-07-14T14:26:56.742409155Z exchange.eventstore | 2017-07-14T14:26:56.742412135Z exchange.eventstore | 2017-07-14T14:26:56.742415147Z exchange.eventstore | 2017-07-14T14:26:56.742418237Z exchange.eventstore | 2017-07-14T14:26:56.742421308Z exchange.eventstore | 2017-07-14T14:26:56.742424477Z exchange.eventstore | 2017-07-14T14:26:56.742427540Z exchange.eventstore | 2017-07-14T14:26:56.742430470Z exchange.eventstore | 2017-07-14T14:26:56.742433510Z exchange.eventstore | 2017-07-14T14:26:56.742436583Z exchange.eventstore | 2017-07-14T14:26:56.742444820Z exchange.eventstore | 2017-07-14T14:26:56.742448621Z exchange.eventstore | 2017-07-14T14:26:56.742451638Z exchange.eventstore | 2017-07-14T14:26:56.742454630Z exchange.eventstore | 2017-07-14T14:26:56.742457717Z exchange.eventstore | 2017-07-14T14:26:56.742460775Z exchange.eventstore | 2017-07-14T14:26:56.742463654Z exchange.eventstore | 2017-07-14T14:26:56.742466649Z exchange.eventstore | 2017-07-14T14:26:56.742469637Z exchange.eventstore | 2017-07-14T14:26:56.742472643Z exchange.eventstore | 2017-07-14T14:26:56.742475581Z exchange.eventstore | 2017-07-14T14:26:56.742478563Z exchange.eventstore | 2017-07-14T14:26:56.742481420Z exchange.eventstore | 2017-07-14T14:26:56.742484384Z exchange.eventstore | 2017-07-14T14:26:56.742488327Z exchange.eventstore | 2017-07-14T14:26:56.742491620Z exchange.eventstore | 2017-07-14T14:26:56.742494582Z exchange.eventstore | 2017-07-14T14:26:56.742497517Z exchange.eventstore | 2017-07-14T14:26:56.742500378Z exchange.eventstore | 2017-07-14T14:26:56.742503245Z exchange.eventstore | 2017-07-14T14:26:56.742506091Z exchange.eventstore | 2017-07-14T14:26:56.742531240Z exchange.eventstore | 2017-07-14T14:26:56.742534922Z exchange.eventstore | 2017-07-14T14:26:56.742538063Z exchange.eventstore | 2017-07-14T14:26:56.742541143Z exchange.eventstore | 2017-07-14T14:26:56.742544204Z exchange.eventstore | 2017-07-14T14:26:56.742547186Z exchange.eventstore | 2017-07-14T14:26:56.742550347Z exchange.eventstore | 2017-07-14T14:26:56.742553357Z exchange.eventstore | 2017-07-14T14:26:56.742556240Z exchange.eventstore | 2017-07-14T14:26:56.742559366Z exchange.eventstore | 2017-07-14T14:26:56.742562286Z exchange.eventstore | 2017-07-14T14:26:56.742568383Z exchange.eventstore | 2017-07-14T14:26:56.742571695Z DB: exchange.eventstore | 2017-07-14T14:26:56.742574805Z exchange.eventstore | 2017-07-14T14:26:56.742577974Z exchange.eventstore | 2017-07-14T14:26:56.742580905Z exchange.eventstore | 2017-07-14T14:26:56.742583893Z exchange.eventstore | 2017-07-14T14:26:56.742586847Z exchange.eventstore | 2017-07-14T14:26:56.742589781Z exchange.eventstore | 2017-07-14T14:26:56.742592742Z exchange.eventstore | 2017-07-14T14:26:56.742595686Z exchange.eventstore | 2017-07-14T14:26:56.742598705Z exchange.eventstore | 2017-07-14T14:26:56.742601803Z exchange.eventstore | 2017-07-14T14:26:56.742604888Z exchange.eventstore | 2017-07-14T14:26:56.742607980Z exchange.eventstore | 2017-07-14T14:26:56.742610951Z exchange.eventstore | 2017-07-14T14:26:56.742613988Z exchange.eventstore | 2017-07-14T14:26:56.742616990Z exchange.eventstore | 2017-07-14T14:26:56.742619994Z exchange.eventstore | 2017-07-14T14:26:56.742622896Z exchange.eventstore | 2017-07-14T14:26:56.742625903Z exchange.eventstore | 2017-07-14T14:26:56.742628761Z exchange.eventstore | 2017-07-14T14:26:56.742631646Z exchange.eventstore | 2017-07-14T14:26:56.742634627Z exchange.eventstore | 2017-07-14T14:26:56.742637548Z exchange.eventstore | 2017-07-14T14:26:56.742640336Z exchange.eventstore | 2017-07-14T14:26:56.742643227Z exchange.eventstore | 2017-07-14T14:26:56.742646108Z exchange.eventstore | 2017-07-14T14:26:56.742649597Z exchange.eventstore | 2017-07-14T14:26:56.742652723Z exchange.eventstore | 2017-07-14T14:26:56.742655662Z exchange.eventstore | 2017-07-14T14:26:56.742661267Z exchange.eventstore | 2017-07-14T14:26:56.742664524Z exchange.eventstore | 2017-07-14T14:26:56.742667495Z exchange.eventstore | 2017-07-14T14:26:56.742670421Z exchange.eventstore | 2017-07-14T14:26:56.742673316Z exchange.eventstore | 2017-07-14T14:26:56.742675828Z exchange.eventstore | 2017-07-14T14:26:56.792298847Z exchange.eventstore | 2017-07-14T14:26:56.842562341Z exchange.eventstore | 2017-07-14T14:26:56.842579833Z exchange.eventstore | 2017-07-14T14:26:56.842583867Z exchange.eventstore | 2017-07-14T14:26:56.842587143Z exchange.eventstore | 2017-07-14T14:26:56.842590426Z exchange.eventstore | 2017-07-14T14:26:56.842593435Z exchange.eventstore | 2017-07-14T14:26:57.051783013Z exchange.eventstore | 2017-07-14T14:26:57.153910175Z exchange.eventstore | 2017-07-14T14:26:57.154750364Z exchange.eventstore | 2017-07-14T14:26:57.154761102Z exchange.eventstore | 2017-07-14T14:26:57.154765152Z exchange.eventstore | 2017-07-14T14:26:57.154768586Z exchange.eventstore | 2017-07-14T14:26:57.154772387Z [00001,01,14:26:56.652]
4.0.0.7 (HEAD/aa8cda89f70ddef785d4e673bf9a336bf88da59f, Thu, 23 Mar 2017 12:08:13 +0200)
[00001,01,14:26:56.687] OS: Linux (Unix 4.4.0.72)
[00001,01,14:26:56.706] RUNTIME: 4.6.2 (Stable 4.6.2.16/ac9e222) (64-bit)
[00001,01,14:26:56.707] GC: 2 GENERATIONS
[00001,01,14:26:56.707] LOGS: /var/log/eventstore
[00001,01,14:26:56.728] MODIFIED OPTIONS:

INT IP: 0.0.0.0 (Config File)
EXT IP: 0.0.0.0 (Config File)
INT HTTP PREFIXES: http://:2112/ (Config File)
EXT HTTP PREFIXES: http://:2113/ (Config File)
ADD INTERFACE PREFIXES: false (Config File)
RUN PROJECTIONS: All (Config File)
START STANDARD PROJECTIONS: true (Config File)

DEFAULT OPTIONS:

CONFIG: /etc/eventstore/eventstore.conf ()
False ()
VERSION: False ()
/var/log/eventstore ()
DEFINES: ()
WHAT IF: False ()
DISABLE HTTP CACHING: False ()
MONO MIN THREADPOOL SIZE: 10 ()
INT HTTP PORT: 2112 ()
EXT HTTP PORT: 2113 ()
INT TCP PORT: 1112 ()
INT SECURE TCP PORT: 0 ()
EXT TCP PORT: 1113 ()
EXT SECURE TCP PORT ADVERTISE AS: 0 ()
EXT SECURE TCP PORT: 0 ()
EXT IP ADVERTISE AS: ()
EXT TCP PORT ADVERTISE AS: 0 ()
EXT HTTP PORT ADVERTISE AS: 0 ()
INT IP ADVERTISE AS: ()
INT SECURE TCP PORT ADVERTISE AS: 0 ()
INT TCP PORT ADVERTISE AS: 0 ()
INT HTTP PORT ADVERTISE AS: 0 ()
INT TCP HEARTBEAT TIMEOUT: 700 ()
EXT TCP HEARTBEAT TIMEOUT: 1000 ()
INT TCP HEARTBEAT INTERVAL: 700 ()
EXT TCP HEARTBEAT INTERVAL: 2000 ()
GOSSIP ON SINGLE NODE: False ()
FORCE: False ()
CLUSTER SIZE: 1 ()
NODE PRIORITY: 0 ()
MIN FLUSH DELAY MS: 2 ()
COMMIT COUNT: -1 ()
PREPARE COUNT: -1 ()
ADMIN ON EXT: True ()
STATS ON EXT: True ()
GOSSIP ON EXT: True ()
DISABLE SCAVENGE MERGING: False ()
SCAVENGE HISTORY MAX AGE: 30 ()
DISCOVER VIA DNS: True ()
CLUSTER DNS: fake.dns ()
CLUSTER GOSSIP PORT: 30777 ()
GOSSIP SEED: ()
STATS PERIOD SEC: 30 ()
CACHED CHUNKS: -1 ()
READER THREADS COUNT: 4 ()
CHUNKS CACHE SIZE: 536871424 ()
MAX MEM TABLE SIZE: 1000000 ()
HASH COLLISION READ LIMIT: 100 ()
/var/lib/eventstore ()
INDEX: ()
MEM DB: False ()
SKIP DB VERIFY: False ()
WRITE THROUGH: False ()
UNBUFFERED: False ()
PROJECTION THREADS: 3 ()
WORKER THREADS: 5 ()
ENABLE TRUSTED AUTH: False ()
CERTIFICATE STORE LOCATION: ()
CERTIFICATE STORE NAME: ()
CERTIFICATE SUBJECT NAME: ()
CERTIFICATE THUMBPRINT: ()
CERTIFICATE FILE: ()
CERTIFICATE PASSWORD: ()
USE INTERNAL SSL: False ()
DISABLE INSECURE TCP: False ()
SSL TARGET HOST: n/a ()
SSL VALIDATE SERVER: True ()
AUTHENTICATION TYPE: internal ()
AUTHENTICATION CONFIG: ()
PREPARE TIMEOUT MS: 2000 ()
COMMIT TIMEOUT MS: 2000 ()
UNSAFE DISABLE FLUSH TO DISK: False ()
BETTER ORDERING: False ()
UNSAFE IGNORE HARD DELETE: False ()
INDEX CACHE DEPTH: 16 ()
GOSSIP INTERVAL MS: 1000 ()
GOSSIP ALLOWED DIFFERENCE MS: 60000 ()
GOSSIP TIMEOUT MS: 500 ()
ENABLE HISTOGRAMS: False ()
LOG HTTP REQUESTS: False ()
ALWAYS KEEP SCAVENGED: False ()

[00001,01,14:26:56.736] Quorum size set to 1
[00001,01,14:26:56.745] Cannot find plugins path: plugins
[00001,01,14:26:56.802] INSTANCE ID: 1415579d-8e9a-49fd-b19c-913bfda016fd
[00001,01,14:26:56.803] DATABASE: /var/lib/eventstore
[00001,01,14:26:56.803] WRITER CHECKPOINT: 0 (0x0)
[00001,01,14:26:56.803] CHASER CHECKPOINT: 0 (0x0)
[00001,01,14:26:56.803] EPOCH CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF)
[00001,01,14:26:56.803] TRUNCATE CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF)
[00001,01,14:26:57.004] MessageHierarchy initialization took 00:00:00.1396124.
[00001,01,14:26:57.124] Opened ongoing /var/lib/eventstore/chunk-000000.000000 as version 3
[00001,01,14:26:57.129] Unhandled exception while starting application:
Corrupt database detected.
[00001,01,14:26:57.142] Corrupt database detected.
Unexpected files: /var/lib/eventstore/chunk-000001.000000, /var/lib/eventstore/chunk-000002.000000, /var/lib/eventstore/chunk-000003.000000, /var/lib/eventstore/chunk-000004.000000.
Corrupt database detected.

``

pieter.germishuys · July 17, 2017, 2:02pm

Thanks for the logs, it does help and shows the issue.

As mentioned before, the error can occur due to the absense of the appropriate checkpoints. The following snippet below shows that the checkpoints are missing.

exchange.eventstore | 2017-07-14T14:26:56.842583867Z [00001,01,14:26:56.803] WRITER CHECKPOINT: 0 (0x0)
exchange.eventstore | 2017-07-14T14:26:56.842587143Z [00001,01,14:26:56.803] CHASER CHECKPOINT: 0 (0x0)
exchange.eventstore | 2017-07-14T14:26:56.842590426Z [00001,01,14:26:56.803] EPOCH CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF)

``

The checkpoints are the files alongside the db directory with the .chk extension.

Do you see them present in the db directory?
When you start the container, does it use a restored db?

Greg_Young1 · July 17, 2017, 2:03pm

Yes

exchange.eventstore | 2017-07-14T14:26:56.842583867Z [00001,01,14:26:56.803] WRITER CHECKPOINT: 0 (0x0)
exchange.eventstore | 2017-07-14T14:26:56.842587143Z [00001,01,14:26:56.803] CHASER CHECKPOINT: 0 (0x0)
exchange.eventstore | 2017-07-14T14:26:56.842590426Z [00001,01,14:26:56.803] EPOCH CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF)
exchange.eventstore | 2017-07-14T14:26:56.842593435Z [00001,01,14:26:56.803] TRUNCATE CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF)

Did you try to restore a backup etc? There are files missing.

julian.heeb · July 17, 2017, 3:30pm

The following files are present in db directory:

-rw-r–r-- 1 root root 8 Jul 14 16:18 chaser.chk
-rw-r–r-- 1 root root 268439552 Jul 14 16:18 chunk-000000.000000
-r–r--r-- 1 root root 268439552 Jul 2 21:18 chunk-000001.000000
-r–r--r-- 1 root root 268439552 Jul 8 16:15 chunk-000002.000000
-r–r--r-- 1 root root 268439552 Jul 14 11:12 chunk-000003.000000
-rw-r–r-- 1 root root 268439552 Jul 14 16:18 chunk-000004.000000
-rw-r–r-- 1 root root 8 Jul 14 16:18 epoch.chk
-rw-r–r-- 1 root root 8 Jul 14 16:18 truncate.chk
-rw-r–r-- 1 root root 8 Jul 14 16:18 writer.chk

``

The container has a mounted volume containing these files.

They should be correct, shouldn’t they?

Greg_Young1 · July 17, 2017, 3:38pm

When you restarted, the .chk files would have been recreated if they were not there to start with.

julian.heeb · July 19, 2017, 6:21am

Thanks for your feedback. It might be that the container may be wrongly started or the volume has not been correctly attached while restarting the container. Unfortunately I cannot remember the exact steps we took since it was on a development machine and there was no backup either.**
Regarding the backup:**Is it possible to just copy away first the *.chk files and then the remaining files even while the event store is running? Is it sufficient to backup all the files in `/var/lib/eventstore or are there other necessary files? The documentation is a bit unspecific about which files needs to be backuped, it only says “``Copy the remaining files and directories”…

Thanks a lot!
`

Hayley-Jean_Campbell · July 19, 2017, 6:27am

The backup can be done while Event Store is still running, and yes, it is enough to simply backup all the files in the data directory (/var/lib/eventstore on ubuntu)

You just need to ensure that you always copy the *.chk files before copying the rest.

julian.heeb · July 19, 2017, 1:05pm

Thanks a lot!