Unexpected error on server: entry is bigger than endKey

EventStoreDB
#1

Hi,

we are getting such errors time after time when trying to read stream (EventStoreConnection.ReadStreamEventsForwardAsync):

“ClassName”: “EventStore.ClientAPI.Exceptions.ServerErrorException”,

“Message”: “Unexpected error on server: entry (7769916169653612193@2) > endKey Stream: 7769916168253838929, Version: 0, stream 7769916168253838929, startNum 0, endNum 99, PTable: d:\Store\Banners\index\bdf3d0be-cb0d-4e14-a5af-b31e81dad976.”

We updated our cluster to Event Store 3.9.1.0 lately.

Could this be related?

Or does this indicate somewhat data corruption?

What could be done to fix the issue?

Thanks.

#2

Hi,

The solution here is to rebuild your indexes.
We are currently working on a patch to fix this particular issue

#3

what is the procedure to rebuild an index?
on single?

on cluster?

#4

To rebuild an index, simply bring down the node, delete the index folder in your data directory, and start the node up again.
The node will detect that the index is missing and will rebuild it.

Nodes in a cluster each have their own db and index, so in order to rebuild all of the indexes for a cluster, you would need to do the above for each node.

You can do this for one node at a time to lessen your cluster’s down time.

Please note that an index rebuild can take a long time depending on the size of your db.

#5

It's not generating now in docs but you can also look here:
https://github.com/EventStore/docs.geteventstore.com/blob/gh-pages/server/3.9.0/64-bit-index.md

#6

Just wanted to touch base with you to see if everything is fine on your side now after rebuilding the indexes?

Thank you.

#7

We do not get any errors that I mentioned in the original post.
However we got one different:

“ClassName”: “EventStore.ClientAPI.Exceptions.ServerErrorException”,

“Message”: "Unexpected error on server: Invalid order of events has been detected in read index for the event stream ‘Tag-6e0bc6c2-988b-4d6c-a7ca-52d95cbabdc2’. The event 1

at position 256718925694 goes after the event 1

at position 239239686010"

Not sure if this could still be a 3.9.1 problem.

Indexes are rebuilt on all nodes.

#8

It seems that this stream is corrupted, soft delete and recreation does not help.

#9

Can you please supply us with logs?

#10

HI, I have sent you logs via email.