Truncating projection stream

Jeremie_Chassaing · July 10, 2017, 9:40am

Hi everybody,

I have projection streams ($ce-category and $et-eventtype kinds) for which loads of original events from the beginning have been deleted from the original steams using $tb.

These projection streams still contain metadata/links to the original events which adds up to lots of space.

Is it safe to use $tb to truncate the projection streams to reclaim space taken by all this old event links with a scavening, or could it cause problemes with the projection system ?

jeremie / thinkbeforecoding

Greg_Young1 · July 10, 2017, 9:51am

See this PR your space will go down when you upgrade. https://github.com/EventStore/EventStore/pull/1286

Jeremie_Chassaing · July 10, 2017, 9:55am

Great !

Thx

Jeremie_Chassaing · July 10, 2017, 10:45am

Do you have an ETA for the release of the 4.0.2 version ?

Jeremie_Chassaing · July 18, 2017, 7:59am

Any idea on a possible release date ?

Ok is it ~safe~ to run a version compiled from source ?

What is the current stability of the 4.0.2 branch ?

pieter.germishuys · July 18, 2017, 8:51am

Jeremie, the 4.0.2 release is currently on hold and the release date pushed back.
Regarding whether it’s safe to run a compiled from source version. The 4.0.2 branch can be seen as a development branch.

Are you looking for the release purely as a means to shrink the size of your DB?

Jeremie_Chassaing · July 18, 2017, 10:25am

Yes

Jeremie_Chassaing · July 18, 2017, 10:37am

Would there be another way to get the same result ?

Greg_Young1 · July 18, 2017, 11:05am

If you are only looking for the scavenge and I will assume you have a production cluster that cannot have downtime.

Compile locally.

Take down one slave node then bring up your locally compiled version.

Scavenge

When complete bring it back to 4.0.1

Repeat.

My understanding though is 4.0.2 is a week or two away from being released so it may be better to just wait.

Jeremie_Chassaing · July 26, 2017, 2:14pm

So the thing didn’t go well… I had to restart a new db from scratch after the db was corrupted.

After starting a 4.0.2, I launched a scavenging to get rid of useless data.

When scavening chunks that where in version 2 got converted to version 3, reporting a lot of freed space:

[PID:04352:043 2017.07.19 10:40:06.154 TRACE TFChunkScavenger ] Forcing scavenged chunk to be kept as old chunk is a previous version.
[PID:04352:043 2017.07.19 10:40:06.154 INFO TFChunkManager ] Switching chunk #2-3 (654052f1-c797-45d9-bd8a-ca522a36f7a8.scavenge.tmp)…
[PID:04352:043 2017.07.19 10:40:06.154 INFO TFChunkManager ] File 654052f1-c797-45d9-bd8a-ca522a36f7a8.scavenge.tmp will be moved to file chunk-000002.000030
[PID:04352:043 2017.07.19 10:40:06.154 DEBUG TFChunk ] Opened completed F:/Storage/EventStore3/Events\chunk-000002.000030 as version 3
[PID:04352:043 2017.07.19 10:40:06.154 INFO TFChunk ] File chunk-000002.000029 has been marked for delete and will be deleted in TryDestructFileStreams.
[PID:04352:043 2017.07.19 10:40:06.201 INFO TFChunkManager ] Old chunk ##2-3 (chunk-000002.000029) is marked for deletion.
[PID:04352:043 2017.07.19 10:40:06.201 TRACE TFChunkScavenger ] Scavenging of chunks:
[PID:04352:043 2017.07.19 10:40:06.201 TRACE TFChunkScavenger ] #2-3 (chunk-000002.000029)
[PID:04352:043 2017.07.19 10:40:06.201 TRACE TFChunkScavenger ] completed in 00:02:56.2062012.
[PID:04352:043 2017.07.19 10:40:06.201 TRACE TFChunkScavenger ] New chunk: 654052f1-c797-45d9-bd8a-ca522a36f7a8.scavenge.tmp --> #2-3 (chunk-000002.000030).
[PID:04352:043 2017.07.19 10:40:06.201 TRACE TFChunkScavenger ] Old chunks total size: 262155372, scavenged chunk size: 71379.
[PID:04352:043 2017.07.19 10:40:06.201 TRACE TFChunkScavenger ] SCAVENGING: started to scavenge & merge chunks: #4-5 (chunk-000004.000019)

But something strange is that the chunks where all resized to the same size (around 256MB), whatever the scavenge size was.

The last few chunks where a bit bigger (a few MB).

And when the last chunk has been scavenged, it tried to merge, and this is where it went wrong:

PID:04352:043 2017.07.19 16:01:08.899 TRACE TFChunkScavenger ] SCAVENGING: stopping scavenging pass due to non-completed TFChunk for position 81604378624.
[PID:04352:043 2017.07.19 16:01:08.899 TRACE TFChunkScavenger ] SCAVENGING: initial pass completed in 05:27:48.6852755.
[PID:04352:043 2017.07.19 16:01:08.914 TRACE TFChunkScavenger ] SCAVENGING: started to scavenge & merge chunks: #0-1 (chunk-000000.000029)
#2-3 (chunk-000002.000030)
#4-5 (chunk-000004.000020)

…

#262-262 (chunk-000262.000001)
#263-263 (chunk-000263.000006)
[PID:04352:043 2017.07.19 16:01:08.914 TRACE TFChunkScavenger ] Resulting temp chunk file: b1f65ca1-a75c-44c3-bc01-578dbea2ce89.scavenge.tmp.
[[PID:04352:043 2017.07.19 16:02:18.894 INFO TFChunkScavenger ] Got exception while scavenging chunk: #0-263. This chunk will be skipped
Exception: System.Exception: Could not read latest stream’s prepare. That should never happen.
at EventStore.Core.Services.Storage.ReaderIndex.IndexReader.GetStreamLastEventNumberUncached(TFReaderLease reader, String streamId) in C:\Development\EventStore\src\EventStore.Core\Services\Storage\ReaderIndex\IndexReader.cs:line 392
at EventStore.Core.Services.Storage.ReaderIndex.IndexReader.GetStreamLastEventNumberCached(TFReaderLease reader, String streamId) in C:\Development\EventStore\src\EventStore.Core\Services\Storage\ReaderIndex\IndexReader.cs:line 375
at EventStore.Core.Services.Storage.ReaderIndex.IndexReader.GetStreamLastEventNumberCached(TFReaderLease reader, String streamId) in C:\Development\EventStore\src\EventStore.Core\Services\Storage\ReaderIndex\IndexReader.cs:line 363
at EventStore.Core.Services.Storage.ReaderIndex.IndexReader.EventStore.Core.Services.Storage.ReaderIndex.IIndexReader.GetStreamLastEventNumber(String streamId) in C:\Development\EventStore\src\EventStore.Core\Services\Storage\ReaderIndex\IndexReader.cs:line 347
at EventStore.Core.TransactionLog.Chunks.TFChunkScavenger.ShouldKeepPrepare(PrepareLogRecord prepare, Dictionary2 commits, Int64 chunkStart, Int64 chunkEnd) in C:\Development\EventStore\src\EventStore.Core\TransactionLog\Chunks\TFChunkScavenger.cs:line 358 at EventStore.Core.TransactionLog.Chunks.TFChunkScavenger.<>c__DisplayClass12_1.<ScavengeChunks>b__3(PrepareLogRecord prepare) in C:\Development\EventStore\src\EventStore.Core\TransactionLog\Chunks\TFChunkScavenger.cs:line 192 at EventStore.Core.TransactionLog.Chunks.TFChunkScavenger.TraverseChunk(TFChunk chunk, Action1 processPrepare, Action1 processCommit, Action1 processSystem) in C:\Development\EventStore\src\EventStore.Core\TransactionLog\Chunks\TFChunkScavenger.cs:line 516
at EventStore.Core.TransactionLog.Chunks.TFChunkScavenger.ScavengeChunks(Boolean alwaysKeepScavenged, IList1 oldChunks, Int64& spaceSaved) in C:\Development\EventStore\src\EventStore.Core\TransactionLog\Chunks\TFChunkScavenger.cs:line 187. [PID:04352:043 2017.07.19 16:02:18.940 ERROR TFChunkScavenger ] Failed to delete the temp chunk. Retrying 0/5. Reason: System.IO.IOException: The process cannot access the file 'F:\Storage\EventStore3\Events\b1f65ca1-a75c-44c3-bc01-578dbea2ce89.scavenge.tmp' because it is being used by another process. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.File.InternalDelete(String path, Boolean checkHost) at EventStore.Core.TransactionLog.Chunks.TFChunkScavenger.DeleteTempChunk(String tmpChunkPath, Int32 retries) in C:\Development\EventStore\src\EventStore.Core\TransactionLog\Chunks\TFChunkScavenger.cs:line 280 [PID:04352:043 2017.07.19 16:02:19.237 ERROR TFChunkScavenger ] Failed to delete the temp chunk. Retrying 1/5. Reason: System.IO.IOException: The process cannot access the file 'F:\Storage\EventStore3\Events\b1f65ca1-a75c-44c3-bc01-578dbea2ce89.scavenge.tmp' because it is being used by another process. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.File.InternalDelete(String path, Boolean checkHost) at EventStore.Core.TransactionLog.Chunks.TFChunkScavenger.DeleteTempChunk(String tmpChunkPath, Int32 retries) in C:\Development\EventStore\src\EventStore.Core\TransactionLog\Chunks\TFChunkScavenger.cs:line 280 [PID:04352:043 2017.07.19 16:02:19.237 ERROR TFChunkScavenger ] Failed to delete the temp chunk. Retrying 2/5. Reason: System.IO.IOException: The process cannot access the file 'F:\Storage\EventStore3\Events\b1f65ca1-a75c-44c3-bc01-578dbea2ce89.scavenge.tmp' because it is being used by another process. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.File.InternalDelete(String path, Boolean checkHost) at EventStore.Core.TransactionLog.Chunks.TFChunkScavenger.DeleteTempChunk(String tmpChunkPath, Int32 retries) in C:\Development\EventStore\src\EventStore.Core\TransactionLog\Chunks\TFChunkScavenger.cs:line 280 [PID:04352:043 2017.07.19 16:02:19.237 ERROR TFChunkScavenger ] Failed to delete the temp chunk. Retrying 3/5. Reason: System.IO.IOException: The process cannot access the file 'F:\Storage\EventStore3\Events\b1f65ca1-a75c-44c3-bc01-578dbea2ce89.scavenge.tmp' because it is being used by another process. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.File.InternalDelete(String path, Boolean checkHost) at EventStore.Core.TransactionLog.Chunks.TFChunkScavenger.DeleteTempChunk(String tmpChunkPath, Int32 retries) in C:\Development\EventStore\src\EventStore.Core\TransactionLog\Chunks\TFChunkScavenger.cs:line 280 [PID:04352:043 2017.07.19 16:02:19.237 ERROR TFChunkScavenger ] Failed to delete the temp chunk. Retrying 4/5. Reason: System.IO.IOException: The process cannot access the file 'F:\Storage\EventStore3\Events\b1f65ca1-a75c-44c3-bc01-578dbea2ce89.scavenge.tmp' because it is being used by another process. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.File.InternalDelete(String path, Boolean checkHost) at EventStore.Core.TransactionLog.Chunks.TFChunkScavenger.DeleteTempChunk(String tmpChunkPath, Int32 retries) in C:\Development\EventStore\src\EventStore.Core\TransactionLog\Chunks\TFChunkScavenger.cs:line 280 [PID:04352:043 2017.07.19 16:02:19.237 ERROR TFChunkScavenger ] Failed to delete the temp chunk. Retry limit of 5 reached. Reason: System.IO.IOException: The process cannot access the file 'F:\Storage\EventStore3\Events\b1f65ca1-a75c-44c3-bc01-578dbea2ce89.scavenge.tmp' because it is being used by another process. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.File.InternalDelete(String path, Boolean checkHost) at EventStore.Core.TransactionLog.Chunks.TFChunkScavenger.DeleteTempChunk(String tmpChunkPath, Int32 retries) in C:\Development\EventStore\src\EventStore.Core\TransactionLog\Chunks\TFChunkScavenger.cs:line 280 [PID:04352:043 2017.07.19 16:02:19.378 ERROR StorageScavenger ] SCAVENGING: error while scavenging DB. System.IO.IOException: The process cannot access the file 'F:\Storage\EventStore3\Events\b1f65ca1-a75c-44c3-bc01-578dbea2ce89.scavenge.tmp' because it is being used by another process. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.File.InternalDelete(String path, Boolean checkHost) at EventStore.Core.TransactionLog.Chunks.TFChunkScavenger.DeleteTempChunk(String tmpChunkPath, Int32 retries) in C:\Development\EventStore\src\EventStore.Core\TransactionLog\Chunks\TFChunkScavenger.cs:line 292 at EventStore.Core.TransactionLog.Chunks.TFChunkScavenger.DeleteTempChunk(String tmpChunkPath, Int32 retries) in C:\Development\EventStore\src\EventStore.Core\TransactionLog\Chunks\TFChunkScavenger.cs:line 287 at EventStore.Core.TransactionLog.Chunks.TFChunkScavenger.DeleteTempChunk(String tmpChunkPath, Int32 retries) in C:\Development\EventStore\src\EventStore.Core\TransactionLog\Chunks\TFChunkScavenger.cs:line 287 at EventStore.Core.TransactionLog.Chunks.TFChunkScavenger.DeleteTempChunk(String tmpChunkPath, Int32 retries) in C:\Development\EventStore\src\EventStore.Core\TransactionLog\Chunks\TFChunkScavenger.cs:line 287 at EventStore.Core.TransactionLog.Chunks.TFChunkScavenger.DeleteTempChunk(String tmpChunkPath, Int32 retries) in C:\Development\EventStore\src\EventStore.Core\TransactionLog\Chunks\TFChunkScavenger.cs:line 287 at EventStore.Core.TransactionLog.Chunks.TFChunkScavenger.DeleteTempChunk(String tmpChunkPath, Int32 retries) in C:\Development\EventStore\src\EventStore.Core\TransactionLog\Chunks\TFChunkScavenger.cs:line 287 at EventStore.Core.TransactionLog.Chunks.TFChunkScavenger.ScavengeChunks(Boolean alwaysKeepScavenged, IList1 oldChunks, Int64& spaceSaved) in C:\Development\EventStore\src\EventStore.Core\TransactionLog\Chunks\TFChunkScavenger.cs:line 270
at EventStore.Core.TransactionLog.Chunks.TFChunkScavenger.Scavenge(Boolean alwaysKeepScavenged, Boolean mergeChunks) in C:\Development\EventStore\src\EventStore.Core\TransactionLog\Chunks\TFChunkScavenger.cs:line 105
at EventStore.Core.Services.Storage.StorageScavenger.Scavenge(ScavengeDatabase message) in C:\Development\EventStore\src\EventStore.Core\Services\Storage\StorageScavenger.cs:line 99
[PID:04352:032 2017.07.19 16:02:22.878 TRACE QueuedHandlerThreadP] SLOW QUEUE MSG [StorageReaderQueue #3]: ReadStreamEventsBackward - 531ms. Q: 0/0.

I tried to restart the service but there was still the error ‘Could not read latest stream’s prepare. That should never happen.’

[PID:07340:024 2017.07.19 16:11:32.476 INFO AdminController ] Request scavenging because /admin/scavenge request has been received.
[PID:07340:024 2017.07.19 16:11:32.508 TRACE TFChunkScavenger ] SCAVENGING: started scavenging of DB. Chunks count at start: 305. Options: alwaysKeepScavenged = False, mergeChunks = True
[PID:07340:024 2017.07.19 16:11:32.523 TRACE TFChunkScavenger ] SCAVENGING: started to scavenge & merge chunks: #0-1 (chunk-000000.000029)
[PID:07340:024 2017.07.19 16:11:32.523 TRACE TFChunkScavenger ] Resulting temp chunk file: 1ab503b9-c697-40c7-90a3-86336b63e8b3.scavenge.tmp.
[PID:07340:024 2017.07.19 16:11:32.554 INFO TFChunkScavenger ] Got exception while scavenging chunk: #0-1. This chunk will be skipped
Exception: System.Exception: Could not read latest stream’s prepare. That should never happen.
at EventStore.Core.Services.Storage.ReaderIndex.IndexReader.GetStreamLastEventNumberUncached(TFReaderLease reader, String streamId) in C:\Development\EventStore\src\EventStore.Core\Services\Storage\ReaderIndex\IndexReader.cs:line 392

If it can help find what the prb is.

thinkbeforecoding

pieter.germishuys · July 26, 2017, 2:29pm

Can you possibly provide us with full logs which include the startup of the node?