I just restarted my ES with --skip-db-verify (because that was taking about 4-5 minutes for each chunk file) and now the StorageWriterQueue is showing 100% busy with a current/last message of "SystemInit / ".
The ES is not responding at all at this time and it won’t accept any events for append.
Help!
Thanks,
Chris
The ES finally came to life after about 20 minutes.
Can anyone take a guess at what may have happened here? Is this normal startup behavior for a 3.2GB db? Is there anyway to speed it up?
Would I have been better without --skip-db-verify?
Thanks,
Chris
Chris,
Where do you store chunk files? Is it regular HDD? NAS?
Br,
Yuriy
It’s an attached data disk in an Azure VM.
It seems like your storage is very very slow, because normally one chunk verification takes few seconds, as it is sequencial buffered read and MD5 hash calculation.
As for what is done during SystemInit in StorageWriterQueue – the node on start-up rebuild last up to 1 mln events into MemTable (in-memory part of our read index). If verification takes 4-5 minutes on your storage, then it’s perfectly ok if more complicated process of index rebuilding takes 20 minutes.
Try this on local computer and it should be A LOT faster.
It’s definitely a lot faster locally. But we’re running this in production. 
Once it’s running, it’s fast as snot.
Besides startup, will the StorageWriterQueue ever rebuild the memory index?
no, just on startups. And it is not always 1mln. We dump to disk every full million of events, so if after last dump you wrote just thousand events, on restart rebuild process will be almost instanteous.
But it is very strange that this process is so slow on Azure, we’ll investigate that soon.
Chris,
what is the exact VM configuration/settings in Azure?
-yuriy
Andrii,
Thank you!
Yuriy,
Windows Server 2012
Medium (2 cores, 3.5 GB Memory)
50 GB Data Disk <- EventStore
This sounds like what we were discussing in the other thread that if you turned off md5 checks then something else would be slow as that included the bringing over of the data. My guess is during this process all the data is getting brought across the network then cached on the local drive.
It’s a real drag. 
Is this any better on AWS? I’m not familiar with their offerings.
Chris,
is you DB on the system drive where windows resides? If you attached another drive for the ES storage, how is caching configured? none/readonly/readwrite?
-yuriy
Nice find, Yuriy.
That disk is currently set to none.
-=Chris
What cache setting should I use for optimal performance?
-=Chris