Windows system file cache

We are just getting ready to begin using Event Store in our project and will be running it on Windows Server 2012 on a virtual machine. We need to decide whether to provision a new virtual machine to dedicate to Event Store or whether we can reasonably run it on the same one where we are hosting RavenDB. From a limited simulation of our anticipated usage, it seems that we have sufficient CPU and I/O resources to host them on the same virtual machine.

However, memory usage has me concerned. I have seen the following comment come up a few times in this group: “We commonly see in windows file cache eating up space and not releasing it in time. You can limit file cache on the machine using things like sysinternals https://technet.microsoft.com/en-us/sysinternals/bb897561” I am concerned that my limited simulation, and even our regular QA verification process, is not going to expose this problem and we won’t find it until it takes our production system down.

So I have some questions:

  1. Is the file cache problem still a potential issue with Event Store 3.9 on Windows 2012?

The following MSDN article leads me to believe that file caching is no longer broken in Windows 2012 and later: https://msdn.microsoft.com/en-us/library/windows/hardware/dn567645(v=vs.85).aspx

  1. Does it only become a problem when the machine approaches saturation of physical memory?

  2. Does the situation change if we change Event Store parameters like the chunk file size, or the number of cached chunks?

  3. If it still is a problem, why is Event Store particularly affected while I’ve never heard of another Windows database server that is?

Thanks.

1) Is the file cache problem still a potential issue with Event Store
3.9 on Windows 2012?
The following MSDN article leads me to believe that file caching is no
longer broken in Windows 2012 and later:
https://msdn.microsoft.com/en-us/library/windows/hardware/dn567645(v=vs.85).aspx

No idea on later versions. Normally we just respond when people are
having the issue.

2) Does it only become a problem when the machine approaches
saturation of physical memory?

Yes.

3) Does the situation change if we change Event Store parameters like
the chunk file size, or the number of cached chunks?

No as its not Event Store.

4) If it still is a problem, why is Event Store particularly affected
while I've never heard of another Windows database server that is?

Because we allow the file cache to be used. Most bypass it as for say
a SQL db the db knows better than the OS which pages to cache. As we
really want exactly what the file cache provides using it instead of
building our own works better in terms of playing well with others.
The OS knows more than we do.

Thank you.

I guess we’ll just have to formulate a stress test on our own systems.

What are the operations in Event Store that would be the most troublesome for the file cache? Scavenge maybe? A catch-up subscription on all events from the beginning?

Just a db larger than the size of memory

the testclient and wrfl command can help make one of these quickly
(try wrfl 10 500000000)

Excellent. Thank you again.

In case anyone is interested, I have done a successful memory stress test on a Windows Server 2012 virtual machine.

Using Event Store 3.9.0, I created a database of about 29GB on a virtual machine with 24GB. Then I started a Scavenge operation and simultaneously ran two clients that read forward through all the events in the store.

I watched the cache working set grow until it filled all the free memory (about 19GB), but when the machine reached zero bytes free, the cache working set did not continue to grow. The Event Store clients continued reading as normal. The scavenge continued to completion. The machine was still responsive both on the network and through the GUI, and other processes (including a RavenDb server) were able to start and consume memory without unusual paging.

During the test, the Event Store process itself used only up to 1.7GB.

So it seems file caching is no longer broken on 64-bit windows.