Single node with multiple stores

Some months ago I was asking whether it is planned to provide the possibility in GES to have a single node that can create/manage multiple independent event stores similar to a normal database like Mongo DB or SQL Server. If yes, then what is the time horizon for this feature?
In our case this would significantly reduce the complexity of the setup since currently we need to run an instance of GES for each event store.

How many instances?

We have up to 14 so far… more are possible

We have talked about this internally as there are many paths to it. With manager in the closed source version doing 14 would be quite easy. What level of isolation is needed between the stores?

This has always been an interesting question for me as the normal answer is “file” however if they reside in the same process there is always a slim but existing possibility that data from one could be returned to a query from another (think bad reads,corrupted memory etc etc).

Doing process level isolation removes this possibility but also limits the scalability of such a solution to say 100 processes (just throwing out a number) even with simplified management. If done at a different level which is also possible what should be shared between the instances?

Greg

Sorry for the delay… Thank you for investigating this.
We have one particular install where we have one application instance serving up to 30 sites. Although each site belongs to the same company we want to keep the data separate. Main reason is ability to add/remove a site and/or backup/restore sites individually.

I am not sure though whether that is really the best way to do it… My thinking is just influenced a lot by how other servers like MongoDB, Oracle or SQL Server provide you the option of having multiple individual schemas or databases.

The instances don’t need to share anything (data wise).

You mentioned possible corruptions but then I ask myself how do services like Mongo DB handle this potential issue?

If they share anything a socket, some memory, etc there is always a risk client 1 will see client 2s data though it can be a small one :slight_smile:

Would you want each client to have their own endpoints or one endpoint for the entire client set?

Comparing with e.g. MongoDB it would make sense to have only one end point but we can also easily handle an end point per connection. The former of course requires less configuration…
Also we have to consider what sys admins expect. Usually they are used to deal with servers like Oracle, Sql Server or Mongo DB where each server runs in a single process. They on the other hand get shocked if we tell them that we need to run 30 instances (processes) for 30 sites… You know probably better than me that ES is still not a main stream “architecture”/ architectural pattern and even less do people from operations know anything about GES. And I am talking about big(!) enterprises…

I can see a few arguments for sharing the TCP stack etc, but not sharing the back end. One is, for example, to have different tenants data stored on disks with different encryption keys. However, if the data is that critical I find it hard to believe that
sharing the rest of the stack would be OK.

One thing we’ve discussed is a “router” module presents a unified set of endpoints for many clusters, which might be a reasonable way of achieving this. It’s very unlikely to be liberally licensed however, and will likely require a license fee for production
use. I assume at the moment you’re bringing up a node or a cluster per tenant?

James

Currently we are using a Windows service that just starts a node per site (up to 30). The service picks the info from a config file where we have an entry per site. Thus we might run 30 processes on a single box which I find aehm… a bit wasteful. We are using version 2.0.1 and we still have the memory problem that I mentioned in an earlier thread. We have to start the individual GES nodes with skipDBverify=true to keep memory under control…

If I compare that with Mongo DB which we use for our read model (next to Lucene) we only have one Mongo DB server process which manages 30 DBs without problems.

…and I should mention that here we talk about one company that has 30 sites. Thus it is not exactly the classical multi tenant situation

Is this the Windows File System Cache using memory problem?

If it is please let us know your operating system version of windows and we can tell you how to limit your file cache. By default windows says it can take up all memory (as the memory is unused). This can cause issues in server enviroments as it will start paging. For most environments its as simple as a powershell script that sets FileSystemCacheSize (win api) http://msdn.microsoft.com/en-us/library/windows/desktop/aa965240(v=vs.85).aspx.

There are things under way to bypass the cache completely as well.

Yes it is the file cache problem…

“There are things under way to bypass the cache completely as well.”

It is a big relief to hear that! You cannot imagine how many meetings with ops guys we already had because of the memory issue. We can see the issue on our developer machines (Windows 8.1 64 bit) as well as Windows Server 2008 R2 and 2012 R2 which we use as server OS in production

For 2008 and 2012 just set the file system cache size and the issue will go away. I have not tried windows 8.1.

Didn’t go all out with it but here:

https://github.com/EventStore/EventStore/commit/e89481e6112c77f7a59bf9775bf0a4b089900105

Its on dev now if you want to test it

You will still have to limit your filesystem cache as the reads etc still hit it but the db for tfchunks and ptables check no longer does (I figured the index map which is 1kb is fine to leave as is).

Cheers,

Greg

That’s great. I’ll give it a try tonight.
Thanks for your (team’s) great work and responsiveness!

Best

Gabriel

Count me in as someone who really wants >1 store per node (embedded too).

My particular use case is we can bundle one or more ES based apps into a single “suite”. Some of these apps are small and so I’m not massively worried about things like size and it’s all for the one customer so not worried about tenancy nor different encryption keys, etc. (I do care about correctness oc). I absolutely want each app to have it’s own logical store and each their own $all stream. NEventStore (yeah yeah I know :slight_smile: works well in this case either by using a different sql db on same sql instance or, in some cases, buckets.

I think this is a separate concern to clustering.

Cheers