Resource contention

EventStoreDB
1

So I’m reaping the benefits of logging events from Github constantly since my last e-mail to this group

I’ve not got any projections on that server, I deliberately didn’t write any because development of projections against a large and changing data-set is a PITA at the moment WRT to development experience.

Now I want to write some projections, so I’m working on a development version (5% of the event count) and then setting the projection code on the full data-set.

However, I’ve ran into a roadblock as my server is now pretty much dead as the dataset is crunched through a single projection (96% usage) just doing a count of pushes.

I can’t load most of the http pages from the server and I certainly can’t view the state of that projection or upload any more.

Is there some setting somewhere to say “Hey, make the projections slower and give priority to other things?”

What is the expected behaviour when giving the server a large number of projections/heavy projection usage? (It’s only 2.5million events or so, it’s a small instance on Amazon though (not a micro, I pay for this))

I can scale up for now, I’m just wondering how I’d insulate myself against this sort of thing in a production environment. Multiple nodes for running projections and a set of write-only nodes?

2

NVM just saw the CL options (and I’ve copied the data over to an xlarge instance while I run the projections)

3

Does this look okay? http://codeofrob.com/entries/evented-github-adventure—database-storage-and-backing-up.html

4

Hi Rob,

projections run on a fixed number of threads (configurable via config file or command line). If a projection emits a lot of events/results it is typically IO bound and does not slow down HTTP interface and reads remain pretty fast (I typically run on SSD). If the projection does not emit much it consumes all the CPU time of the thread, assuming event reader is fast enough (typically it is).

The default projection settings are 3 threads and it is too high value for EC2 small instance. I think 1 projection thread is the max possible value for this configuration.

-yuriy

5

Likely you also want lower numbers for top and http sending threads as well as http processing threads (all can be set through command line)