Subscription performance in a cluster?

EventStoreDB
#1

I’m doing some performance testing. I’m seeing very slow reads from my cluster. Please give me a sanity check on this.

On a single node, I can WRITE about 100K events/sec.

On a single node, I can READ (via catchup sub) about 200K events/sec.

On a 3-node cluster, I can WRITE about 17K events/sec.

On a 3-node cluster, I can READ (via catchup sub) about 33K events/sec.

Of these, only the last number seems weird. It feels low. I expect the cluster penalty on writes, but not on reads. I would have expected that reads could be serviced by the local node, and that throughput would be close to the 200K single node read speed.

What is the expected relative read speed on a cluster?

Thanks,

  • Sam
#2

What node types etc are you running? Is one test local while the other is over a network etc? You should see much better read performance over a cluster (also worth checking that you are not connecting all subscriptions to same node…)

#3

Nothing fancy. No mode specified, so 2 slaves and 1 master. All three nodes are on identical machines, with command lines much like this:

EventStore.ClusterNode.exe --db d:\cluster\db --log d:\cluster\logs --int-ip 192.168.17.155 --ext-ip 192.168.17.155 --int-tcp-port=1212 --ext-tcp-port=1213 --int-http-port=2212 --ext-http-port=2213 --cluster-size=3 --discover-via-dns=false --gossip-seed=192.168.17.151:2212,192.168.17.164:2212 --RunProjections=All

/gossip shows that they all look healthy, unless I’m missing something.

The systems are only for my testing, and only 1 subscription is being run at a time!

The non-clustered instance runs on my local box. As does one of the clustered nodes, and this the one I’m connecting to.

-Sam

#4

I’m confused by your benchmarks…

  1. you are reading twice as fast as you are writing. I assume this is write a bunch of stuff the later read a bunch of stuff since doing at the same time makes no sense?

The non-clustered instance runs on my local box. As does one of the clustered nodes, and this the one I’m connecting

Nothing fancy. No mode specified, so 2 slaves and 1 master. All three nodes are on identical machines, with command lines much like this:

So all three are identical to your local box?

You mention that you connect to your local box … You are not using a clustered connection? Also if it’s only one subscription you would not see a performance increase, you would likely see the same performance (with n subscriptions you would see an increase then) a cluster connection btw will by default not connect to your local node but to the master node

But again benchmarking can be tricky telling as much about your setup as possible helps as very often it’s just config differences etc that make a big difference.

Greg

#5

Yep. Testing different parts in isolation. I wrote a few tens of million events, let the cluster settle and default projections catch up. Now I’m testing how fast I can read from it.

Yes, all machines are identical, and all nodes are also configured the same. Basically, that command line is cut n’ pasted, with only IP addresses being changed.

OK, I may have missed something. I don’t know what a “clustered connection” is. I’m just calling EventStoreConnection.Create() with default options. Other than changing the endpoint IP, the client code is 100% unchanged from when I tested against unclustered.

I get that I wouldn’t see a performance increase, I just didn’t expect such a drastic decrease from the non-clustered case.

-Sam

#6

OK, now that I know it exists, I found the docs.

I’ll amend my test code to use a clustered connection later tonight.

But to sanity check this: if a client uses the wrong connection type, I should expect this poor performance as a result? Perhaps an exception would be a better result?

-Sam

#7

You shouldn’t get a drastic performance decrease which makes me question the test. We have run many such similar tests. On a single subscription btw the single largest thing that controls performance is latency on a read operation. It could be many things that cause a difference in latency of a single operation (even things like cached vs uncached data).

As I understand you have a basic app that uses connect to specific node (local) then either writes 10m events and opens a catchup subscription from same node after some period of time…

  1. you should be able to improve write performance by connecting to master node. The connection if running with seeds/DNS tries to do this for you by default (cluster config settings)

  2. I’d be interested in seeing what happens in a general read from in terms of latency between the two running environments of an event that’s a ways back (eg not recently written) my guess is you are seeing higher latency on the clustered node.

  3. your node is definitely in a slave state? If not it might be forwarding to another node the requests… If you bring up queues screen do you see that node fulfilling the reads?

#8

Greg,

Sorry for the late responses. Product launches don’t like to wait. :slight_smile:

TLDR; I’m all good now.

I think the problem has cleared itself. When I added the ClusterSettings, on my first run one of the nodes crashed. It apparently took the log file with it, so I’m not sure what happened. :frowning: Upon restart, it’s been rock solid.

Oddly, since the restart, performance numbers have been good across the board. I’m up to 160K reads/sec. To verify, I reverted my test code back to the non-clustered revision, and the performance stayed high.

So my best guess is that the cluster was, in some way, unhealthy. It hasn’t happened before or since. Some these are desktop PCs, so I’m going to shrug it off as some weird environmental interaction unless it happens again.

Thanks again for the fast turnaround; it reflects well on the product.

-Sam

#9

"When I added the ClusterSettings, on my first run one of the nodes
crashed. It apparently took the log file with it, so I'm not sure
what happened."

I would love to see it. I'm not sure how a node crash could take out a
log its normally just written to a file/etc. There are times a node
will come up and commit suicide as the best way to fix itself you
might have hit one of those and maybe missed logging as it was only to
stdout etc.

" Upon restart, it's been rock solid."

I'm guessing it might have been something like the above.