Performance tuning of EsConnection all-streams Publisher?

EventStoreDB
#1

Hi guys,

I have a local Java app that creates an
EsConnection.allStreamsPublisher to read all events from an EventStore
(3.6.3) running on EC2. Client library version is 2.2.2.

I set initial demand to be 4096, and call subscription.request(1)
whenever an event has been processed, which is done asynchronously on
onNext.

The performance I'm seeing is less than 200 events per second, which
feels a bit low. I check my router traffic manager, and it's not even
close to hitting the bandwidth limit, so feels like it shouldn't be a
network issue. read-batch-size is set to 1024.

Do you have any idea why the throughput is so low? Any settings that
can be tweaked? This is basically a batch read, so I don't care about
latency, only throughput.

thanks, Rickard

#2

". read-batch-size is set to 1024."

You should be able to get way more than 200/second barring the events
are not huge (say 1m each etc)

What do you get bringing up a simple subscriber that just counts the
number of events?

#3

Hi there,

The events are "normal sized", nothing out of the ordinary at all, so
I don't think it's that. We have a Logstash input plugin which does
the same thing but through the HTTP interface, and it finishes our
2.5M events fairly quickly. It's just this EsConnection client that
seems wrong somehow, or I'm doing something wrong.

#4

If http is fast tcp should be faster.

How far away is your code from the node? eg is it running locally with
a node off in the cloud etc where latency could be an issue?

#5

Hey there,

If http is fast tcp should be faster.

How far away is your code from the node? eg is it running locally with
a node off in the cloud etc where latency could be an issue?

Yeah, so I'm in Malaysia, the node is off in US East. Latency could be
an issue, for sure.

However, to try and check whether it's a network issue I built my own
REST API client, and tried that. Iterating the $all stream backward
from the beginning, 4096 items at a time, with gzip compression, I get
about 1500 events/sec.

Without gzip compression, dead slow. I can also see that a single
uncompressed fetch retrieves a mean of 5.3MB, or ~1300B/event. My own
network has no issue with this, but it's far from here to US East.

So, it seems to be network related, and my current fix appears to be
to use my own REST API client, with gzip compression, unless
EsConnection can do that as well?

Any other ideas?

regards, Rickard