Hi,
Is there documentation on the working of what each commands do and their impact ?
I just tripled the size of the DB using verify and the 1 000 000 default value for events does not seem to be working, except maybe if it means 1 million per readers / writers
Thanks
TestClient is mostly an internal testing/diagnostic tool. We
distribute it so we have known measurements. What command were you
running typing help will tell you what the parameters are.
Oh, I thought it was a tool that could be used by anybody to test / verify (as in validate, not the verify command) the event store.
Well in this case, I ran verify thinking it would check the data that was inside the event store (kind of like a checkdb) but it actually creates fake banking data. But I have no clue how the parameters are linked to one another since using the default it created about 5 million events which corresponds to no combination of: writes operand events operand streams
So we get something like:
WR => writes something to a stream. Though I have no idea on how to give it the data it wants at this time.
WRJ => best guess is that it requires JSON and it is a write json to a specific stream
WRFL => Writes flood: Writes events to streams does not wait for the flush to disk
WRFLCA => no clue what the CA stands for
WRFLW => Writes flood and wait for the flush to disk for each event
MWR => Memory Writes for when the event store runs as mem db is my guess
MWRFLW => Memory writes flood and wait for flush to memory when the event store runs as mem db is my guess
TWR => Writes to a specific thread is my guess
RDALL => read all
RD => Read from a stream
RDFL => read flood
WRLT => Writes on a timer is my guess
VERIFY => Creates fake banking data, around 5 million events i think
SCAVENGE => Scavenge, but what, I got no clue. Or maybe it is just a purge for everything that is defined as being ‘old’ via the ‘max’ metadata of the streams
SUBSCR => Subscribe to streams
CHKTCP => Check TCP, but no clue what it does. Open the tcp port to see if it communicates ? Considering the bugs / issues I sometimes have with the event store I have a feeling that one does not work properly. IE: If I run RDFL or WRFL while one of our dev runs a read test from the application I can actually get disconnected from the Event Store or I also get a lot of fails on writing data. Concurrent connections don’t seem to be appreciated at all. And I end up with a TCP handle error.
And I have no clue what these do especially when seeing the parameters:
SST
PING
PINGFL
PINGFLW
And USAGE seems to do the same thing as HELP
Also some of the parameters befuddle me like commit position and prepare position on the read all call.
"Open the tcp port to see if it communicates ? Considering the bugs /
issues I sometimes have with the event store I have a feeling that one
does not work properly. IE: If I run RDFL or WRFL while one of our dev
runs a read test from the application I can actually get disconnected
from the Event Store or I also get a lot of fails on writing data.
Concurrent connections don't seem to be appreciated at all. And I end
up with a TCP handle error."
I have yet to see any reported bugs from you, please do send a reproduction.
Test client does no retries or reconnects. The client connection (java
or .net apis) does do reconnections/retries on timeouts. The point of
the testclient is deliberately to not do such things.
That said if you are getting "lots of failures on writing data" you
are probably doing something wrong. Getting a working test with say
multiple connections is pretty trivial (and there are thousands of
nodes out there running with multiple connections per day). When you
run say WRFL 10 100000 (this is 10 connections) RDFL 10 100000 is also
10 connections. Perhaps you would like to share what you are running?
Also the VERIFY command is doing reading and writing and verifying
things that have been written are able to be read etc.
Greg
Don’t have standard support just yet. We are just running some tests internally at the moment and once we get support active I’ll start sending bugs your way 
The RDFL and WRFL with 10 or 20 clients works fine. The conflicts and issues happens when one of our application is running some read tests on the event store and I also run some write tests then I get 3 to 5% write failures (2 distinct machines running the tests) and sometimes it stops running altogether with a: Unexpected TCP package: NotHandled
Another example since I still have the logs from running verify:
[PID:04072:009 2016.03.14 14:48:46.062 ERROR Client ] Error during execution of command: Unexpected TCP package: NotHandled…
[PID:04072:009 2016.03.14 14:50:57.368 ERROR Client ] Error during execution of command: Unexpected TCP package: NotHandled…
[PID:06260:009 2016.03.14 14:51:09.004 ERROR Client ] Error during execution of command: Unexpected TCP package: NotHandled…
[PID:07312:009 2016.03.14 14:51:36.517 ERROR Client ] Error during execution of command: Unexpected TCP package: NotHandled…
[PID:07656:009 2016.03.14 14:52:06.026 ERROR Client ] Error during execution of command: Unexpected TCP package: NotHandled…
[PID:03128:009 2016.03.14 15:27:24.514 ERROR Client ] Error during execution of command: Connection was closed prematurely…
[PID:07124:009 2016.03.14 15:30:00.679 ERROR Client ] Error during execution of command: Unexpected TCP package: NotHandled…
[PID:04636:024 2016.03.14 17:58:32.556 FATAL Client ]
"Error during execution of command: Unexpected TCP package: NotHandled.."
Your node hasn't actually started up yet (probably rebuilding an index)
The TestClient does not retry on errors by design.
if you look in node output you probably see messages like : rebuilding
index 100000 (15.4%)
You are also getting timeouts. what hardware / settings / etc are you
running with?
We plan on running in a VM environment, but since we have perf issues in the virtual environment with our application code we also built a physical box. We have the same failures whether it is physical or virtual box.
The attached files are from the following physical server configuration:
System disk is a raid 1, 7.2k RPM disks
Data + Log are on a raid 0, 10k RPM disks (i know log should be split but don’t have the hardware)
1CPU, 4 cores, 1.8 Ghz speed
Memory is 12 GB
So the first section of the log shows tests when only the eventstore.testclient is running. The part where we have slow read and fail writes is when our application is reading and the test client is writing / reading.
Regards
client.log (38.1 KB)
client-err.log (276 Bytes)
"So the first section of the log shows tests when only the
eventstore.testclient is running. The part where we have slow read and
fail writes is when our application is reading and the test client is
writing / reading."
What is your client reading? What workload is it putting out? What is
your db size, are your reads causing your spindles to start seeking
(should be able to hear this). Read flood and write flood are
basically trying to saturate your node as their tests.
Your failure is a socket timeout, the clientapi as example would
reconnect/retry in this case
For testing the performances, the application uses ReadAllEventsForward in slices of 3000 and that is all it does and we get a 9 000/sec read speed which is less than 10 times what we get with the test client.
RDFL is reading a single event. Its also using ReadEvent not
ReadAllEventsForward which is a different message type.
I attached the speed test using the testclient only on one of the VM boxes. I find it pretty good considering the setup behind. It is a VHDX on a CSV on a virtual disk on a storage pool composed of 3 iscsi disk turning at 7.2k over switches that do iSCSI and LAN traffic (dev env hardware)
vm box.log (17.1 KB)
The DB size is between 8GB and 14GB