that
those nodes then returned Service Unavailable. I had just succeeded to get the .$all to work with node 1114, so I had to try
C:\Windows\system32>netsh http add urlacl url=http://+:1114/ user=everyone
URL reservation successfully added
on it too. Of course that node immediately turned into Service Unavailable also…
So…:
netsh http show urlacl
and netsh http delete url=http://+:[port]
on all of them, and problem was solved.
I had a bunch that was added and didn’t need to remove all to get it to work again, so you might have to put some back in again, if you get this problem.
I’ve seen on several occasions (not with Event Store, just HTTP on windows generally) these get corrupted and need removing and re-adding otherwise everything returns 503. No idea why.
Today, when running the same cluster, and kicking up the catch-up subscriber, all of the events from yesterday were gone. What’s up with that? Okay, mostly they were stats collected events, but quite some of other events too, that I had tossed in.
How can I check what’s actually in the db in a case like this?
Ah, sorry, yes a redirect link to 1114. So, I tried same for that one but got a timeout.
Then I noticed all urlacls that I had deleted were back. Does ES add them upon start?
Anyway, keeps being weird. Got 401, removed the urlacl, added it again, still 401… I guess I’ll leave it at that.
But not getting all the events in the catchup subscriber, that must be something else, no? Have subscribetoall from Position.Start. There were plenty of events from yesterday, just didn’t show up, but I see them in the streambrowser (after defeating IIS).
I guess you’re trying to say that I suck on this. I would hardly argue against it.
But you mean that the info on how to avoid these problems is actually available somewhere in this forum or in the documentation?
I can reproduce almost everything you are talking about through
misunderstanding ... As an example since its the most easy one, I can
reproduce your exact $all subscription error with the jvm client very
easily. Record a checkpoint. Delete my database (or start up a new
temp database) and then start my subscription again passing the first
checkpoint. Subscriptions don't work from one instance of a db to
another (they assume to be running against the same database as
switching makes no sense and will give the exactly what you are
seeing). Maybe you are doing this inadvertently with your config/what
you have done previously.
I can reproduce most of your other issues really easily by restarting
with new dbs and interacting through a browser as I will get some very
interesting cached behaviour (as mentioned this is why a command line
option exists to disable caching for a dev environment). You can
normally figure this out by using something that bypasses cache if
needed.
Beyond all that I would really question my measurements since you seem
to be unable to come up with a reproduction of any of them? If you
have a reproduction I would be happy to run it here.
I would even love as an example to see the db where an ACL existed
then did not exist. By default its an append only model so we should
be able to see quite easily what happened my guess though is this is
http etc caching or running on different databases without realizing
etc. I am happy to look through that db though.
As I said you have a database that is showing an error and is small
zip it and send it to me.
[thumbs up]
If you had a Complete Guide To EventStore of, say 600-1500 pages, at maybe around $200 I’d buy it at once. It sucks that it takes so much time to write books.
Okay. But this is actually not that. Yesterday I did reinstall the db that’s true. But the subscriber was not on at that time. I ran cluster during the day after reinstalling, an instance of a console app in debug mode subscribing to all events running along with it showing me all the events it picked up. And sometimes I’d spin up an instance of a couple of other projects that would send some events to the db. One I have on a tight loop as to see how many events I could push into the cluster per second.
Alright.
Hibernate computer, hit the pillow. Comes next day, spin everything up again. (I might have stopped the debugger night before). But the events from yesterday don’t show up in the console window.
I really don’t want to waste your time having to look through things, I know this is me not getting it right. Still have to try and toss the question though, maybe it’s just a simple “hey fool, this is what you did wrong”.
If I got an appliance from you, would everything be good then? I will seriously consider taking it from my own salary, because I really must work with ES (as it rocks).
Start by running a single node not a cluster (you don't need a cluster
for development?! and they are more complex to setup and easier to
misunderstand/misconfigure). Run dev branch with development mode
option (or if you decide to make a new database put it on a new port
for tcp/http this gets rid of issues with caching as well). This alone
will probably save a huge portion of your issues.
Beyond that feel free to send an reproductions. Noone else is seeing
behaviour such as you are and we would be happy to look through any
reproduction. For "disappearing events" its an append only model. We
can certainly go look what was written.
Just run EventStore.ClusterNode.exe ... it defaults this way. TCP will
be port 1113 http will be 2113 though since you have used 2113 I might
change the http port. --help will tell you the command line arguments.
You can ignore most of them and just use --ext-http-port say to 4324
since you haven't used it before.
There is absolutely no reason ever to run a cluster on a dev machine
unless you are say a systems person testing the clustering? Much of
your behaviour could also be caused by incorrectly configured clusters
where though you may have thought it was a cluster was perhaps more
than one cluster and switching between nodes etc. Clusters are also
exponentially more difficult to debug (which node were you talking
to?)
Thanks for the tips.
I am testing the clustering also, want to deploy it to azure vms. We’ll see how that goes in the end -.o
Alright, thanks again. I’ll try with what you propose, and certainly if I manage to reproduce weird behavior I’ll send it over.
You mentioned above that you are performance testing. Its not a good
idea to performance test with 3 nodes on one machine.
1) You lose the overhead of communications since they are on localhost
2) They all compete for the same resources (especially disk!)
I was given some very good advice coming up on two decades ago: If you
want to performance test run in an environment as close to your
predicted environment as possible.
True.
I have been testing it locally while trying to get the azure VM:s up and configured.
Tried to use the powershell script written about here https://groups.google.com/forum/?hl=sv#!topic/event-store/SBgFKYLdGaw by Phil Bolduc to set it all up. Something is not working properly with the disks for me though (wrote about it in the thread, also posted on stackoverflow about it, but no clues what’s wrong). I wish he would take a look in that thread and point out the error for me!
I’ll resort to setting it up manually, so eventually I’ll test the clustering in the right environment.
