CommandNotExpectedException with NotHandled on subscription catchup

EventStoreDB
#1

Hi,

Got this error that I can’t figure out, maybe you guys can help. I have a 3 instance cluster, with each instance running EventStore and my app, using JVM client. ExtIp for each store is 127.0.0.1 (app is expected to connect to localhost and let EventStore do the clustering part), and ports are all defaults.

When the app starts up and does a catchup from beginning the subscription immediately terminates with:

ERROR eventstore.SubscriptionActor - eventstore.CommandNotExpectedException: Expected: ReadAllEventsCompleted, actual: NotHandled(NotMaster(MasterInfo(/127.0.0.1:1113,/127.0.0.1:2113,None)))

Code to start the subscription:

ActorSystem system = ...;
final ActorRef connection = **...**;
final ActorRef observerActor = system.actorOf( SubscriptionObserverActor.props(observer,
        scala.reflect.ClassTag$.MODULE$.apply(IndexedEvent.class)));
Option<Position> currentOptionalPosition = Option.apply( null );
final ActorRef subscription = system.actorOf( SubscriptionActor.props(connection,
        observerActor, currentOptionalPosition, false, Option.empty(), 10));
this.subscription = ActorCloseable.apply(observerActor);




Looking into the EventStore log the cluster has formed, one master and two slaves, no errors.



Any idea what might be wrong here? Is there any issue with using ExtIp as 127.0.0.1?



cheers, Rickard
#2

Some update:
If I temporarily change the config on one server to be cluster size: 1
and restart EventStore and my app, then the subscription works. If I
then switch back to cluster size 3, it works too, but now
currentOptionalPosition is not null anymore, if that makes the
difference. No idea why.

/Rickard

#3

If I wipe the eventstore data, and the reader database (so no stream
position on startup), and then start up the cluster, one app is fine
and two are not. I'm guessing those two are slaves, and the NotMaster
event they get (which fails the subscriber) is related to that?

Can I change my "subscribe to all events" to not include "system
events" somehow? Or what is the recommended way of handling this?

/Rickard

#4

Found the issue. Turns out to be yet another problem caused by relying
on settings being in application.conf, rather than using the supplied
settings. SubscriptionActor calls ReadAllEvents, but doesn't include
the requireMaster setting, which therefore is picked up from defaults,
which reads application.conf, which has require-master set to true. If
I add require-master=false in that file it works on all instances.

I would strongly suggest that you go through the code and remove all
reliance on Settings.Default,
or make sure that provided configuration is used where appropriate,
or remove the possibility to provide custom configuration.

Any of the above would be better than the current state.

thanks, Rickard

#5

Normally the "NotHandled" is due to the subscription starting before
the system is actually up. The system responding "NotHandled" normally
happens during a very short period of time while authentication stuff
is loading. Are you seeing this behaviour after nodes have been
running for more than a few seconds or is it on start up?

#6

Fixed in master, wil be included in next release. Thanks for reporting.