EventStore in a worker role, Azure, blob

esr · March 18, 2015, 5:09pm

So, I deleted it all, and reinstall the whole cluster. Tried several times,

but I keep getting these error for all of the instances, but only one is dies, the other two are running according to Task manager, but I can’t access the web interface or any other data from them.

(some errors are left out here)

[PID:02804:015 2015.03.18 00:01:20.101 ERROR StorageReaderWorker ] Error during processing ReadStreamEventsBackward request.
System.NullReferenceException: Object reference not set to an instance of an object.
at EventStore.Core.Services.Storage.ReaderIndex.IndexReader.CheckStreamAccessInternal(TFReaderLease reader, String streamId, StreamAccessType streamAccessType, IPrincipal user) in c:\EventStore\src\EventStore.Core\Services\Storage\ReaderIndex\IndexReader.cs:line 311
at EventStore.Core.Services.Storage.ReaderIndex.IndexReader.EventStore.Core.Services.Storage.ReaderIndex.IIndexReader.CheckStreamAccess(String streamId, StreamAccessType streamAccessType, IPrincipal user) in c:\EventStore\src\EventStore.Core\Services\Storage\ReaderIndex\IndexReader.cs:line 275
at EventStore.Core.Services.Storage.ReaderIndex.ReadIndex.EventStore.Core.Services.Storage.ReaderIndex.IReadIndex.CheckStreamAccess(String streamId, StreamAccessType streamAccessType, IPrincipal user) in c:\EventStore\src\EventStore.Core\Services\Storage\ReaderIndex\ReadIndex.cs:line 93
at EventStore.Core.Services.Storage.StorageReaderWorker.ReadStreamEventsBackward(ReadStreamEventsBackward msg) in c:\EventStore\src\EventStore.Core\Services\Storage\StorageReaderWorker.cs:line 188
[PID:02804:014 2015.03.18 00:01:20.116 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Core.Messages.ClientMessage+ReadStreamEventsBackwardCompleted in queued handler ‘Projection Core #0’.
System.Exception: Cannot start control reader. Read result: Error
at EventStore.Projections.Core.Services.Processing.ProjectionCoreServiceCommandReader.d__c.MoveNext() in c:\EventStore\src\EventStore.Projections.Core\Services\Processing\ProjectionCoreServiceCommandReader.cs:line 70
at EventStore.Core.Helpers.IODispatcherAsync.Run(IEnumerator1 actions) in c:\EventStore\src\EventStore.Core\Helpers\IODispatcherAsync.cs:line 338 at EventStore.Core.Helpers.IODispatcherAsync.<>c__DisplayClass8.<>c__DisplayClassa.<BeginReadBackward>b__7(ReadStreamEventsBackwardCompleted response) in c:\EventStore\src\EventStore.Core\Helpers\IODispatcherAsync.cs:line 98 at EventStore.Core.Messaging.RequestResponseDispatcher2.Handle(TResponse message) in c:\EventStore\src\EventStore.Core\Messaging\RequestResponseDispatcher.cs:line 64
at EventStore.Core.Messaging.RequestResponseDispatcher2.EventStore.Core.Bus.IHandle<TResponse>.Handle(TResponse message) in c:\EventStore\src\EventStore.Core\Messaging\RequestResponseDispatcher.cs:line 53 at EventStore.Core.Bus.MessageHandler1.TryHandle(Message message) in c:\EventStore\src\EventStore.Core\Bus\MessageHandler.cs:line 33
at EventStore.Core.Bus.InMemoryBus.Publish(Message message) in c:\EventStore\src\EventStore.Core\Bus\InMemoryBus.cs:line 324
at EventStore.Core.Bus.InMemoryBus.Handle(Message message) in c:\EventStore\src\EventStore.Core\Bus\InMemoryBus.cs:line 308
at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o) in c:\EventStore\src\EventStore.Core\Bus\QueuedHandlerMRES.cs:line 121

and this

[PID:03488:005 2015.03.18 12:10:27.311 ERROR StorageReaderWorker ] Error during processing ReadStreamEventsBackward request.
System.NullReferenceException: Object reference not set to an instance of an object.
at EventStore.Core.Services.Storage.ReaderIndex.IndexReader.CheckStreamAccessInternal(TFReaderLease reader, String streamId, StreamAccessType streamAccessType, IPrincipal user) in c:\EventStore\src\EventStore.Core\Services\Storage\ReaderIndex\IndexReader.cs:line 311
at EventStore.Core.Services.Storage.ReaderIndex.IndexReader.EventStore.Core.Services.Storage.ReaderIndex.IIndexReader.CheckStreamAccess(String streamId, StreamAccessType streamAccessType, IPrincipal user) in c:\EventStore\src\EventStore.Core\Services\Storage\ReaderIndex\IndexReader.cs:line 275
at EventStore.Core.Services.Storage.ReaderIndex.ReadIndex.EventStore.Core.Services.Storage.ReaderIndex.IReadIndex.CheckStreamAccess(String streamId, StreamAccessType streamAccessType, IPrincipal user) in c:\EventStore\src\EventStore.Core\Services\Storage\ReaderIndex\ReadIndex.cs:line 93
at EventStore.Core.Services.Storage.StorageReaderWorker.ReadStreamEventsBackward(ReadStreamEventsBackward msg) in c:\EventStore\src\EventStore.Core\Services\Storage\StorageReaderWorker.cs:line 188
[PID:03488:016 2015.03.18 12:10:28.726 ERROR QueuedHandlerThreadP] Error while processing message EventStore.Core.Messages.ClientMessage+ReadStreamEventsBackwardCompleted in queued handler ‘Worker #5’.
System.Exception: Failed to initialize password change notification reader. Cannot read $users-password-notifications Error: Error
at EventStore.Core.Services.Transport.Http.Authentication.PasswordChangeNotificationReader.b__0(ReadStreamEventsBackwardCompleted completed) in c:\EventStore\src\EventStore.Core\Services\Transport\Http\Authentication\PasswordChangeNotificationReader.cs:line 44
at EventStore.Core.Messaging.RequestResponseDispatcher2.Handle(TResponse message) in c:\EventStore\src\EventStore.Core\Messaging\RequestResponseDispatcher.cs:line 64 at EventStore.Core.Messaging.RequestResponseDispatcher2.EventStore.Core.Bus.IHandle.Handle(TResponse message) in c:\EventStore\src\EventStore.Core\Messaging\RequestResponseDispatcher.cs:line 53
at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message) in c:\EventStore\src\EventStore.Core\Bus\MessageHandler.cs:line 33
at EventStore.Core.Bus.InMemoryBus.Publish(Message message) in c:\EventStore\src\EventStore.Core\Bus\InMemoryBus.cs:line 324
at EventStore.Core.Bus.InMemoryBus.Handle(Message message) in c:\EventStore\src\EventStore.Core\Bus\InMemoryBus.cs:line 308
at EventStore.Core.Bus.QueuedHandlerThreadPool.ReadFromQueue(Object o) in c:\EventStore\src\EventStore.Core\Bus\QueuedHandlerThreadPool.cs:line 106

what’s up with this?

Greg_Young1 · March 18, 2015, 5:13pm

Have never seen this issue can you send full logs?

Greg_Young1 · March 18, 2015, 5:17pm

Also can you describe your setup there are some weird errors here like

System.Exception: Cannot start control reader.

Sounds like there is something else failing. You mention azure blob. Whats the command line you are using or are you hosting directly if directly what is your setup code

esr · March 18, 2015, 5:30pm

I’ll send it by email

Oh, and I do get to the login page now, but I get an “Server does not exist or wrong user data” when trying to log in (haven’t changed from default credentials).

The setup is explained a couple of posts up in this thread. It runs in the OnStart(). No error is thrown from that code.

Greg_Young1 · March 18, 2015, 5:33pm

I dont think you are actually working my guess is something like a uri filename etc which isnt working. Im on a phone now so cannot easily find your code based setup

Greg_Young1 · March 18, 2015, 5:35pm

See osx/linux http://docs.geteventstore.com/server/3.0.1/

esr · March 18, 2015, 6:26pm

Thank you Greg, you have all you need in your inbox.

Uri filename seems odd to me since I haven’t changed any of that since it worked last. Not saying that it’s impossible, but doesn’t seem likely to me.

This is what the one node that died spit out before giving up:

[PID:04012:006 2015.03.18 12:59:47.751 ERROR StorageReaderWorker ] Error during processing ReadStreamEventsBackward request.
System.NullReferenceException: Object reference not set to an instance of an object.
at EventStore.Core.Services.Storage.ReaderIndex.IndexReader.CheckStreamAccessInternal(TFReaderLease reader, String streamId, StreamAccessType streamAccessType, IPrincipal user) in c:\EventStore\src\EventStore.Core\Services\Storage\ReaderIndex\IndexReader.cs:line 311
at EventStore.Core.Services.Storage.ReaderIndex.IndexReader.EventStore.Core.Services.Storage.ReaderIndex.IIndexReader.CheckStreamAccess(String streamId, StreamAccessType streamAccessType, IPrincipal user) in c:\EventStore\src\EventStore.Core\Services\Storage\ReaderIndex\IndexReader.cs:line 275
at EventStore.Core.Services.Storage.ReaderIndex.ReadIndex.EventStore.Core.Services.Storage.ReaderIndex.IReadIndex.CheckStreamAccess(String streamId, StreamAccessType streamAccessType, IPrincipal user) in c:\EventStore\src\EventStore.Core\Services\Storage\ReaderIndex\ReadIndex.cs:line 93
at EventStore.Core.Services.Storage.StorageReaderWorker.ReadStreamEventsBackward(ReadStreamEventsBackward msg) in c:\EventStore\src\EventStore.Core\Services\Storage\StorageReaderWorker.cs:line 188
[PID:04012:006 2015.03.18 12:59:47.876 ERROR QueuedHandlerThreadP] Error while processing message EventStore.Core.Messages.ClientMessage+ReadStreamEventsBackwardCompleted in queued handler ‘Worker #1’.
System.Exception: Failed to initialize password change notification reader. Cannot read $users-password-notifications Error: Error
at EventStore.Core.Services.Transport.Http.Authentication.PasswordChangeNotificationReader.b__0(ReadStreamEventsBackwardCompleted completed) in c:\EventStore\src\EventStore.Core\Services\Transport\Http\Authentication\PasswordChangeNotificationReader.cs:line 44
at EventStore.Core.Messaging.RequestResponseDispatcher2.Handle(TResponse message) in c:\EventStore\src\EventStore.Core\Messaging\RequestResponseDispatcher.cs:line 64 at EventStore.Core.Messaging.RequestResponseDispatcher2.EventStore.Core.Bus.IHandle.Handle(TResponse message) in c:\EventStore\src\EventStore.Core\Messaging\RequestResponseDispatcher.cs:line 53
at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message) in c:\EventStore\src\EventStore.Core\Bus\MessageHandler.cs:line 33
at EventStore.Core.Bus.InMemoryBus.Publish(Message message) in c:\EventStore\src\EventStore.Core\Bus\InMemoryBus.cs:line 324
at EventStore.Core.Bus.InMemoryBus.Handle(Message message) in c:\EventStore\src\EventStore.Core\Bus\InMemoryBus.cs:line 308
at EventStore.Core.Bus.QueuedHandlerThreadPool.ReadFromQueue(Object o) in c:\EventStore\src\EventStore.Core\Bus\QueuedHandlerThreadPool.cs:line 106
[PID:04012:007 2015.03.18 13:00:12.809 ERROR ClusterStorageWriter] Master [{ipaddress},{b1d9a2db-8817-4417-ac00-9d36f94b6d38}] subscribed us at 4495 (0x118F), which is less than our last epoch and LastCommitPosition 5508 (0x1584) >= lastEpoch.EpochPosition 5508 (0x1584)!!! That might be bad, especially if the LastCommitPosition is way beyond EpochPosition.
ATTEMPT TO TRUNCATE EPOCH WITH COMMITTED RECORDS. THIS MAY BE BAD, BUT IT IS OK IF JUST-ELECTED MASTER FAILS IMMEDIATELY AFTER ITS ELECTION.

Haven’t done anything with it, didn’t even log in to the web ui, just deployed and had it sitting there.

Greg_Young1 · March 18, 2015, 6:35pm

The issue is your first error.

I responded to your email to verify what you are actually running as
the exception is an impossible to happen exception in dev and in 3.0.3
so I want to make sure I am looking at what you are running.

[PID:03452:006 2015.03.18 12:57:52.402 ERROR StorageReaderWorker ]
Error during processing ReadStreamEventsBackward request.
System.NullReferenceException: Object reference not set to an instance
of an object.
  at EventStore.Core.Services.Storage.ReaderIndex.IndexReader.CheckStreamAccessInternal(TFReaderLease
reader, String streamId, StreamAccessType streamAccessType, IPrincipal
user) in c:\EventStore\src\EventStore.Core\Services\Storage\ReaderIndex\IndexReader.cs:line
311
  at EventStore.Core.Services.Storage.ReaderIndex.IndexReader.EventStore.Core.Services.Storage.ReaderIndex.IIndexReader.CheckStreamAccess(String
streamId, StreamAccessType streamAccessType, IPrincipal user) in
c:\EventStore\src\EventStore.Core\Services\Storage\ReaderIndex\IndexReader.cs:line
275
  at EventStore.Core.Services.Storage.ReaderIndex.ReadIndex.EventStore.Core.Services.Storage.ReaderIndex.IReadIndex.CheckStreamAccess(String
streamId, StreamAccessType streamAccessType, IPrincipal user) in
c:\EventStore\src\EventStore.Core\Services\Storage\ReaderIndex\ReadIndex.cs:line
93
  at EventStore.Core.Services.Storage.StorageReaderWorker.ReadStreamEventsBackward(ReadStreamEventsBackward
msg) in c:\EventStore\src\EventStore.Core\Services\Storage\StorageReaderWorker.cs:line
188

https://github.com/EventStore/EventStore/blob/dev/src/EventStore.Core/Services/Storage/ReaderIndex/IndexReader.cs#L311

note:

var sysSettings = _backend.GetSystemSettings() ?? SystemSettings.Default;
sysAcl = sysSettings.SystemStreamAcl ?? defAcl;

not sure how sysSettings could be null unless SystemSettings.Default
is null (which it can't be).

https://github.com/EventStore/EventStore/blob/f0e38e85aa473b7b4c57f1e413661da86e828e31/src/EventStore.Core/Data/SystemSettings.cs#L12

tracking through any code that could be run there (even assuming
inlining or calling into the cctor doesn't seem possible in this code
to get this exception.

Can you verify what version you are running? I must not be looking at
equivalent code.

Cheers,

Greg

Mike_D · April 18, 2016, 4:25pm

I’m interested in what you’re doing here, could you explain why you are doing this please?