Stuck PersistentSubscriptions

EventStoreDB V5.X
1

We are starting to have issues with persistent subscription on one eventstore instance. Was 4.1.1, updated to 5.0.8 to see if it resolved the issue, but same behaviour. Certain subscriptions stop processing. I see connected client, but status is Behind, OutstandingPageRequest.

Only way I found to get them going again is to delete and recreate. The latest case are very recently created subscriptions groups.

Found this in logs:
[PID:04540:010 2020.05.20 00:31:52.375 ERROR QueuedHandlerThreadP] Error while processing message EventStore.Core.Messages.MonitoringMessage+GetStreamPersistentSubscriptionStats in queued handler ‘PersistentSubscriptions’.
EXCEPTION OCCURRED
System.NullReferenceException: Object reference not set to an instance of an object.
at EventStore.Core.Services.PersistentSubscription.PersistentSubscriptionStats.GetStatistics() in D:\a\1\s\oss\src\EventStore.Core\Services\PersistentSubscription\PersistentSubscriptionStats.cs:line 73
at EventStore.Core.Services.PersistentSubscription.PersistentSubscriptionService.<>c.b__48_0(PersistentSubscription sub) in D:\a\1\s\oss\src\EventStore.Core\Services\PersistentSubscription\PersistentSubscriptionService.cs:line 737
at System.Linq.Enumerable.WhereSelectListIterator2.MoveNext() at System.Collections.Generic.List1…ctor(IEnumerable1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source)
at EventStore.Core.Services.PersistentSubscription.PersistentSubscriptionService.Handle(GetStreamPersistentSubscriptionStats message) in D:\a\1\s\oss\src\EventStore.Core\Services\PersistentSubscription\PersistentSubscriptionService.cs:line 737
at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message) in D:\a\1\s\oss\src\EventStore.Core\Bus\MessageHandler.cs:line 26
at EventStore.Core.Bus.InMemoryBus.Publish(Message message) in D:\a\1\s\oss\src\EventStore.Core\Bus\InMemoryBus.cs:line 290
at EventStore.Core.Bus.QueuedHandlerThreadPool.ReadFromQueue(Object o) in D:\a\1\s\oss\src\EventStore.Core\Bus\QueuedHandlerThreadPool.cs:line 134

2

Seems like an internal issue, mind opening an issue on GitHub?

3

We have a support ticket, I think it’s being escalated. IO issue during startup and checksumming chunks, a restart fixed it.

(my main concern with this kind of issue is visibility. I would expect a consumer connecting to an essentially dead subscription to get an error, but everything looked normal)

4