Hi,
On the competing consumers, What do these numbers really mean and does red mean something is not processed? and why does it stay as so? I have number of subscriptions that have show up like this and I think it means those messages have not been processed (acked) by a subscriber.
Thanks
Q
This means your processing of messages is behind the total events available to process for given subscription group AND you are not processing any events. So yes, essentially your subscribers are not ack’king received messages, or not receiving them at all.
Numbers next to red dot mean “number of message behind” / “number of seconds behind” Generally you want these numbers to be close to 0.
Thanks. What do you mean by “or not receiving them at all”? In what case would they not receive them?
Not sure why they are not being processed though, my subscribers are sitting idle.
interestingly, I deleted one of my competing consumer subscription (and restarted by subscriber) from the UI and it caught up all the way.
Under what circumstances would messages be stuck and not be sent to the subscribers?
None that I know of. More likely a client isn't acking. What is does
your subscriber look like? What does your subscription configuration
look like?
Thanks Greg.
If a client did not ack, should the message not be resent again. When I debug my subscriber, that event is not raised.
The subscription is created as such
await connection.ConnectToPersistentSubscriptionAsync(StreamCategoryName,
StreamName,
OnDomainEventAppeared,
OnSubscriptionDropped,
null,
10,
false)
.ConfigureAwait(false);
and on OnDomainEventAppeared is as such _domainEventAppeared is ReactiveExtensions. We wont ack if it goes into one of the catch blocks.
private void OnDomainEventAppeared(EventStorePersistentSubscriptionBase subscriptionBase, ResolvedEvent resolvedEvent)
{
Logger.Trace($"{_className}.OnDomainEventAppeared called.");
try
{
var domainEvent = MapStoredEventToDomainEventAsync(resolvedEvent);
var eventAppeared = new DomainEventAppeared(domainEvent, resolvedEvent.OriginalEventNumber,
resolvedEvent.Event.EventStreamId);
_domainEventAppeared?.OnNext(eventAppeared);
subscriptionBase.Acknowledge(resolvedEvent);
}What is the configuration of the subscription? Retries etc are setup
in your configuration.
Also you have autoack set to false do you ack in your handler?
Yes we are Acking in the handler.
The settings are like this (in some cases we will override this to startfromcurrent and that is all we are doing).
protected virtual PersistentSubscriptionSettings CreateSubscriptionSettings()
{
Logger.Trace($"{_className}.CreateSubscriptionSettings called.");
PersistentSubscriptionSettings settings = PersistentSubscriptionSettings.Create()
.ResolveLinkTos()
.StartFromBeginning()
.WithMaxRetriesOf(25);
return settings;
}We are running a 3 node cluster (maybe that is useful info).
One thing we experienced, the persistent subscriptions do not work on a cluster on 3.8 so we upgraded that to 3.9.
Is the node you are checking on the master?
Actually, I have the same behaviour on my local environment which is not running any nodes.
On the actual dev/qa environments we have an elb infront of it and we connect the ui to the elb url.
I am seeing very inconsistent behavior with this. I deleted all the subscribers to a stream. It was showing 207 known messages and 106 red. Every time I delete all my subscribers and restart them, it ends up with a different number of messages showing as not processed. I have no exceptions in the processing and am acking after the handler is done. It shows 95 messages right now. Yet it is not sending anything to my subscribers. Even if I restart nothing gets sent.