Competing Consumers Parked Messages & .NET API

EventStoreDB
#1

Reading the docs, http://docs.geteventstore.com/introduction/competing-consumers/ explains how messages can be parked (a.k.a. deadlettered) by a subscription client. However, there is no information about how this works on the .NET API competing consumers doc http://docs.geteventstore.com/dotnet-api/3.3.1/competing-consumers/.

I did find something about it on the HTTP API docs http://docs.geteventstore.com/http-api/3.4.0/competing-consumers/#replaying-parked-messages.

I also can’t seem to find anything in the .NET API on how to park messages, or how to replay parked messages. Does this mean we have to use the HTTP API to do these things? If not, how do you use the .NET API to:

  • signal the database that a message should be parked, and

  • replay parked messages for a given subscription (stream + group)?

#2

Replay is only via http (its an http post).

Parking is just the status given on NotAcknowledge
https://github.com/EventStore/EventStore/blob/release-v3.5.0/src/EventStore.ClientAPI/PersistentSubscriptionNakEventAction.cs

#3

I see now, second arg on the EventStorePersistentSubscriptionBase.Fail overloads. I shouldn’t have missed this, thanks.

#4

Note unknown is the default failure mode and the server will decide
how to handle things. Depending on config for retries etc it will get
parked based on config

#5

Hi Greg,
Is it possible to replay a particular parked event or the recent events only, since we have many parked events?

#6

hmm I believe you can only replay all ... That said ... These items
are kept in a stream. I believe you could truncate that stream (eg
$tb) or even write things to it (not highly recommend! :-D). Warning I
have not tested this but believe it should work! Replay itself is only
reading items from this stream (its in the details of the
subscription) and pushing them out...

I have looked in the past at a feature around this and will revisit
it. Basically the idea was to be able to do more "custom" type things
on replay

#7

BTW can you or others here give me an idea what you would *like* to be
able to do with parked messages?

How to better see them?
How to better replay them?
How to better notify of things?
What else you may want?

I think the How questions there are fairly reasonable now (the stream
is pretty easy to integrate) though something nicer can be done.

#8

Hi Greg,
Thanks for your reply.
My suggestions are below:

BTW can you or others here give me an idea what you would like to be

able to do with parked messages?

How to better see them?

- There’s a link (UI) from subscription to the parked stream. The names in my case are not very short and error-prone when typing.

- For each parked event, we’d like to see how many times it has been replayed, and the timestamp of the last one.

How to better replay them?

*- It’s possible to replay selected events and/or events created in a period of time. *

#9

I recently did some tooling of my own to handle persistent subscriptions in production. The major points:

UI with number of parked message per subscription (can also follow link to see events)

Replay all subscriptions, all events

Healthcheck that looks at status, connections, parked messages, etc

/Peter

#10

Send over a screenshot?

#11

We’d be very interested in a set of features to better manage DLQs.

This is very timely actually as we had issues related to this that took down production (replaying 120k DLQs seemed to take down the Persistent Subscription part of eventstore until we re-elect master - seemed to occur a few times during our recovery so we’ll be searching for an isolated reproduction soon).

Some things we’d be keen to see:
* Transactional DLQ replay - even if everything stays as is, you still “lose” messages if your master node goes down whilst DLQs are replaying.
* Selective DLQ replay - eg persistent sub on a $ce- stream, we’d like to replay a certain underlying stream’s DLQs eg just Foo-123 DLQs from parked messaged on some PS on $ce-Foo.
* With the above point, to be able to see the DLQ counts on a single persistent subscription split by underlying stream.
* Batched DLQ replay, as mentioned, replaying huge numbers of DLQs becomes impossible (repro pending) so can we replay them in batches (eg replay oldest 1000, then next 1000 etc) - should be relatively easy given the current $tb mechanism of replaying.

Cheers, hope this adds some ideas to the pot.

James

#12

Screenshot 2019-08-29 at 09.35.12.png

Not much to see really. (Realised I haven’t merged the folow-link-to-events-code yet.)

The healthchecks are just App.Metrics.

/Peter