Imagine we have competing consumers setup and the consumers end up acking messages out of order, what is the expected behaviour when clients reconnect after event store shutdown?
Observed behaviour:
Subscriber 1 receives Message 96 and is busy doing long work
Message 97,98,99 - Processed in Subscriber 2 and ack
Shutdown eventstore - When eventstore comes up 97,98,99 is resent . 96 is lost.
Shutdown subscriber 1 and subscriber 2. Followed by evenstore -when eventstore comes up 96 is lost but 97,98,99 is not resent.
We are trying to come up with a pattern to handle this situation and are unsure the best way to proceed and would like to know what the expected behaviour is from ES
You should get loads of retries. Checkpoints will not be written into
the future until all messages up to that point have been processed.
EG:
1,2,3,4,5
if something is still processing 1 and 2,3,4,5 have been processed a
checkpoint will not be written past 1 until 1 is processed. Of course
if your long running thing had timed out multiple times and parked the
message the checkpoint could be written (no longer in process). This
depends on configuration settings.
Checkpoint writes are logged, checkpoints are loaded on restart again logged.
Greg, in our case (as described in the example above) we are losing message 96 altogether. Is there anyway we can make sure message 96 is also resent? I do not see the message in the parked queue as well.