Reactive stream API with catchup semantics?

Clients
1

We were having problems with the regular catchup subscriptions throwing “consumer too slow” when we are doing rebuilds, and so wanted to switch to the reactive API (readStreamReactive), which on paper would solve it. Unfortunately it seems at the end of the stream it calls onComplete() on the subscriber instead of waiting for more events.

Is there a way to use the reactive streams API with catchup subscription semantics? If not, why? If yes, how?

thanks, Rickard

2

What Client is it ?
( I’m guessing Java)
ReadStream reads a stream until the end then closes . that’s the semantics of that operation.
typical usage: readStream, fold over the events, make some decision , append new events.
Catch-up semantics is , “give me everything from , … and keep sending any new events until the sucbscription is ended”
Can you describe what you do in those catch-up subscriptions ?

3

Building the read model for our application, specifically projecting events into Neo4j. During rebuilds some batches of events may take longer than others, and so the regular catchup subscription throws an exception. We were hoping that the reactive stream with backpressure would avoid that, since that is the typical semantics for reactive streams.

It is not defined in reactive streams itself that it has to close if there are no more events. That’s purely a design and implementation choice. I would consider that a fairly logical read option to provide on connect (close or wait for more events) actually.

If it was considered when adding reactive streams support to the API, then why was it decided not to support catch-up subscription semantics, i.e. simply wait for more events?

4

Hey @Rickard_Oberg, could you share what you did to make catch-up subscription reactive?

5

I think it is solvable by a separate product we are discussing now, which supposes to take away the pain of subscriptions that need to be always ready to receive things.

6

I was using the reactive stream API, hoping that it would have catch-up semantics (wait at end instead of close), but unfortunately that didn’t work, and there was no way to change it, hence this discussion.

7

Interesting. Is it more of a pull API like the Atom feed?

8

I’m asking you because we don’t expose a reactive streams api for catchup subscriptions. I was wondering what approach you did to see if I can help.