Read All Events TCP API

EventStoreDB
1

Hi there

We are using the Dotnet Client for our production servers and its working well so thank you very much for the community effort.

We are busy working on generating projections in Python for data analytics. The best client to work from that I found is Photon-pump. This does not have the ability to read all events so I am busy implementing that at the moment.

I am able to call the message ReadAllEvents and then whatever I set at the “max_count” is returned. There are currently three confusing things:

  1. I am just a bit confused by the other required parameters in the ReadAllEvents message. What is the purpose of “commit_position” and “prepare_position”? Currently I just set them to 0 and request 100 events. Now I need to get the next 100 events because I am paging through the events (I assume this is best practice)
  2. How does one know when we have read through all the events? Will ReadAllEventsCompleted return an error or will the I have to look at the “next_commit_position”
  3. Once I have read all the events then how do I subscribe to all events so that any new commits will be added to my projection in python?

Hope someone can provide some insight into the workings of EventStore for me!

I am going to be pushing all my changes back into the community when it works

2

I am just a bit confused by the other required parameters in the
ReadAllEvents message. What is the purpose of "commit_position" and
"prepare_position"? Currently I just set them to 0 and request 100
events. Now I need to get the next 100 events because I am paging
through the events (I assume this is best practice)

If you look in the events being processed there is a prepare/commit
position on them! This is basically saying where to start the read.
This allows you to continue on restart by remembering the last thing
you processed.

How does one know when we have read through all the events? Will
ReadAllEventsCompleted return an error or will the I have to look at
the "next_commit_position"

It will return empty IIRC and there is an IsEndOfStream
https://eventstore.org/docs/dotnet-api/reading-events/index.html gives
an example.

Once I have read all the events then how do I subscribe to all events
so that any new commits will be added to my projection in python?

SubscribeToStream/All (assuming its implemented in the client). You
likely also want to be a bit more fancy here and subscribe *before*
you are caught up (put in queue and check your queue as you move
forward in your read data if that data is in the queue just read off
the queue until complete, note you likely want to limit queue size
here ...). CatchUpSubscription (and the all catch up subscripton)
implement this logic in the C# client code if you want to look through
it.

3

Thanks for the prompt reply! think we missed each other a bit, see my responses under yours

I am just a bit confused by the other required parameters in the

ReadAllEvents message. What is the purpose of “commit_position” and

“prepare_position”? Currently I just set them to 0 and request 100

events. Now I need to get the next 100 events because I am paging

through the events (I assume this is best practice)

If you look in the events being processed there is a prepare/commit

position on them! This is basically saying where to start the read.

This allows you to continue on restart by remembering the last thing

you processed.

Could you please clarify what the prepare position does and what commit position does.

As I said, I set both to 0 because I do not understand them really.

I have now managed to implement the functionallity to read all the events by using the next_commit_position and next_prepare_position

as the commit position and prepare position on the next read. But it would be nice to know what these mean as they are required.

How does one know when we have read through all the events? Will

ReadAllEventsCompleted return an error or will the I have to look at

the “next_commit_position”

It will return empty IIRC and there is an IsEndOfStream

https://eventstore.org/docs/dotnet-api/reading-events/index.html gives

an example.

The IsEndOfStream is not a field in the ReadAllEventsCompleted message. It seems it is only a field in the ReadStreamEventsCompleted.

Therefore, am I to just detect EndOfStream when the number of events returned is 0?

Once I have read all the events then how do I subscribe to all events

so that any new commits will be added to my projection in python?

SubscribeToStream/All (assuming its implemented in the client). You

likely also want to be a bit more fancy here and subscribe before

you are caught up (put in queue and check your queue as you move

forward in your read data if that data is in the queue just read off

the queue until complete, note you likely want to limit queue size

here …). CatchUpSubscription (and the all catch up subscripton)

implement this logic in the C# client code if you want to look through

it.

I see the SubscribeToStream message proto but do not see one for SubscribeToAll or CatchUpSubscription

Are you implying that I should mimic how the Dotnet client implements SubscribeToAll or CatchUpSubscription?

4

inline.

Thanks for the prompt reply! think we missed each other a bit, see my responses under yours

I am just a bit confused by the other required parameters in the
ReadAllEvents message. What is the purpose of "commit_position" and
"prepare_position"? Currently I just set them to 0 and request 100
events. Now I need to get the next 100 events because I am paging
through the events (I assume this is best practice)

If you look in the events being processed there is a prepare/commit
position on them! This is basically saying where to start the read.
This allows you to continue on restart by remembering the last thing
you processed.

Could you please clarify what the prepare position does and what commit position does.
As I said, I set both to 0 because I do not understand them really.
I have now managed to implement the functionallity to read all the events by using the next_commit_position and next_prepare_position
as the commit position and prepare position on the next read. But it would be nice to know what these mean as they are required.

I would have to look in the CatchUpAllSubscription which has the
appropriate logic but it is there.

How does one know when we have read through all the events? Will
ReadAllEventsCompleted return an error or will the I have to look at
the "next_commit_position"

It will return empty IIRC and there is an IsEndOfStream
https://eventstore.org/docs/dotnet-api/reading-events/index.html gives
an example.

The IsEndOfStream is not a field in the ReadAllEventsCompleted message. It seems it is only a field in the ReadStreamEventsCompleted.
Therefore, am I to just detect EndOfStream when the number of events returned is 0?

This seems likely. Again this logic is in class mentioned above.

Once I have read all the events then how do I subscribe to all events
so that any new commits will be added to my projection in python?

SubscribeToStream/All (assuming its implemented in the client). You
likely also want to be a bit more fancy here and subscribe *before*
you are caught up (put in queue and check your queue as you move
forward in your read data if that data is in the queue just read off
the queue until complete, note you likely want to limit queue size
here ...). CatchUpSubscription (and the all catch up subscripton)
implement this logic in the C# client code if you want to look through
it.

I see the SubscribeToStream message proto but do not see one for SubscribeToAll or CatchUpSubscription
Are you implying that I should mimic how the Dotnet client implements SubscribeToAll or CatchUpSubscription?

The subscriptions are message interchanges the CatchUp/CatchUpAll
Subscriptions are the logic to handle the switching between live
subscriptions and paged reads. They should contain *exactly* the logic
needed to implement.