atomically writting to two streams

EventStoreDB
#1

Is it possible to write to two different streams atomically?

The reason why I’m asking is that we are looking to use GES as queue with different subscribers doing work based on the appearance of events in a source stream. Each subscriber will need to keep their own position (in a separate stream) and write down the result of their processing to another stream (output). The write to the position stream and the output stream needs to be done atomically. Can this be done?

Thanks,

Karl

#2

It can be though there are many other options available to you such as using idempotency to get around the want of it.

If I read a piece of work from stream 1

I then write output to stream 2

I then write checkpoint to stream 3

If you use deterministic event ids and expected version number on write to 2 you will have full idempotency so in a failure will just retry on startup without duplication. This is what projections does internally.

Greg

#3

What are some of the other options other than deterministic event generation?

#4

We do not allow cross stream transactions as it will not work when you shard.

#5

okay, I’ll ask the dumb question.

Why doesn’t the server handle position/checkpoint then if requested?

#6

Which server? GES?

You want to save the checkpoint in a transaction with your processing results.

#7

"okay, I’ll ask the dumb question.

Why doesn’t the server handle position/checkpoint then if requested?"

what do you mean by position checkpoint if requested?

#8

GES.

Using the example Greg outlined above:

If I read a piece of work from stream 1

I then write output to stream 2

I then write checkpoint to stream 3

It would seem that the only answer that works is to store my position in stream 1 in my output written to stream 2.

So given that i am effectively store my position/checkpoint on the server, it would seem natural (to me at least) that the server could take care of this for me. That way I wouldn’t necessarily pollute my output stream with this control information and it would save me (and other developers) from worrying explicitly about this. Obviously this would be only if I asked for GES to do this as there are many cases where keeping position is not necessary.

#9

See branch competing it will be shortly released.

#10

btw this can be trivially done without using competing so long as you have deterministic ids (eg guid based off previous guid) then idempotency will handle everything for you and you will not need transactions or server side state. This is how projections works btw. Here is an example of deterministic guids https://gist.github.com/jen20/33d28a6ed7415f1aaa58

Cheers,

Greg

#11

Also to add competing supports having many subscribers load balanced on a stream subscription (eg competing consumers)