Newbie Question: Pub/Sub

Okay,

I’m trying to figure out a good way to handle pub/sub logic when using GES.

In Johnathan Oliver’s Event Store – now NEventStore (NES), I think I understand the general principles (but correct me if I’m wrong):

  1. Natural concurrency checks do not require DTC transactions if only one aggregate is modified per unit of work (equivalent to writes to one stream per unit-of-work in GES)

  2. The order of events written for a specific Aggregate will be preserved (in GES this is guaranteed ordering per-stream)

  3. A good way to build read models is to use subscriptions (works about the same in both GES and NES) – you get ordering guarantees per aggregate/stream this way

  4. It’s fine to persist the same event into several streams (e.g. a single event changes the state of an aggregate, and can also be used to change the state of some kind of process-manager/saga thing, if that “thing” is also event-sourced)

  5. With NES, the idea is that your events to dispatch on your bus (and possibly also commands for a JO-Saga) are persisted as part of the “commit” and are guaranteed to be dispatched to the (external) Bus as part of the commit process.

I’m trying to figure out how to handle #5 w/ GES – essentially coordinating work between streams.

My understanding of the general way to handle this is to modify a single stream during each unit of work, then issuing-a-command/publishing-an-event and having another handler that does the some work with another stream.

With GES, handling events seems to make sense, you’d do it the same way you’d build your Read Model – use a subscription (probably using competing consumers so we are getting exactly-once delivery) and then figure out what logic needs to handle the event.

For commands, would it make sense to take the same approach? For example, simply write commands into the same GES stream (or create some kind of pseudo event “CommandDispatched(cmd)”) and use a competing-consumer-subscription to execute the commands? Only JO-Saga style entities would be dispatching commands.

I can think of a couple ways of handling this:

a) Use my Bus for handling dispatching commands/events and write events to GES to persistence, using DTC to tie the two together (probably a bad idea)

b) Totally ditch the Bus, and use GES – but now there’s issues dealing with SLAs, retries, error queues, poison messages, etc. I suppose I could use a few different streams to handle this logic, but I think I end up writing my own Bus at this point…

c) Write everything to GES, then have a subscriber that takes all events and “DispatchedCommands” and sends them directly to the Bus. It’s only responsibility is getting these messages to the Bus infrastructure.

Does anybody have any thoughts/experience/guidance on how they’ve handled this sort of situation?

Am I even heading down the right track here?

Thanks.

"
With GES, handling events seems to make sense, you'd do it the same
way you'd build your Read Model -- use a subscription (probably using
competing consumers so we are getting exactly-once delivery) and then
figure out what logic needs to handle the event."

CC doesn't give exactly once it gives at least-once

"b) Totally ditch the Bus, and use GES -- but now there's issues
dealing with SLAs, retries, error queues, poison messages,"

This is EXACTLY what competing consumers does (retries, error queues, etc)

Correction: after reading this thread: https://groups.google.com/forum/#!topic/event-store/VcxJ-Y1p6OE it seems that competing consumers does not guarantee ordering, even with a single consumer.

If I’m concerned about ordering, I believe I can use regular catchup subscriptions and use standard idempotency techniques to ensure exactly-once delivery (if I care about that).

Sorry Greg, didn’t see your reply.

Seems like I need a bit more education – where can I go to learn more about competing consumers?

I’ve been unable to get find current info via any of the Google terms I’ve thought of using.

Just realized my last post could have been more specific:

Where can I go to learn more about competing consumers in GES?

I like the idea of ditching my bus, as much of what it does is duplicating what’s in GES.

They were just recently moved into dev. I would expect the docs to be
up within the next week or two (I am writing them but am traveling to
London for a week tomorrow so likely will finish them when I get back
on the 19th)

Overall they are pretty simple and work roughly like any other
competing consumer implementation. You can connect N clients to a
subscription group (on a stream). The N connections will receive
messages from that stream at least once between them (e.g. if you have
5 consumers it will attempt usually to round robin between the 5).
There is a limit of in flight messages per connection that is honored.

First thing you do is CreatePersistentSubscription
https://github.com/EventStore/EventStore/blob/dev/src/EventStore.ClientAPI/IEventStoreConnection.cs#L393
(needs to be done once at deployment etc). Clients then
ConnectToPErsistentSubscription
https://github.com/EventStore/EventStore/blob/dev/src/EventStore.ClientAPI/IEventStoreConnection.cs#L290

Clients can either autoack or explicitly ack/nack messages from the
server. You can specify configuration on the subscription group things
like how long until a message gets retried. After how many retries
should the message get parked (dead letter queue) etc. The XML docs on
the methods are fairly reasonable if you want to look through it
and/or play with it.

Scroll down from
https://github.com/EventStore/EventStore/blob/dev/src/EventStore.ClientAPI/IEventStoreConnection.cs#L272
next few methods....

https://github.com/EventStore/EventStore/blob/dev/src/EventStore.ClientAPI/PersistentSubscriptionSettingsBuilder.cs
https://github.com/EventStore/EventStore/blob/dev/src/EventStore.ClientAPI/PersistentSubscriptionNakEventAction.cs

There is also statistics available and a ui being built to show them
(part of what we are waiting for on binary push). To access the
statistics of your subscriptions try /subscriptions there is a rel
link in each for more information which is
/subscriptions/stream/subscribergroupname it includes connections,
messages/second overall + per connection, retries, etc etc

Greg

Thanks! that’s really helpful

Hello! I can’t find out what GES stands for. Could you please expand the acronym?

Well 8 to 10 years ago EventStoreDB used to be called GetEventStore.

1 Like