Integrating Bounding Contexts with ES as the Source of Truth

EventStoreDB
#1

After watching Gregs Polyglot data video

http://www.ustream.tv/recorded/46673907

I now understand why using a message bus isn’t a great idea for integrating my bounded contexts. Its better that I have multiple client driven subscriptions (lets call them synchronisers), each polling streams in the Event Store to know what actions they need to take (populating a read model or calling another bounded context via an api).

The question is this, how are people managing their synchronisers (one set per bounded context)? If I have 5 bounded contexts to integrate, do I have a little management console for each one? Do you have one application\management console where you deploy the various event handlers from each bounded context? One of the aspects I was interested in was, that each bounded context and it synchronisers would be independently deployable without having to test the others.

I did work on a system like this before, a custom message queue\publish-subscribe approach. So I’m wondering how are other in the community are going about this? How do you manage and monitor that a call to another bounded context failed when processing an event (downstream)? In the previous system, we would move that message\event to a dead letter queue, and further messages with a shared correlation id would have to be left unprocessed, which would mean, the entire stream could not be processed, as messages needed to consumed sequentially.

#2

So I wouldn't recommend lots of downstream processing through services
subscribing to each others events. This often leads to a tangled mess
just through events. Normally what you are looking for here is a
process manager etc unless its very simple.

"I did work on a system like this before, a custom message
queue\publish-subscribe approach. So I'm wondering how are other in
the community are going about this? How do you manage and monitor
that a call to another bounded context failed when processing an event
(downstream)? In the previous system, we would move that
message\event to a dead letter queue, and further messages with a
shared correlation id would have to be left unprocessed, which would
mean, the entire stream could not be processed, as messages needed to
consumed sequentially."

For processing (not for projections of any kind) take a look at
competing consumers. I believe it does what you are looking for.
http://docs.geteventstore.com/introduction/competing-consumers/

#3

I face a similar question on how to integrate bounded contexts - for example a master data management BC with production BC.
In my production BC I would expect to have streams for each aggregate in that context, but the events from the master data management BC might not map directly to such streams - for example if the aggregate is bigger in the master data management, while I need more granular aggregates in my production BC. I also wouldn’t feel comfortable coupling the aggregates of both BC this way.

Instead of subscribing directly to services in the other BC, I would put a single “integrator” on the boundary of each BC, acting as an ACL.
This integrator subscribes to $all and processes events from the BCs it has to integrate with.
Additionally, it needs to track its position persistently. I’m hesitant to use competing consumers for this because I would still have to detect duplicate events (due to at-least-once).
Since I have to track my last position anyway, I could just go with plain subscriptions and reads:

  • Each event that the integrator publishes has metadata about the corresponding position it processed
  • The integrator publishes checkpoint-events to a count-limited stream (for example: every 100.000 events)
  • On startup, it loads the last checkpoint and reads forward from the position indicated by the checkpoint and looks for events with such metadata
  • If it finds any, it will use the last position it found this way when starting processing - otherwise it will use the position indicated by the checkpoint as a starting point
    Anyway, that’s the idea I’m toying with at the moment. Any comments?
#4

sounds sensible enough, is it only 2 BC you are integrating? perhaps if it was more then that you’d get into the complexity issues greg was talking about.

#5

It most likely is going to be more than 2 BC. Aside from the master data, we’re dealing with an entire ERP - orders (sales and purchases) and current stock influence demand for production, planned productions cause demand on materials etc.