Occasionally Connected + By-User-Subscribed-Data-Subset

EventStoreDB
#1

I’m working on a project that requires me to distribute subsets of a centralized database to N devices based on user ‘subscriptions’. In Oil & Gas, users on offshore oil platforms never have reliable internet connections and must be able to work offline for periods of time, as well as be able to pass uncommitted data to another device totally offline… Think: the helicopter comes to end someone’s rotation before he’s had a chance to sync up his data… I’ve seen the CQRS-not-just-for-server video and believe that those methods + Event Store could satisfy my requirements, but I just cannot wrap my head around architecturally how to do it. Each user must be able to subscribe/unsubscribe himself to aggregates/streams. Upon subscribing, he must receive all data from beginning of stream. But the subscription can also happen by an external process while he’s offline, such that when he reconnects, the data is waiting for him.

Best I could come up with was having a Catch-Up subscriber always running, keeping the state of everyone’s subscriptions to aggregates, and also listening for $all events. When events come through, the metadata is checked to receive an AggregateRootId which is used against the subcription-state to determine all users who should “get” that event. From there… umm… message bus to each device? Or perhaps copy the event to a special per-user stream (<- I like this because then the user can use a very easy catch-up subscription against this stream). But this seems like bastardizing the event store into a message broker (which Greg said never to do in the “Event Store as a Read Model” video).

Any advice would be greatly appreciated. I’d __love __to be able to setup a fully working demonstration of the occasionally connected functionality with Event Store after <3 days of “development” to my superiors.

Thanks,

Chris

#2

To be clear your event store is a form of topic based pub sub you just don’t want to get tons of logic in it.

As a question: how many aggregates will each client be listening to? 10? 100? 1000?

Cheers,

Greg

#3

I could go one way and have them be subscribed to ~20,000 tiny aggregates (~40bytes) that change infrequently plus ~100s of “big” (10MB demoralized) aggregates. Or I could probably wrap the 20K into a single stream, making the answer 100s with 1000s being an edge case.

#4

how many aggregates go to all clients vs how many are client specific?

#5

The 20,000 set would go to all, the rest are subscribed to by the user or by some external process.

#6

So what I would do is setup a single stream for things that go to all (eg all support data etc). You can still keep it in many streams and use a small projection to link it to a single stream all clients can read eg put in metadata on the events that they should be available for all then

fromAll().when({

$any : function(s,e) {if(e.metaData.routeToAll) linkTo(‘alldevicenotifications’, e)

})

#7

Feel free to drop me a note offline for further information btw.

Cheers,

Greg

#8

Thank you Greg, I just sent an email to you. I will also update this thread after running some experiments with your proposed solution in case someone else might have a similar requirement.

#9

I do have a similar requirement. Is there any update? How can the ocassionally connected client “push” events to the server continously, since the server can not ‘pull’ the events from the client that does not have a public static IP address. Thanks!

#10

Or is my question out of the Event Store scope? Thanks in advance

#11

Why not just use an atom feed over http?

#12

Thanks for reply. AFAIK the atom feed belongs to the server, and it is useful to ‘pull’ events from the server. Buth what happen when the client that is constantly changing its network (public wifi, 3G) wants to push its own events to the server. The server does not know where the client is. I just do not want to write a REST Api in the server and go RPC if there is a better and battle tested solution.

#13

You mean like posting the event to the atom feed?

#14

http://docs.geteventstore.com/http-api/3.7.0/writing-to-a-stream/

#15

That look nice! Thanks. So the client will commit to his own database, and another process will try to push to the post api in the event store, and retry after some period of time when offline. Is this a correct approach?

#16

That can work. Might just be another thread though not another
process. It also doesn't have to be a "database" could just be a file.
But the client just has to POST the event to send it to event store.
For receiving events the atom feed delivers the events.

#17

You are right. Thanks a lot Greg. Nice work!