Projections vs. Subscriptions

EventStoreDB
#1

Hi all!
I’m still fairly new with the topic, so please pardon my ignorance.

I have got something going with EventStore and I think I basically got the hang of things.

I have more of a best-practice question now: are projections any good for serious usage, or should I start doing things right away in my code outside ES, by just subscribing to streams, and essentially building out projections myself?

The pros of projections:

  • works out of the box, easy to get started

  • monolithic with ES, no need to worry about maintaining a persistent connection between ES and my own code

  • nice GUI to quickly test and debug projections

The cons of projections:

  • once the grow and start to contain more and more business logic, they become less DRY and in general hard to version, maintain and deploy

  • impossible to use some more advanced logic in them. E.g. if I would need to calculate an md5 hash or something of the sort.

First, is my evaluation above correct?

If so, then I would think it’s better for most usecases to do away with built-in projections, and take care of building them out in user code.

But then I’d love to hear your experience how you would actually do that…

My preferred languages of choice would be nodejs.

  1. I can see that there are 3 competing libraries available for nodejs. Any tips which one is more mature?

  2. Would I just use a catch-up subscription and project needed state into memory? Then on every start-up I’d essentially rebuild all projections. Shouldn’t be too slow since I only have thousands of events, but still doesn’t feel too reliable, right?
    Or should I build-out a persistent projection into something like redis? But in this case how will I make sure that I didn’t miss any events? Is that something I would use persistent subscriptions for?

  3. If you got examples how you deal with projections I’d really appreciate, especially in nodejs.

Regards,

Dmitri

#2

Ah, and one more thing, it would have been nice if there existed some sort of projection library (for nodejs) that would mimic the built-in projection api, i.e. all those things like fromStream, when, transofrmBy etc…

#3

There seems to be a limit on number of projections according to this thread: https://groups.google.com/forum/#!topic/event-store/o90XyXZ9N70

quote : "“100 projections seems like a lot, I would question exact use-cases and
whether things like stateparition on a single projection can be used.
Projections are quite like stored procs they should be used sparingly. “””

I did not get an answer so I don’t know why the limit.

#4

I only need a handful projections, that’s not a problem for me.
I mostly care about using custom logic and code reuse.

Any reason why this code is not published as an npm package?

https://github.com/EventStore/EventStore/tree/release-v4.1.1/src/EventStore.Projections.Core/Prelude

#5

The code is not published as a npm package as it won’t work in a standard js environment, it requires the C++ code ES adds.

#6

Hey Greg,
I understand that. But it shouldn’t be too much effort to refactor it to be able to work standalone, right?

In that case transition from running projections inside ES to running them standalone would be quite smooth and painless.

I could maybe give it a try at some point, but not sure if I’d find the time.

#7

E.g. if that JS code hand some interface that would define all things that need to be injected from the host environment. In case of Node that would be simple binding, and for other languages it could work the way you do it currently. Anyways, I see a lot of value in extracting that logic.

#8

Running standalone would be quite a bit of effort. I have scoped the work before. Have you worked much in C++?

#9

If not clear there is a large amount of C++ code around projections. Likely it would be best to host it directly in C++ not managed code. I have started a C client API which would be a good starting point towards this. https://github.com/gregoryyoung/libesclient/

#10

I have been lazy on getting it done btw though that may change in the near future :slight_smile:

#11

On a side note you might get a laugh at https://github.com/gregoryyoung/libesclient/blob/master/src/wtf_uuid.c

#12

Have you worked much in C++?

Did some projects back in the uni years… But ain’t no expert.

Perhaps I underestimated the topic, would need to dig a bit deeper anyways.

My thinking was this: projections inside ES are not really maintainable, so I’d need to do something at my side. If I have to re-implement it, then I’d rather have the compatible syntax… then I spotted that prelude code and hoped it would be easy to migrate…

On a side note you might get a laugh

WTF?!

#13

Looking at your link, there doesn’t seem to be much (any?!) code related to projections, more about communicating with ES itself.
I didn’t plan to re-implement that part.

I merely wanted to subscribe to all events from ES via one of existing nodejs clients, and then just feed those events to my projections in nodejs, build up the state, and then export it to be later persisted or kept in memory.

#14

There is already a node client for ES. That would be a good starting point (and is an interesting idea). That would be much simpler than getting the C++ code to work standalone outside of eventstore.

#15

Yes, that’s exactly what I said. I just want to re-use some code you have there in prelude. But if it was possible to extract it, it would have been even better.

#16

You can’t reuse the code as things like

fromStream(“foo”) are actually implemented in C++ (prelude is external definitions). You would need to provide the same functions in javascript.

#17

I see, will try to look through the code and see how hard that would be.