Projections should be persisted once they are built.
Given there are a lot of deployments per day when using continuous delivery - you dont want to replay all events (it could be millions)
To continue building read model after process restart - you can store some kind of “position” - global sequence number for example (no idea how to do it without global sequence though)
In my implementation we had few steps in RM building evolution.
- Just listen to “All” sequence, and pass event to all interested handlers
Here the most obvious problem is with events processing in memory.
suppose we are on the step of processing event number 255
and there are two handlers interested in it A and B
if we just do it like :
for(var handler in handlers) handler.Process(e);
if A fails - B cannot process it,
if we wrap processing in try catch or if we do it thread per handler a bit different problem appears
for(var handler in handlers) handler.ProcessAsync(e);
A is ok, but B fails - - but the position is marked as processed (read model is marked to be caught up with 255 position)
but that’s not true
now if B tried to push this to dead letter queue - that’s not guaranteed to be done if the process is exiting for example
So the second stage was creating second version of processor:
- Do processing in two stages.
in first stage one handler reads the “All” and sends to secondary event store by inserting to as many streams as are views to be built - event per view, write are idempotent - so
for(var view in views) view.Enqueue(e);
may fail - but thats ok - after restart or re-connection it can be idempotently written (or not) to the stream again
second stage handlers having their own stream save their positions independently
also having second ES - the processing can be done on different server
the secondary handlers also log per-view dead events in per - view dead streams (for debugging) - I sketched down the idea of secondary processing here http://blog.devarchive.net/2015/11/how-to-process-sequence-of-events.html
The system became stable
but the most uncomfortable problem still exists
You need to rebuild read model after schema changes.
So I have idea about 3rd rewrite
This will differ from previous by inserting one processing stage between per-view stream and original stream
So the next version will have 3 stages
main handler writing to secondary ES, by writing event type per stream
per view intermediate handlers keeping per view streams by tracking all type streams as input - these are known from projections itself
per view handlers work as before
so now when schema changes you will need to increment a “projection version” - this version will be used to locate correct per view stream (it’s name will include version)
now - the input to the 3-rd handler will be 2 channels
a) the “main” handler provider reading all secondary ES events - and whoever is up to date will use them
b) the per view stream - and if its not kept up with main - will be rebuilt from per-event-type events
this way once you change read model schema and mark it’s version with higher number - the system will catch up very quickly once started
One might think there can be high latency for 3 stage processing
but I use secondary “fast” channels for triggering next stage faster (http://blog.devarchive.net/2015/11/reducing-latency-in-cqrs-applications.html)
But in summary I think the way I chose led me to re-creating map-reduce framework, so I try to find some existing alternative which provide some good and reliable map reduce (because I still not sure about performance implications when using home grown processor)