Handling poison events in projections

EventStoreDB
1

We are looking at the best way to ensure that our projections are robust, as during the Dev phase we have found that occasionally changes to event structure or issues with the code producing the events cause the projections to fault and we end up having to delete the streams.

What are best practices here? Our concerns are that a bug in the code that produces the events might mean an event has a null property where we are not expecting one and if the projections rely on this property then the projection will fault.

Should we be writing our projections in such a way as to handle these cases and maybe project these events into a separate 'error' stream?

What are others doing to ensure that bugs of this type don't bring the whole system to a halt? And what is the best way to recover from this situation? Usually in a message based systems poison messages could be adjusted and replayed, but what is the best course of action when the events are immutable?

2

Long answer http://leanpub.com/esversioning

Short answer by default projections will fault, you can put code into them defensively to prevent them from faulting but the results will be incorrect for some amount of data in the projection.

3

In our projector for example, we simply ignore events that are in an invalid state. (see the example below, where we check that metadata exists, then check that the eventName property exists, then that the stream Id follows our required format, otherwise skipping that event)

fromAll().when({

$any:(function(s,e) {

if (e.metadata === null) return;

var eventName = e.metadata.eventName;

if (!eventName) return;

var lastDot = e.streamId.lastIndexOf(’.’);

if (lastDot == -1) return;

var namespace = e.streamId.slice(0, lastDot);

linkTo(‘en-’ + namespace + ‘.’ + eventName, e);

})

})