Moar errors

EventStoreDB
1

I wrote a really crap projection

fromStream(‘github’)

.partitionBy(function(ev) {

if(ev.body.repo) {

return ev.body.repo.full_name

}

})

.when({

$init: function(state, ev) {

return {}

},

“PushEvent”: function(state, ev) {

if(state.lastPush) {

var newDate = new Date(ev.body.created_at)

, lastDate = new Date(state.lastPush.body.created_at)

, difference = (newDate.getTime() - lastDate.getTime()) / 1000

if(difference < 120) {

emit(‘paranoidpushes’, “ParanoidPush”, {

first: state.lastPush,

next: ev

})

}

}

state.lastPush = ev

}

})

My error is

“null”

The error log has a stack trace with js1 saying “the projection failed due to ‘null’”

Any ideas? :stuck_out_tongue:

2

Rob,

I suspect this is due to partitionBy returning null. It was not intended to
pre-filter events, but it seems reasonable to ignore with no partition
assigned.
Anyway, this error is just a guard against missing check in other place.

I'll fix the error message. Let me know if I guessed right about the
reason.

-yuriy

3

There is a chance that is the case, I know some of my events are missing repo information because Github changed their API about two weeks before I shut down the event pump

4

Btw, I changed the projection to only emit the data I actually need in the new event

This will cut down a lot on the disk space taken by the new stream, I also thought that perhaps it would make the projection run faster - making the assumption I was IO bound and writing huge events would be slow but this doesn’t seem to be the case.

What are the potential bottlenecks when running projections that generate new events to a stream?

5

Rob,

writes are quite fast if you write a lot of events to a limited number of streams, as projections batch writes to the same stream. However, if you do writes to a very large umber of streams from a projection, each write goes a separate request and it is slower. In this case you often see /paused on projection status meaning that projections is not fast enough to handle events with the pace of the event reader.

-yuriy

6

Okay, well I’m writing all of these to the same stream so they should be batched right?

What about that partitionBy call? Does this play with the batching capabilities? Are we theoretically in a different projection when it comes to batching? I’m seeing myself capped at 40 events per second and I’m fairly sure I had it faster on my local machine (I’ve not been smart configuring the EC2 maching and the disk is likely quite slow, hence wondering if making smaller events would help)

On the original topic

.partitionBy(function(ev) {

if(ev.body.repo) {

return ev.body.repo.full_name

}

return ‘’

Seems to still get a ‘null’ error

7

And a ‘Disposed’ error

I think something has gone wrong :slight_smile:

8

Is it possible that ev.body.repo.full_name is null?

9

Do you see anything in error log?

10

No, if I have repo it has full_name, this is part of the contract from Github

That said, I’ll bullet-proof this and try anyway

11

[PID:22433:016 2013.05.09 07:11:48.150 ERROR ProjectionManager ] The ‘paranoid_pushes’ projection faulted due to ‘Disposed’

[PID:22433:016 2013.05.09 07:11:48.150 ERROR ProjectionManager ] The ‘paranoid_pushes’ projection faulted due to ‘Disposed’

Aaah we also now have

[PID:22433:016 2013.05.09 07:07:16.875 ERROR ProjectionManager ] The ‘paranoid_pushes’ projection faulted due to ‘null’

[PID:22433:018 2013.05.09 07:07:16.875 ERROR QueuedHandlerAutoRes] Error while processing message EventStore.Core.Messages.ClientMessage+ReadStreamEventsBackwardCompleted in queued handler ‘Projection Core #1’.

System.InvalidOperationException: An event emitted in recovery differ from the originally emitted event. Existing(’$Checkpoint’, ‘github: 288459’). New(’$Checkpoint’, ‘github: 288343’)

at EventStore.Projections.Core.Services.Processing.EmittedStream.ValidateEmittedEventInRecoveryMode (System.Tuple`3 topAlreadyCommitted, EventStore.Projections.Core.Services.Processing.EmittedEvent eventsToWrite) [0x00000] in :0

at EventStore.Projections.Core.Services.Processing.EmittedStream.SubmitWriteEventsInRecovery () [0x00000] in :0

at EventStore.Projections.Core.Services.Processing.EmittedStream.ReadStreamEventsBackwardCompleted (EventStore.Core.Messages.ReadStreamEventsBackwardCompleted message, EventStore.Projections.Core.Services.Processing.CheckpointTag upTo) [0x00000] in :0

at EventStore.Projections.Core.Services.Processing.EmittedStream+c__AnonStorey1D.<>m__42 (EventStore.Core.Messages.ReadStreamEventsBackwardCompleted completed) [0x00000] in :0

at EventStore.Core.Messaging.RequestResponseDispatcher`2[EventStore.Core.Messages.ClientMessage+ReadStreamEventsBackward,EventStore.Core.Messages.ClientMessage+ReadStreamEventsBackwardCompleted].Handle (EventStore.Core.Messages.ReadStreamEventsBackwardCompleted message) [0x00000] in :0

at EventStore.Projections.Core.Services.Processing.ProjectionCoreService.Handle (EventStore.Core.Messages.ReadStreamEventsBackwardCompleted message) [0x00000] in :0

at EventStore.Core.Bus.MessageHandler`1[EventStore.Core.Messages.ClientMessage+ReadStreamEventsBackwardCompleted].TryHandle (EventStore.Core.Messaging.Message message) [0x00000] in :0

at EventStore.Core.Bus.InMemoryBus.PublishByType (EventStore.Core.Messaging.Message message, System.Type type) [0x00000] in :0

at EventStore.Core.Bus.InMemoryBus.DispatchByType (EventStore.Core.Messaging.Message message) [0x00000] in :0

at EventStore.Core.Bus.InMemoryBus.Publish (EventStore.Core.Messaging.Message message) [0x00000] in :0

at EventStore.Core.Bus.InMemoryBus.Handle (EventStore.Core.Messaging.Message message) [0x00000] in :0

at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue (System.Object o) [0x00000] in :0

I assume this is because of the original error,

12

Does “projections” screen show the projection position? Or “projection details” screen? github:NNNN ?

13

Yeah, it’s github: 288482

/streams/github/288482?format=json

Which is all very well and good, but there is nothing weird about that event at all - it has repo information, isn’t too big and seems totally legit

14

One event before/after?

15

Ditto

16

I have another projection which is about to reach the same point (the ‘lite’ version, we’ll see if that dies too I guess)

17

I was wrong about my guess with partitionBy, it handles nothing/null.

Looking for other reasons…

18

If you need/want root access to the box just ping me on Skype and send me your public key

19

Oooh

Event github/288488?format=json

Is the one with the mega huge commit, is it possible that it failed on this and didn’t write the preceding events?