Recreation projection (with same) name fails

EventStoreDB
1

With event store version 3.5.0 we cannot recreate a projection with the same name.

(reason to do this: projection is in a faulted state; no clue why this is the case; how to prevent it from getting into that state)

Deleting was already a quest:

  • via UI I deleted the projection (named projects)

  • via command prompt I deleted two associated streams:

curl.exe -v -X DELETE http://127.0.0.1:2113/streams/projects -H “HardDelete:true”

curl.exe -v -X DELETE http://127.0.0.1:2113/streams/$$projects -H “HardDelete:true”

  • Then I did a scavenge

After recreating the projection with the same name the systems ends up in some infinite loop.

It looks like the system is still holding some internal references to the old projection with the same name.

I guess this is the case because using a different name for the projection works.

However I need to have the projection with the original name.

So definitelly I’m not doing it the correct way.

Documentation is not very helpful so I had to try something (and found some hints like deleting the associated streams).

Or do I need to upgrade to version 3.8.0?

However, we did a trial on a test machine, it looks like that creating a projection with a previously used name, is still a problem.

Can someone help me out?

Kind regards,

Michel

2

If the projection was emitting (e.g. linkTo or emit) was used, recreating that projection would cause issues. With version 3.8.0 that has been addressed and details about your issue is probably explained in the 3.8.0 blog post (https://geteventstore.com/blog/20160713/event-store-3.8.0-released/index.html) if you are emitting to a stream via projections.

Do you perhaps have logs that you can share about the projection that went into a faulted state? If you do you can send them to pieter at geteventstore dot com.

3

Emit enabled is used (and checkpoint enabled as well, continuous mode).

Thx for the links. The error lloks familiar. However as far as we know we do have one projection that is emitting events to that stream.

Sure I can send you the error log. This what happened the time before we saw the projection was in faulted state.

We have several projections (one is projects) which are aggregated in one other projection maed domainevents.

[PID:02280:016 2016.07.15 07:24:08.498 ERROR ProcessingStrategySe] ‘emit’ is not allowed by the projection/configuration/mode

[PID:02280:013 2016.07.15 07:24:08.529 ERROR ProjectionManager ] The ‘projects’ projection faulted due to ‘‘emit’ is not allowed by the projection/configuration/mode’

[PID:02280:016 2016.07.15 11:28:08.482 ERROR ProcessingStrategySe] ‘emit’ is not allowed by the projection/configuration/mode

[PID:02280:013 2016.07.15 11:28:08.497 ERROR ProjectionManager ] The ‘projects’ projection faulted due to ‘‘emit’ is not allowed by the projection/configuration/mode’

[PID:02280:016 2016.07.15 11:30:34.902 ERROR ProcessingStrategySe] ‘emit’ is not allowed by the projection/configuration/mode

[PID:02280:013 2016.07.15 11:30:34.961 ERROR ProjectionManager ] The ‘projects’ projection faulted due to ‘‘emit’ is not allowed by the projection/configuration/mode’

[PID:02280:014 2016.07.15 11:33:39.742 ERROR ProcessingStrategySe] ‘emit’ is not allowed by the projection/configuration/mode

[PID:02280:013 2016.07.15 11:33:39.903 ERROR ProjectionManager ] The ‘domainevents’ projection faulted due to ‘‘emit’ is not allowed by the projection/configuration/mode’

After this point in time I tried to delete the projection and recreate it. Which failed as I explained in my first post.

– Michel

4

In the cluser-node log around this time you can see this in the log:
[PID:02280:016 2016.07.15 07:24:08.498 ERROR ProcessingStrategySe] ‘emit’ is not allowed by the projection/configuration/mode

[PID:02280:016 2016.07.15 07:24:08.498 DEBUG ResponseWriter ] PROJECTIONS: Scheduling the writing of $faulted to $projections-$master. Current status of Writer: Busy: True

[PID:02280:016 2016.07.15 07:24:08.529 DEBUG ResponseWriter ] PROJECTIONS: Finished writing events to $projections-$master: $faulted

[PID:02280:013 2016.07.15 07:24:08.529 DEBUG ProjectionManagerRes] PROJECTIONS: Response received: 314@$faulted

[PID:02280:013 2016.07.15 07:24:08.529 ERROR ProjectionManager ] The ‘projects’ projection faulted due to ‘‘emit’ is not allowed by the projection/configuration/mode’

5

It does appear that you have a projection that is either using linkTo or emit and since “emit” is not enabled for the projection it fails.

6

So if you are doing something like

fromCategory(‘project’)

.whenAny(function(s,e) {

linkTo(‘projects3’,e);

});

I have to turn on “emit enabled”?

I tought I only had to do that when I am really creating new events from the projection (with emit(…)) .

7

I just upgraded to 3.8.0

Created a new projection (projects3projection)

fromCategory(‘project’)

.whenAny(function(s,e) {

linkTo(‘projects3’,e);

});

Then deleted the projection with all checkboxes on (so including “Delete Emitted Streams”).

(still I see stream projects3 in UI)

Then I did a scavange.

Then I tried to recreate the projection it gives:

The ‘projects3projection’ projection faulted due to ‘Multiple projections emitting to the same stream detected. Stream: ‘projects3’. Last event projection: ‘13’. Emitting projection: ‘21’’

After deleting a projection I cannot recreate it looks like.

Kind regards,

Michel

8

When you created the projection, did you check the “track emitted streams” checkbox?

9

The following blog post on the 3.8.0 explains a little more about what is going on with this particular issue https://geteventstore.com/blog/20160713/event-store-3.8.0-released/index.html
In addition to that a similar issue was encountered by the following user and another explanation posted which might be valuable to read https://groups.google.com/forum/#!topic/event-store/LW55hGJ4UfI

10

I did check that!

11

This story (https://groups.google.com/forum/#!topic/event-store/LW55hGJ4UfI) is like our story. Thanks.

"We were going to publish a blog post about this but i’ll answer you here.

There were 2 PRs that made the EmittedStreams a little more tolerable as well as a new feature in projections that tracks emitted streams and on deletion you can choose to delete the emitted streams.

Unfortunately I couldn’t find a work around for this release and to resolve the issue you have to delete the emitted streams from the projection and re-create the projection.

The reason is that because when a projection emits events it places metadata in the event. This metadata contains the ID of the projection. When a projection emits events to a stream it will validate the metadata from the last event and determine whether it is the only projection emitting to that stream by checking that it’s ID matches that of the last event in the emitted stream.

The PRs addresses this issue in first making the Emitted Streams a little more tolerable in the case when the emitted stream has been deleted. This means that you are able to delete the streams the projection emitted to and able to create a new projection that emits to those same streams.

The second is giving the user the ability to specify in the creation and the deletion of a projection whether they wish to track and delete the streams that the projection emits to."

’ PR’ stands for?

So apart form doing the delete in the UI I need to do an extra delete with HTTP call for the stream? Both streams (projects3 and $$projects3)?

The blogpost you are referring to is already available?

The post here https://geteventstore.com/blog/20160713/event-store-3.8.0-released/index.html does not say anything about doing an extra HTTP DELETE. So I guess some other blogpost is coming?

Kind regards,

Michel

12

Created a new test projection: projects42projection

fromCategory(‘project’)

.whenAny(function(s,e) {

linkTo(‘projects42’,e);

});

After doingdelete projection in UI

then

curl.exe -v -X DELETE http://127.0.0.1:2113/streams/projects42 -H “ES-HardDelete:true” (result HTTP/1.1 410 Stream deleted)

curl.exe -v -X DELETE http://127.0.0.1:2113/streams/$$projects42 -H “ES-HardDelete:true” (result HTTP/1.1 410 Stream deleted)

And then recreating the projects42projection I got this error

Internal error
at EventStore.Projections.Core.Services.Processing.EmittedStream.SubmitWriteEvents() in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\EmittedStream.cs:line 466
at EventStore.Projections.Core.Services.Processing.EmittedStream.ProcessWrites() in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\EmittedStream.cs:line 385
at EventStore.Projections.Core.Services.Processing.EmittedStream.EmitEvents(EmittedEvent[] events) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\EmittedStream.cs:line 179
at EventStore.Projections.Core.Services.Processing.ProjectionCheckpoint.EmitEventsToStream(String streamId, EmittedEventEnvelope[] emittedEvents) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\ProjectionCheckpoint.cs:line 147
at EventStore.Projections.Core.Services.Processing.ProjectionCheckpoint.ValidateOrderAndEmitEvents(EmittedEventEnvelope[] events) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\ProjectionCheckpoint.cs:line 76
at EventStore.Projections.Core.Services.Processing.CoreProjectionCheckpointManager.EventsEmitted(EmittedEventEnvelope[] scheduledWrites, Guid causedBy, String correlationId) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\CoreProjectionCheckpointManager.cs:line 207
at EventStore.Projections.Core.Services.Processing.ResultWriter.EventsEmitted(EmittedEventEnvelope[] scheduledWrites, Guid causedBy, String correlationId) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\ResultWriter.cs:line 94
at EventStore.Projections.Core.Services.Processing.EventSubscriptionBasedProjectionProcessingPhase.FinalizeEventProcessing(EventProcessedResult result, CheckpointTag eventCheckpointTag, Single progress) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\EventSubscriptionBasedProjectionProcessingPhase.cs:line 541
at EventStore.Projections.Core.Services.Processing.CommittedEventWorkItem.WriteOutput() in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\CommittedEventWorkItem.cs:line 78
at EventStore.Projections.Core.Services.Processing.StagedProcessingQueue.Process(Int32 max) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\StagedProcessingQueue.cs:line 115
at EventStore.Projections.Core.Services.Processing.CoreProjectionQueue.ProcessOneEventBatch() in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\CoreProjectionQueue.cs:line 138
at EventStore.Projections.Core.Services.Processing.CoreProjectionQueue.ProcessEvent() in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\CoreProjectionQueue.cs:line 50
at EventStore.Projections.Core.Services.Processing.EventSubscriptionBasedProjectionProcessingPhase.ProcessEvent() in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\EventSubscriptionBasedProjectionProcessingPhase.cs:line 144
at EventStore.Projections.Core.Services.Processing.CoreProjection.Tick() in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Processing\CoreProjection.cs:line 637

So unfortunately no success yet.

– Michel

13

And to be more precise: the event store fails at line 406 of the SubmitWriteMetadata

System.ArgumentOutOfRangeException was unhandled by user code

HResult=-2146233086

Message=$tb should be non-negative value.

Parameter name: truncateBefore

ParamName=truncateBefore

Source=EventStore.Core

StackTrace:

at EventStore.Core.Data.StreamMetadata…ctor(Nullable1 maxCount, Nullable1 maxAge, Nullable1 truncateBefore, Nullable1 tempStream, Nullable`1 cacheControl, StreamAcl acl) in C:\data\code\EventStore\src\EventStore.Core\Data\StreamMetadata.cs:line 35

at EventStore.Projections.Core.Services.Processing.EmittedStream.SubmitWriteMetadata() in C:\data\code\EventStore\src\EventStore.Projections.Core\Services\Processing\EmittedStream.cs:line 406

at EventStore.Projections.Core.Services.Processing.EmittedStream.PublishWriteEvents() in C:\data\code\EventStore\src\EventStore.Projections.Core\Services\Processing\EmittedStream.cs:line 549

at EventStore.Projections.Core.Services.Processing.EmittedStream.SubmitWriteEvents() in C:\data\code\EventStore\src\EventStore.Projections.Core\Services\Processing\EmittedStream.cs:line 504

at EventStore.Projections.Core.Services.Processing.EmittedStream.SubmitWriteEventsInRecovery() in C:\data\code\EventStore\src\EventStore.Projections.Core\Services\Processing\EmittedStream.cs:line 609

at EventStore.Projections.Core.Services.Processing.EmittedStream.ReadStreamEventsBackwardCompleted(ReadStreamEventsBackwardCompleted message, CheckpointTag lastCheckpointPosition) in C:\data\code\EventStore\src\EventStore.Projections.Core\Services\Processing\EmittedStream.cs:line 316

at EventStore.Projections.Core.Services.Processing.EmittedStream.<>c__DisplayClass45_0.b__0(ReadStreamEventsBackwardCompleted completed) in C:\data\code\EventStore\src\EventStore.Projections.Core\Services\Processing\EmittedStream.cs:line 395

at EventStore.Core.Messaging.RequestResponseDispatcher`2.Handle(TResponse message) in C:\data\code\EventStore\src\EventStore.Core\Messaging\RequestResponseDispatcher.cs:line 75

at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message) in C:\data\code\EventStore\src\EventStore.Core\Bus\MessageHandler.cs:line 33

at EventStore.Core.Bus.InMemoryBus.Publish(Message message) in C:\data\code\EventStore\src\EventStore.Core\Bus\InMemoryBus.cs:line 324

at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o) in C:\data\code\EventStore\src\EventStore.Core\Bus\QueuedHandlerMRES.cs:line 122

InnerException:

14

Hi Michel,

This issue is happening because you have hard deleted the stream. In addition, you do not need to delete the metadata stream.

If you are creating this projection after upgrading to v3.8.0, you need to create it with “Track emitted streams” enabled. When you delete this projection, check “Delete emitted streams”. After this, you should be able to create a new projection emitting to the same stream without any issue.

If you have upgraded after the projection was created, and therefore can’t enable tracking of the emitted streams, you would need to delete the projection, and then the emitted stream.

So in the second case, say you have a projection “test_projection” emitting to stream “emitted_stream.” You would delete “test_projection” from the UI, and then issue a curl command to delete “emitted_stream:”

curl.exe -v -X DELETE http://127.0.0.1:2113/streams/emitted_stream

15

I didn’t understand the difference between soft and hard delete. Now it all makes more sennse. Thx!