Projection Faulted: An event emitted in recovery differ from the originally emitted event.

EventStoreDB
#1

Hi there,

we are running Event Store in version 3.0.0 final on Windows and experienced the following problem: the VM our Event Store is running on was unexpectedly killed from the outside (VMWare). Now after the server restart we have a projection that is in the faulted state, showing the following error message:

An event emitted in recovery differ from the originally emitted event. Existing(’$>’, ‘C:613381856/P:613381856’). New(’$>’, ‘C:613378130/P:613378130’)

I suspect the writing to the checkpoint was interrupted, so the projection finds an invalid position when trying to start. Is there an easy way we can recover from this situation, or do I have to reset the projection?

In the case of a reset I would have to delete all previously emitted events of the projection, which is possible but still quite an effort.

Thanks, Regards

Sebastian

#2

Can we grab your db to take a look at where it is?

#3

As the DB contains user specific data (like names, emails of our company’s employees) this is not so easy. Last time I only got the permission from management to do a remote debugging session, so I could make sure the data was deleted on your side afterwards ;-D
If this is a possibility for you I could create an issue on Zendesk first and then we could talk about the details.

Regards,

Sebastian

#4

Probably easier for us to just recreate with dummy data :slight_smile: What does
your projection look like so we can begin on that

#5

I thought so too :wink: This is our projection:

fromAll()
.when({
$any: function (s, e) {
if (e.metadata && e.metadata.$correlationId) {
linkTo(‘company.corid-’ + e.metadata.$correlationId, e);
}
},
})

``

We add a metadata field “$correlationId” containing a Guid, to all of our events, so whenever a new conversation starts, we create a new stream "company.corid-05c6c7ea-3331-4948-b79a-f2b5ad0d4ce1’ that contains all events of the conversation.

And here is the error log from ES during / after the VM restart (which happend at ~ 9:16:01 UTC).

[PID:06484:025 2014.10.03 09:16:01.518 ERROR MonitoringService ] Error while collecting stats
System.UnauthorizedAccessException: Access to the registry key ‘230’ is denied.
at Microsoft.Win32.RegistryKey.Win32Error(Int32 errorCode, String str)
at Microsoft.Win32.RegistryKey.InternalGetValue(String name, Object defaultValue, Boolean doNotExpand, Boolean checkSecurity)
at Microsoft.Win32.RegistryKey.GetValue(String name)
at System.Diagnostics.PerformanceMonitor.GetData(String item)
at System.Diagnostics.PerformanceCounterLib.GetPerformanceData(String item)
at System.Diagnostics.PerformanceCounterLib.GetCategorySample(String machine, String category)
at System.Diagnostics.PerformanceCounter.NextSample()
at System.Diagnostics.PerformanceCounter.NextValue()
at EventStore.Core.Services.Monitoring.SystemStatsHelper.GetPerfCounterInformation(Dictionary2 stats, Int32 count) in c:\EventStore\src\EventStore.Core\Services\Monitoring\SystemStatsHelper.cs:line 123 at EventStore.Core.Services.Monitoring.SystemStatsHelper.GetSystemStats() in c:\EventStore\src\EventStore.Core\Services\Monitoring\SystemStatsHelper.cs:line 42 at EventStore.Core.Services.Monitoring.MonitoringService.CollectStats() in c:\EventStore\src\EventStore.Core\Services\Monitoring\MonitoringService.cs:line 127 [PID:06484:025 2014.10.03 09:16:02.595 ERROR MonitoringService ] Error while collecting stats System.UnauthorizedAccessException: Access to the registry key '230' is denied. at Microsoft.Win32.RegistryKey.Win32Error(Int32 errorCode, String str) at Microsoft.Win32.RegistryKey.InternalGetValue(String name, Object defaultValue, Boolean doNotExpand, Boolean checkSecurity) at Microsoft.Win32.RegistryKey.GetValue(String name) at System.Diagnostics.PerformanceMonitor.GetData(String item) at System.Diagnostics.PerformanceCounterLib.GetPerformanceData(String item) at System.Diagnostics.PerformanceCounterLib.GetCategorySample(String machine, String category) at System.Diagnostics.PerformanceCounter.NextSample() at System.Diagnostics.PerformanceCounter.NextValue() at EventStore.Core.Services.Monitoring.SystemStatsHelper.GetPerfCounterInformation(Dictionary2 stats, Int32 count) in c:\EventStore\src\EventStore.Core\Services\Monitoring\SystemStatsHelper.cs:line 123
at EventStore.Core.Services.Monitoring.SystemStatsHelper.GetSystemStats() in c:\EventStore\src\EventStore.Core\Services\Monitoring\SystemStatsHelper.cs:line 42
at EventStore.Core.Services.Monitoring.MonitoringService.CollectStats() in c:\EventStore\src\EventStore.Core\Services\Monitoring\MonitoringService.cs:line 127
[PID:01548:013 2014.10.03 09:19:07.520 ERROR ProjectionManager ] The ‘$by_correlationId’ projection faulted due to ‘An event emitted in recovery differ from the originally emitted event. Existing(’$>’, ‘C:613381856/P:613381856’). New(’$>’, ‘C:613378130/P:613378130’)’
[PID:01544:014 2014.10.03 09:29:51.094 ERROR ProjectionManager ] The ‘$by_correlationId’ projection faulted due to ‘An event emitted in recovery differ from the originally emitted event. Existing(’$>’, ‘C:615812407/P:615812407’). New(’$>’, ‘C:615809124/P:615809124’)’
[PID:01544:014 2014.10.03 12:05:51.115 ERROR ProjectionManager ] The ‘$by_correlationId’ projection faulted due to ‘An event emitted in recovery differ from the originally emitted event. Existing(’$>’, ‘C:613381856/P:613381856’). New(’$>’, ‘C:613378130/P:613378130’)’
[PID:01788:011 2014.10.03 12:10:17.234 ERROR ProjectionManager ] The ‘$by_correlationId’ projection faulted due to ‘An event emitted in recovery differ from the originally emitted event. Existing(’$>’, ‘C:613381856/P:613381856’). New(’$>’, ‘C:613378130/P:613378130’)’

``

Hope that helps.

Regards

Sebastian

#6

moving off list

#7

Hi,

We’re getting the same error on 3.0.1 on Windows. Is there some general resolution or the db will need to be looked at as with Sebastian?

Thanks,

Zurab

#8

We are experiencing the same issue on (version 3.2)
Any solution to this?

From what I can tell, our projection has stopped running as well.

AllSales - Faulted (Enabled)

An event emitted in recovery differs from the originally emitted event. Existing(’$>’, ‘C:243467370/P:243467370’). New(’$>’, ‘C:243350716/P:243350716’)

Here is our Projection:

fromAll()

.when({

‘$any’ : function (s, e) {

linkTo(‘AllSalesCompany’, e);

}

});

#9

We got the same today (after we started from a fresh install)

The only pattern I can see, is this issues appears after the Event Store Process has been restarted.

The first time, it was stopped by hand, and restarted, and the second, the Event Store crashed.

We are not sure how to rectify this problem just now, but I am assuming we will have to Reset our Projection (how this would work in a Production environment I have no idea).

#10

This is a known issue in projections. There is a card up to resolve it.

Resetting the projection will resolve it. You can also manually edit
the checkpoint.

Projections are at this point in beta (likely to come out later this
fall) and are not intended to be used in critical production
environments.

#11

It still happens in ES4 and there is no documented way to resolve it.

#12

I’ve also experienced this today on EventStore 4.0.1 (3 node cluster) with the built in projections:

An event emitted in recovery for stream $streams differs from the originally emitted event. Existing(’$>’, ‘C:9722680354/P:9722680354’). New(’$>’, ‘C:9722679820/P:9722679820’)

#13

We have been hard at work at root causing and resolving this particular issue. It’s a high priority item on the backlog.
If you are willing to share logs, possibly even the DB that would be most useful. If there is sensitive information in your logs or database, please don’t hesitate to send them over to pieter at geteventstore dot com and i’ll forward to the relevant members of the team.

The way we recommend that customers currently resolve this issue is to either

  1. Reset the projection (recommended, events will be re-emitted)

  2. Set the projection checkpoint to the existing position which in this case would be 9722680354/P:9722680354

#14

How can we set a checkpoint for system projections?

#15

The issue has been resolved and the bug fix will be out in 4.0.3

#16

We were hit hard by the issue tonight. Here is how we solved it:

  1. Create File “set-by_category-checkpoint.txt”

[

{

"eventId": "43770587-AB58-4946-B1F5-2C94AACA846E",

"eventType": "$ProjectionCheckpoint",

"data": {

},

"metadata": {

  "$v": "3:-1:5:3",

  "$c": 94924010595,

  "$p": 94924010595

}

}

]

  1. Fire event

curl -i -d @set-by_category-checkpoint.txt “http://${HOST}:${PORT}/streams/%24projections-%24by_category-checkpoint” --user ‘admin:${YOURPASSWORD}’ -H “Content-Type:application/vnd.eventstore.events+json”

  1. I then stopped and started the projection

Hope this helps.

Cheers

Siamak

#17

I got the same error today, in 4.0.3.0

#18

Hi Ndangthieu,

How long have you been running 4.0.3, and were you running an old version before this?

We want to determine if this is a new occurrence of the issue or if it’s a remnant from the previous one.

Also, could you provide us with some logs showing the issue? If you don’t want to post them here, you can email them to hayley.campbell at eventstore.org

Thanks

#19

Hi Harley,
It was a fresh 4.0.3.0 installation, using a docker image in my development environment. My PC restarted for some reason and I encountered that error. The event numbers in the error message were 0/2.

I’m so sorry that I’ve just re-installed the 4.1 so I lost all the information.

Regards,

Thieu

#20

Hi Harley,
I encountered the same error in my UAT cluster just now. I’ll email you the log.