During a restart of the Event Store Server, the system projections got deleted.
Is it possible to add the system projections again?
Hi Kim,
What version of Event Store are you running?
Are other projections loading/showing in the projections tab?
Hi,
It is version 4.0.1.0
We only use the system projections and there are no projections showing.
I can see that the $projections-$all contains ProjectionDeleted events
Would you be able to provide us with the logs from your cluster when this happened?
If you don’t want to post them here, you can email them to hayley at geteventstore.com
Also, I presume that the projections were deleted without being explicitly told to do so, is that correct?
To answer your initial question, it is possible to create new system projections, however, simply adding them back will cause errors when the new projections try to write to the old streams.
If we can review your logs, we may be able to suggest a way to correct this.
Hi Hayley
You are correct, the logs were deleted without being explicitly told.
We have recovered from the situation by restoring an old backup where the projections still worked and publishing the missing events into it.
But this is quite time-consuming, so I am still looking for an easier way to restore the system projections, should this happen again.
I have sent you the logs by e-mail
Hi Kim,
Thank you for providing your logs.
Am I correct in saying that the second set of logs you provided is the restart that caused the projections to disappear?
How many ProjectionDeleted events were there in $projections-$all, and for which streams?
If you still happen to have the data for the node that was showing this behaviour, would you be able to send me the contents of the $projections-$all stream?
The reason I am asking for this is because after the restart, the projections in question were found after the restart, and I don’t see the projections being deleted later in the logs :
Found the following projections in $projections-$all. $by_event_type,$by_category,$users,$stream_by_category,$streams
However, when the projections were trying to start, they each ran into an error, and were disposed :
[PID:02868:016 2017.11.04 13:41:42.138 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Projections.Core.Messages.CoreProjectionManagementMessage+CreateAndPrepare in queued handler ‘Projections Master’.
System.NullReferenceException: Object reference not set to an instance of an object.
-
at EventStore.Projections.Core.Messages.Persisted.Commands.PersistedProjectionConfig…ctor(ProjectionConfig config) in c:\projects\eventstore\src\EventStore.Projections.Core\Messages\Persisted\Commands\PersistedProjectionConfig.cs:line 27*
-
at EventStore.Projections.Core.Services.Management.ProjectionManagerCommandWriter.Handle(CreateAndPrepare message) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Management\ProjectionManagerCommandWriter.cs:line 51*
-
at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message) in c:\projects\eventstore\src\EventStore.Core\Bus\MessageHandler.cs:line 32*
-
at EventStore.Core.Bus.InMemoryBus.Publish(Message message) in c:\projects\eventstore\src\EventStore.Core\Bus\InMemoryBus.cs:line 322*
-
at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o) in c:\projects\eventstore\src\EventStore.Core\Bus\QueuedHandlerMRES.cs:line 120*
[PID:02868:016 2017.11.04 13:41:42.372 DEBUG MultiStreamMessageWr] PROJECTIONS: Scheduling the writing of $dispose to $projections-$ce13e754-1966-4dba-859c-00fb7e121b8b. Current status of Writer: Busy: False
From this, it looks more like the projections in question failed to load, rather than were deleted. It would be great if we could verify this.
As an aside, you do not need the $users projection any more, and it can be safely deleted.
Hi Hayley
OK, I think I need to tell you the whole story, so here it is:
We restarted the Event Store due to maintenance on the server. The restart is in the logs, that I sent you.
The next day, we realised that the projections were not running correctly (a problem that we sometimes experience), so we restarted the event store to fix this.
After the second restart, the projections were no longer in the UI and after some time, we found the $ProjectionDeleted events in the $projections-$all stream
I have sent you the log from the second restart.
There were 5 $ProjectionDeleted events in the $projections-$all stream and I have sent you the content.
Hi Kim,
Thank you very much for helping us identify this issue. We have found the root cause for the projections being deleted.
The projections were previously set to delete themselves if they hadn’t responded after 5 minutes. I’ve submitted a PR to change that so that they are disabled instead.
Until this fix has been released, you can avoid having this happen again by trying to speed your system up so that you don’t run into issues where projections take so long to start.
The first step I’d recommend would be to upgrade to 4.0.2 or 4.0.3. There is a known bug in 4.0.1 that causes commit timeouts and idempotent writes, which may slow your system down.
Hi Hayley-Jean
Thank you for your support and suggestions. We will upgrade event store to the newest version and check if we can do something to speed up our system.
It is good to know, that the source of the problem is being handled.
However, I still have the initial question of whether it is possible to recover from a situation where the system projections are deleted?
Hi Kim,
Currently there is not an easy way to recover from a situation like this apart from restoring from a backup.
In order to recreate the projection, you would need to first delete any streams emitted by the projection.
If this were a custom projection emitting to a few known streams, you would be able to delete them and recreate the projection.
However, while possible, this would be harder and much more time-consuming with something like the system projections.
We just ran into a similar problem this weekend.
Our ES was restarted on Friday evening (we think due to a resource issue) and the projections (system and custom) were not started.
We just restarted the ES 5 minutes ago, and all our projections have returned.
So just so we are clear, if the Event Store is slow to restart, our projections might not get started?
Can you provide a log?
Hi Greg,
Attached are our log files from Friday.
The problem seems to be around 18:00 - 18:10.
It seems our Event Store was killed by the OS, then restarted automatically.
Steven
2017-11-24.zip (348 KB)