Upgrading from 1.x to 2.x with Projections

EventStoreDB
#1

Hi All,

As you've probably spotted now, version 2.0.0 of the Event Store was released a few days ago. In version 2.0.0, projections are now considered "beta" - that means the API is highly likely to be stable, but the documentation isn't quite up to scratch yet :slight_smile:

If you've used projections in a 1.x database and want to use that database with 2.0.0, you'll need to run an upgrade tool, which is available here: http://download.geteventstore.com, as the stream naming has changed for projections definition (they're now consistent with other system streams in that they are prefixed with a $ symbol). To use the upgrade tool, start the 2.0.0 Event Store with the 1.x database (using the --db command line parameter), and point the upgrade tool at the server - if you run it with the --upgrade flag, the database will be upgraded, otherwise the tool will log what it will do if you run it with the --upgrade flag. Obviously depending on the criticality of your data it might be a good idea to back up the database first!

Thanks,

James

#2

After running the upgrade tool, I get an error saying:

[PID:05940:019 2013.07.25 13:02:14.972 ERROR QueuedHandlerMRES ] Error while processing message EventStore.Core.Messages.ClientMessage+ReadStreamEventsBackwardCompleted in queued handler ‘Projection Core #0’.
System.NullReferenceException: Object reference not set to an instance of an object.
at EventStore.Projections.Core.Services.Processing.TransactionFilePositionTagger.AdjustTag(CheckpointTag tag) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Projections.Core\Services\Processing\TransactionFilePositionTagger.cs:line 43
at EventStore.Projections.Core.Services.Processing.CheckpointTagVersion.AdjustBy(PositionTagger tagger, ProjectionVersion version) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Projections.Core\Services\Processing\CheckpointTagExtensions.cs:line 52
at EventStore.Projections.Core.Services.Processing.EmittedStream.CollectAlreadyCommittedEvents(ReadStreamEventsBackwardCompleted message, CheckpointTag upTo) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Projections.Core\Services\Processing\EmittedStream.cs:line 294
at EventStore.Projections.Core.Services.Processing.EmittedStream.ReadStreamEventsBackwardCompleted(ReadStreamEventsBackwardCompleted message, CheckpointTag upTo) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Projections.Core\Services\Processing\EmittedStream.cs:line 272
at EventStore.Core.Messaging.RequestResponseDispatcher2.Handle(TResponse message) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Core\Messaging\RequestResponseDispatcher.cs:line 85 at EventStore.Core.Bus.MessageHandler1.TryHandle(Message message) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Core\Bus\MessageHandler.cs:line 59
at EventStore.Core.Bus.InMemoryBus.Publish(Message message) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Core\Bus\InMemoryBus.cs:line 348
at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o) in c:\BuildAgent1\work\oss\windows\releasebuilds\src\EventStore\EventStore.Core\Bus\QueuedHandlerMRES.cs:line 139

I backed up my data before I ran the tool, and when I retry running the tool, the error is consistent. Also the projection called $streams gets stuck in a state called “Running/PausedRequest” and at 0.1% done.

If I try to restart the EventStore, it gets stuck at the same error.

I have no idea how to get pass this?

#3

Roy,

is it a " fromStreams(’…’, ‘…’) " projection?

-yuriy

#4

I believe this was one of the built-in projections, that was auto-created at some point.

Where can I find the answer to your question?

#5

Roy,

thank you. auto-created answers my question.

-yuriy

#6

Roy,

could you send me what the following query returns to my e-mail? (assuming there are no confidential data).

Or just entries like


 "linkMetaData": "{\r\n  \"$v\": \"1:-1:1:2\",\r\n  \"$c\": 7951,\r\n  \"$p\": 5544,\r\n  \"$causedBy\": \"d6806482-6166-46e6-b642-e4649d8bad6e\"\r\n}",



-yuriy

http://127.0.0.1:2113/streams/%24streams?embed=tryharder&format=json

#7

Yuriry,

Here you go. The data is not confidential at this level.

-Roy-

$streams.json (25.2 KB)

#8

Roy,

I found why the projection fails. This is due to the incorrectly handled old “$stream-created-implicit” event on the output stream.

I’m fixing the problem and I’ll let you know when I push. We will make binaries available as well after either as nightly builds or just one more build with fixes.

-yuriy

#9

Roy,

I’ve pushed a fix for the problem to the dev branch. We will make binaries available just a little later.

You can still build the dev branch yourself.

-yuriy

#10

I could not make further tests earlier and stumbled on the same problem today…

I was wondering if it was due to my PS script or something else… saw the errors at startup, and came here to report…

Just to see that the problem has already been solved.

You rock guys :smiley:

jeremie / thinkbeforecoding

#11

I just tried with the dev branch, and it works - using the PS script to upgrade!

thx.

jeremie / thinkbeforecoding

#12

I just noticed this error on startup (everything seems to work though):

[06040,15,10:08:09.370] Error while processing message EventStore.Core.Messages.ClientMessage+ReadStreamEventsBackwardCompleted in queued handler ‘Projection Core #0’.
An event emitted in recovery differ from the originally emitted event.

Existing(’$>’, ‘C:81546/P:81284’). New(’$>’, ‘C:847/P:143’)

Is it expected ?

#13

Jérémie,

is it a standard predefined projection failing or your custom JavaScript projection?

The error message you see is the internal check validating that the projection produces the same output after the restart from the checkpoint as it produced while has been running previously.

When you upgrade from the v1 projections run from the beginning ignoring all output that is already written to streams. I understand that it causes problems if the projection was edited since it was originally created and I’m now enhancing the projection upgrade tool to upgrade old projection checkpoints. It will likely this problem in the most of cases.

-yuriy

#14

Jérémie,

I updated projection-upgrade-tool and it now upgrades projection checkpoints as well.

However, before upgrading the DB you need to STOP all the projections with the EventStore v1x. It forces projections to write a checkpoint with writing any other output after the checkpoint position.

So, the process is:

  1. Stop all the projection in v1x

  2. Move the DB to v2x (ie. run the DB with EventStore v2.0)

  3. Run the projection upgrade tool with --upgrade

  4. Restart the ES

  5. Enable projections one-by-one watching for any errors in the log files.

The projection upgrade tool is at: https://github.com/EventStore/EventStore/tree/dev/src/EventStore/Tools/EventStore.UpgradeProjections

kind regards

yuyriy

#15

Ok I’ll try this. And see if I can change this in the PS script also

jeremie / thinkbeforecoding

#16

I’ll post some new binaries of the .exe upgrade tool as well now.

Cheers,

James

#17

By the way, how can I read/write event metadata using the http api ?

jeremie / thinkbeforecoding

#18

https://github.com/EventStore/EventStore/wiki/Stream-Metadata-(HTTP)

There is a rel link on the head of the stream that points to metadata

    {
      "uri": "http://127.0.0.1:2113/streams/%24users/metadata",
      "relation": "metadata"
    }

It is recommended that you read the rel link to get the location.

Cheers,

Greg

#19

Yep, found it.

It was more about event metadata than Stream data…

But it was because I requested events with ?format=json which only returns the data…

When removing it, the whole event description is returned.

thx

jeremie / thinkbeforecoding

#20

New pull request for the PowerShell conversion script, this time with checkpoint upgrade.

cheers

jeremie / thinkbeforecoding