Hi,
We have almost 2TB of messages and want to archive them, because we are running out of space on that partition.
Our goal is to archive old messages (e.g. older that 3 month).
Saying ‘archive’ I mean to put them somewhere (e.g. S3) and in case we need to reprocess all messages just to ‘attach’ old messages to existing stream.
What options do we have?
pavel
Do a backup and always keep the original chunk files. They end in all 0s. Then, delete the streams you don’t want anymore, or use stream metadata. Finally, run a scavenge. This will actually cause more space to be taken up initially. But after this you can safely remove the earlier chunk versions.
http://docs.geteventstore.com/server/3.4.0/database-backup/
http://docs.geteventstore.com/dotnet-api/3.3.1/stream-metadata/
We have single huge stream and want to archive most of it, leaving only last 3 month of data
$maxAge will make them automatically eligible for scavenge. Just make
sure you have archived them away within that time frame.
Why is it one huge stream? You can use many streams still and do the
same (reading from $all)
By ‘archiving’ you mean backing up *.chk files?
In that case how to ‘restore’ those ‘archived’ messages, later if we need to?
Why is it one huge stream? You can use many streams still and do the
same (reading from $all)
We did not know better at that time.
Archiving as in putting them some place else
How to restore those archived messages?
depends how you are storing them etc in S3 etc
but maybe I am not understanding what you are asking for
For example:
In course of 1 year eventstore got 1,000,000 messages.
I have couple projections running.
I want to free up space and leave only last 2 month of messages.
I copy all *.chk and remaining files and directories to archiving disk.
Then I set $maxAge to 2 month and run scavenge.
After scavenge I will be left with 200,000 messages.
3 month and 300,000 messages later, I found out that I need new projection that needs all messages from beginning to calculate correctly. But now I have only 500,000 messages in eventstore and 1,000,000 archived.
I need to merge those archived messages back, so I would have 1,300,000 messages in total again.
Ah ok so using chunks. There is no easy way of doing that. Instead you
would write a from all subscriber then a little script to put things
back.
Hi Greg,
Maybe you have any example how you would achieve this?
2016 m. sausis 25 d., pirmadienis 18:50:24 UTC+2, Greg Young rašė:
How do do an $all subscription? SubscribeToAllFrom(...)
Then in your projection you would have to read from the archive store
then switch to the "real time' store
How to restore those archived messages?
On Friday, January 22, 2016 at 1:59:37 PM UTC+2, Greg Young wrote:Archiving as in putting them some place else
> By 'archiving' you mean backing up *.chk files?
>
> In that case how to 'restore' those 'archived' messages, later if we need
> to?
>
>> Why is it one huge stream? You can use many streams still and do the
>> same (reading from $all)
>
> We did not know better at that time.
>
>
>>
>> $maxAge will make them automatically eligible for scavenge. Just make
>> sure you have archived them away within that time frame.
>>
>> Why is it one huge stream? You can use many streams still and do the
>> same (reading from $all)
>>
>> > We have single huge stream and want to archive most of it, leaving only
>> > last
>> > 3 month of data
>> >
>> >>
>> >> Do a backup and always keep the original chunk files. They end in all
>> >> 0s.
>> >> Then, delete the streams you don't want anymore, or use stream
>> >> metadata.
>> >> Finally, run a scavenge. This will actually cause *more* space to be
>> >> taken
>> >> up initially. But after this you can safely remove the earlier chunk
>> >> versions.
>> >>
>> >> http://docs.geteventstore.com/server/3.4.0/database-backup/
>> >>
>> >> http://docs.geteventstore.com/dotnet-api/3.3.1/stream-metadata/
>> >>
>> >>>
>> >>> Hi,
>> >>> We have almost 2TB of messages and want to archive them, because we
>> >>> are
>> >>> running out of space on that partition.
>> >>> Our goal is to archive old messages (e.g. older that 3 month).
>> >>> Saying 'archive' I mean to put them somewhere (e.g. S3) and in case we
>> >>> need to reprocess all messages just to 'attach' old messages to
>> >>> existing
>> >>> stream.
>> >>> What options do we have?
>> >>>
>> >>> pavel
>> >>>
>> >>> --
>> >>> You received this message because you are subscribed to the Google
>> >>> Groups
>> >>> "Event Store" group.
>> >>> To unsubscribe from this group and stop receiving emails from it, send
>> >>> an
>> >>> email to [email protected].
>> >>> For more options, visit https://groups.google.com/d/optout.
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Sent from my regular computer
>> >> http://twitter.com/thefringeninja
>> >> http://www.thefringeninja.com/
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Event Store" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to [email protected].
>> > For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> Studying for the Turing test
>
> --
> You received this message because you are subscribed to the Google Groups
> "Event Store" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> For more options, visit https://groups.google.com/d/optout.
--
Studying for the Turing test