Hi,
I’m new to ES, going through the docs I’ve seen that events can be persisted to a backend storage cluster “k/v or column” rather than to disk. If I’m understanding that right it’s done through the manual setup of an event and not as a configuration option. What are some of the reasons/benefits to having a more complicated persistent setup for the storage? Is it to avoid hot backups and centrally archive data somewhere or are there other EventStore specific prod considerations that I’m overlooking?
I’ve not seen any reading on setting this up, or restoring from it in official docs or in blogs. How common is it to have this kind of set up and does it depends on usage? I really don’t fancy managing .chk files, the usage for our production ES instances will quickly have the kitchen sink thrown at it, standard practice here. I want to make sure it’s as durable as possible and we already have a few internal storage clusters.
Thanks
"I'm new to ES, going through the docs I've seen that events can be
persisted to a backend storage cluster "k/v or column" rather than to
disk."
I am unaware of this.
Errr, if I misunderstood then let me know.
“There are many other options available for backing up an Event Store database. For example it is possible to set up a durable subscription that would write all of the events to another storage mechanism such as a key/value or column store. These methods would require a manual set up for restoring back to a cluster group.”
Coupled with:
“The EventStore is designed to store a series of events, known as an event stream, to durable storage. Because of the simplicity of the model of an event stream, it can easily be persisted to durable storage using a large number of storage engines, including relational databases, plain files, NoSQL document databases, or even key-value stores. Herein is a brief summary that describes some of the guiding architectural principles and the underlying philosophy that governs the implementation of the EventStore and dictates the programming interface and calling contracts.”
Er…
"There are many other options available for backing up an Event Store
database. For example it is possible to set up a durable subscription
that would write all of the events to another storage mechanism such
as a key/value or column store. These methods would require a manual
set up for restoring back to a cluster group.""
This is about backups and how you can use a subscription and write the
data to another database such as a KV store or a column store
Your second quote is not from our documentation it is from
http://blog.jonathanoliver.com/cqrs-eventstore-v2-architectural-overview/
which is talking about a completely different library which currently
is known as neventstore
Right…
So, I’m new to ES, going through the docs I’ve seen that events can be persisted to a backend storage cluster “k/v or column” rather than to disk (for backups). If I’m understanding that right it’s done through the manual setup of an event and not as a configuration option. What are some of the reasons/benefits to having a more complicated persistent setup for the storage (backups)? Is it to avoid hot backups and centrally archive data somewhere or are there other EventStore specific prod considerations that I’m overlooking?
"What are some of the reasons/benefits to having a more complicated
persistent setup for the storage (backups)?"
Its more of an archiving than a backup. If putting into say cassandra
you can also still access them through their apis (eg they are indexed
not just files).
Event Store itself does not use these as a storage engine you are
writing an external projection that moves the data to them.
Hey,
In our case we have a Logstash plugin that uses the Atom feed to index
all events into ElasticSearch. All our reporting goes through that,
but we could potentially use that as a backup, if needed.
/Rickard
There are other options as well. As example you can just use a backup
http://docs.geteventstore.com/server/3.9.0/database-backup/ or use the
es-cli (commercial version). You can also use a remote node that is a
readonly real time subscriber to the main nodes (this would be a
manual switch over).
Excellent thank you so much.