0

We are looking to use Elastic Search in a major software system, which needs financial logs and records kept for many years. We are using Elastic Search, but are a little concerned whether Elastic is a workable solution for permanant storage. I've been told it can become corrupt potentially.

Are there solutions where the system can have multiple regions and can self repair if one node fails or gets corrupted? Do we have to write a system which backs up every single object written to Elastic Search to a separate database as a back up, and write a re-indexer, or is elastic search a viable solution, without a database back up?

Holland Risley
  • 123
  • 5

1 Answers1

1

My understanding is ElasticSearch is a search service, rather than a durable storage service. According to the FAQ you can take snapshots of your ElasticSearch service as a backup.

The feature page says

Durability: You can build data durability for your Amazon Elasticsearch cluster through automated and manual snapshots. You can use snapshots to recover your cluster or to create a new cluster with preloaded data. By default, the Amazon Elasticsearch Service will automatically create hourly snapshots of each domain and retain them for 14 days at no extra charge. These snapshots are stored in Amazon S3, which is designed for 99.999999999% (11 9’s) durability.

I can't find any other reference to durability, and there's no claim I can see that says it's means to be a durable data store.

I would be very careful putting any data on just ElasticSearch. I'd want it on S3 / EBS / EFS volume which are all durable services.

Tim
  • 30,383
  • 6
  • 47
  • 77