To quote the documentation:
The index snapshot process is incremental. In the process of making the index snapshot Elasticsearch analyses the list of the index files that are already stored in the repository and copies only files that were created or changed since the last snapshot. That allows multiple snapshots to be preserved in the repository in a compact form.
Having spent a career in backup and disaster-recovery planning, I understand your concern. Like all backups, dealing with this strategy requires a bit of analysis. Some things to consider:
- Data turnover rate. If you only ever keep (n) weeks of data in the index, you can tolerate some backup strategies. If the index is an accumulator table where nothing gets deleted and it gets bigger over time, different styles are worth it.
- Growth rate. Like turnover, how big it gets over time.
- Backup storage constraints. Pretty obvious. If it's continual-increments, and you have a high turnover rate, your backup repo will contain a lot of things it doesn't need.
- Backup I/O constraints. While the operation is non-blocking, it is not zero-resource. Incrementals are faster than fulls, but fulls may be needed for other reasons.
The snapshot procedure is a continual-incremental strategy. For an accumulator table (no turnover), it is sufficient to take one full and keep the incrementals forever. Except...
During snapshot initialization, information about all previous snapshots is loaded into the memory, which means that in large repositories it may take several seconds (or even minutes) for this command to return even if the wait_for_completion parameter is set to false.
This is your incentive to not actually keep everything. An hourly snapshot history going back two years will take up a lot of heap. Happily, they do have a DELETE functionality to prune this history.
If you do have a high turnover rate, definitely plan on issuing DELETE to older snapshots over time. Per the documentation, the ES snapshot process is smart enough to correctly handle the data purge process. Your GFS policy of snapshots can definitely be done, even with a 'continual-incremental' backup system. Think of it like a deduplicating backup-to-disk system; you don't purge the dedupe cluster every 2 months, you let the backup system reap out the blocks/files no longer needed.
If you need to offsite this stuff, you can copy the snapshot-repo itself and do the usual media rotation on it. The es-repo for snapshots is for hot backup/restore. If you need to load an older one for some reason, you should be able to restore the offline copy over the hot copy and call a restore from the ES API and it will load from the data you put into the snapshot repo.
