31

How can zfs pools be continuously/incrementally backed up offsite?

I recognise the send/receive over ssh is one method however that involves having to manage snapshots manually.

There are some tools I have found however most are no longer supported.

The one tool which looks promising is https://github.com/jimsalterjrs/sanoid however I am worried that non-widely known tool may do more harm then good in that it may corrupt/delete data.

How are continuous/incremental zfs backups performed?

Greg
  • 1,557
  • 5
  • 24
  • 35
  • 2
    I'll respond a little later, but I have a solution that performs this type of replication every 15 seconds from primary ZFS server to secondary. – ewwhite Apr 04 '17 at 16:11

3 Answers3

50

ZFS is an incredible filesystem and solves many of my local and shared data storage needs.

While, I do like the idea of clustered ZFS wherever possible, sometimes it's not practical, or I need some geographical separation of storage nodes.

One of the use cases I have is for high-performance replicated storage on Linux application servers. For example, I support a legacy software product that benefits from low-latency NVMe SSD drives for its data. The application has an application-level mirroring option that can replicate to a secondary server, but it's often inaccurate and is a 10-minute RPO.

I've solved this problem by having a secondary server (also running ZFS on similar or dissimilar hardware) that can be local, remote or both. By combining the three utilities detailed below, I've crafted a replication solution that gives me continuous replication, deep snapshot retention and flexible failover options.

zfs-auto-snapshot - https://github.com/zfsonlinux/zfs-auto-snapshot

Just a handy tool to enable periodic ZFS filesystem level snapshots. I typically run with the following schedule on production volumes:

# /etc/cron.d/zfs-auto-snapshot

PATH="/usr/bin:/bin:/usr/sbin:/sbin"

*/5 * * * * root /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=24 //
00 * * * * root /sbin/zfs-auto-snapshot -q -g --label=hourly --keep=24 //
59 23 * * * root /sbin/zfs-auto-snapshot -q -g --label=daily --keep=14 //
59 23 * * 0 root /sbin/zfs-auto-snapshot -q -g --label=weekly --keep=4 //
00 00 1 * * root /sbin/zfs-auto-snapshot -q -g --label=monthly --keep=4 //

Syncoid (Sanoid) - https://github.com/jimsalterjrs/sanoid

This program can run ad-hoc snap/replication of a ZFS filesystem to a secondary target. I only use the syncoid portion of the product.

Assuming server1 and server2, simple command run from server2 to pull data from server1:

#!/bin/bash

/usr/local/bin/syncoid root@server1:vol1/data vol2/data

exit $?

Monit - https://mmonit.com/monit/

Monit is an extremely flexible job scheduler and execution manager. By default, it works on a 30-second interval, but I modify the config to use a 15-second base time cycle.

An example config that runs the above replication script every 15 seconds (1 cycle)

check program storagesync with path /usr/local/bin/run_storagesync.sh
        every 1 cycles
        if status != 0 then alert

This is simple to automate and add via configuration management. By wrapping the execution of the snapshot/replication in Monit, you get centralized status, job control and alerting (email, SNMP, custom script).

The result is that I have servers that have multiple months of monthly snapshots and many points of rollback and retention within: https://pastebin.com/zuNzgi0G - Plus, a continuous rolling 15-second atomic replica:

# monit status

Program 'storagesync'
  status                            Status ok
  monitoring status                 Monitored
  last started                      Wed, 05 Apr 2017 05:37:59
  last exit value                   0
  data collected                    Wed, 05 Apr 2017 05:37:59
.
.
.
Program 'storagesync'
  status                            Status ok
  monitoring status                 Monitored
  last started                      Wed, 05 Apr 2017 05:38:59
  last exit value                   0
  data collected                    Wed, 05 Apr 2017 05:38:59
ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • 4
    Thank you for posting, your answer is phenomenal and exactly what I was looking for (from latency to monitoring the process). Also reading https://github.com/ewwhite/zfs-ha/wiki and I am thoroughly impressed. Thank you once again :) – Greg Apr 05 '17 at 12:57
  • 1
    great post. Thanks. I gave you the credits on my puppet module: https://forge.puppet.com/maxadamo/zfs_nas/readme – maxadamo Mar 17 '20 at 18:43
  • @maxadamo Tell us more about it. – ewwhite Mar 17 '20 at 20:02
  • 1
    @ewwhite the module creates exactly the same setup that you have described in the post. I use puppet. It implemente IPv6 (optional).... it's still under development: I need to write unit test for it. – maxadamo Mar 20 '20 at 20:18
8

You have two different ways you can do that:

  1. The traditional, filesystem-agnostic way that is/was used for the last decades, with tools like rsync or Bacula. There you have tested and (hopefully) stable, large software that can be customized for huge deployments and can be used even if you switch away from ZFS
  2. One of the tools that leverage ZFS send/recv. This can either be your own solution, a script or extended script from the various ones on Github et al., or more feature-rich tools like Sanoid or ZnapZend (send/recv with mbuffer support and retention plans). In this case you will most likely not find any big, "enterprisey" (in the negative sense) solutions, but tools that do just the single task and can be combined with other tools to cater to your specific setup.

In general, I would only trust a tool whose source code is available, and I would keep it as simple as possible. If using send/recv, you don't have to manage much, you just have to delete snapshot n-1 on the local side when transmission and establishment of snapshot n on the remote side was successful.

You can split your transport any way you like, it can even be async (snapshots do not have to be received immediately), if you just keep the iron rule that you can only send a diff between the local current/new and local previous snapshot, and that the local previous snapshot is the most recent one on the remote side (until the backup finishes and everything is reset).

Now that I think of it, you could probably encode that in a state machine and then be sure that no unforeseen cases can slip through.

user121391
  • 2,452
  • 12
  • 31
  • I don't see how an `rsync`-based solution would scale to continuously replicating a large enterprise-scale filesystem. Changes could happen faster than `rsync` could discover them. – Andrew Henle Apr 05 '17 at 10:45
  • 2
    @AndrewHenle I would not advocate for it either, I just wanted to present it, because the question did not specify the scope/size of data or the timeframe. So in case of infrequent action it might be a possibility if it should be filesystem-agnostic. Of course, you would lose the nice block-level deltas... – user121391 Apr 05 '17 at 11:10
  • @user121391 Completely agree with you regarding opensource being the way to go. Thank you for your detailed response. – Greg Apr 05 '17 at 12:29
  • @Dave just as I'm typing... – ewwhite Apr 05 '17 at 12:30
  • @ewwhite I was looking forward to your answer and will really appreciate if you still contribute your response. – Greg Apr 05 '17 at 12:31
  • 1
    highly recommend znapzend – Trent Lloyd Apr 06 '17 at 07:32
0

Also checkout ZFS autobackup. Its easy to use and very powerfull. It has more features compared to the other solutions mentioned: https://github.com/psy0rz/zfs_autobackup

Psy0rz
  • 1
  • 2