2

We have two large storage servers (+100TB), one runs on ZFS the other one runs XFS, we intent to use the XFS as our work server and use the ZFS as the backup server (snapshots <3). Now the problem is keeping these beasts in sync ... (sync as in daily synced)

The easiest option is to use rsync, but sadly the directory structure is deep, and full of hard links, all over the place. So this means we would need to do a "global" scan which would take ages... On top of that, most of the data is created and never modified. So rsync might just not be the way to go.

I looked into inotify, which seems relatively cheap and since we only wanne sync on a daily base, would be able to offload to a good time... sadly, if we only look to the created files we would copy hard links as data which would double the amount of storage used in our backup ... (basically there is no way to do the -H check from rsync)

The only option left I could think off would be to reorganize our storage to use a date based directory, sadly moving around that much data is not something we would prefer ...

Are there other options ?

For reference :

  • Server with XFS has a raid controller (no JBOD option), and SATA disks (WD RE). 32Gb RAM
  • Server with ZFS has a HBA controller and SAS disks. 126Gb RAM

When I reference ZFS as beeing slow, I see 'ls' taking seconds...

SvennD
  • 739
  • 5
  • 18
  • 1
    Why not migrate the work server to ZFS as well. Then you can use the native ZFS send/receive mechanisms. – EEAA Aug 31 '16 at 16:30
  • Well we tried that, but ZFS with this amount of (meta-) data is just to slow on our setup... ZFS send/receive also looks like a full copy which is even more data over the line no ? – SvennD Aug 31 '16 at 19:29
  • No, with snapshots, send/receive is only changed blocks. – EEAA Aug 31 '16 at 19:40
  • Oh, I should test this feature better then, while I can't / won't go back to a double ZFS in this particular case, its handy to know! Thanks. – SvennD Aug 31 '16 at 19:45
  • Why are the server specifications so mismatched? – ewwhite Sep 01 '16 at 10:36
  • We "inherited" the raid controller machine, and wanted to move to ZFS, hence the ZFS server is a bit bulkier. (also newer) – SvennD Sep 01 '16 at 12:05

3 Answers3

6

You really should be using ZFS on both sides, coupled with a block-level snapshot/replication routine like Sanoid.

Without that, you're stuck with file-based operations and the pain of the rsync file scan.

ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • Excellent answer as always ewwhite, sadly ZFS is just to slow, perhaps its the workload or more likely the fact that we don't know anything about tuning ZFS properly... – SvennD Aug 31 '16 at 19:36
  • Then the question should be about ZFS tuning, which I can help with :) I've replaced all of my XFS production workloads with ZFS on Linux over the past 4 years. – ewwhite Aug 31 '16 at 20:14
  • @SvennD What hardware are you using? Especially disk controller(s) and disk types on your "slow" system(s)? Because `rsync` is going to be a lot slower than ZFS send/receive with complex file systems - `rsync` has to *find* what's changed while ZFS already *has the difference set* in the snapshot. – Andrew Henle Sep 01 '16 at 08:49
  • @AndrewHenle I added the hardware for reference, however, I doubt I can 'tune' ZFS enough to sell it again to he colleagues... – SvennD Sep 01 '16 at 08:56
  • I understand that ZFS gets pretty slow if you don't have enough RAM. That's a fairly easy to fix problem now (at least it is easier today than, say, in the 32-bit days) – TomOnTime Sep 01 '16 at 17:21
  • Is "ls" slow or is "ls -l" slow? if "ls -l" is slow, I'd suspect uid->name lookups before anything else. ZFS is incredibly fast. – TomOnTime Sep 01 '16 at 17:27
  • 1
    I'm confused. It sounds like you are using the faster hardware for backups... something that runs once a day. Use the faster hardware as the regular server and the older hardware for backups. – TomOnTime Sep 01 '16 at 17:39
  • @SvennD Right. This is not a technical problem anymore. ZFS is more than capable of performing well (with tuning and good design). It also solves the replication issue with ease. This is more of a political and budgeting situation, since your resources aren't allocated well. – ewwhite Sep 01 '16 at 17:41
  • @ewwhite I think it is, we accepted the fact that we don't have the technical knowledge to run ZFS fine tuned, XFS works perfectly out-of-the box, still being able to take advantage of ZFS features and security is a big plus... – SvennD Sep 01 '16 at 19:51
  • But your backup server is much more robust than the production server. That's the real issue with this. – ewwhite Sep 01 '16 at 19:52
  • Well my hope is to get more juice out of ZFS some time in the future, but at this point getting the data in 2 servers is priority over getting better knowledge of ZFS... It also makes me sleep better when i'm tuning ZFS to know that our live server isn't going to break cause of me ... Unlike where you'd expect this setup, we are a small company with a very tight budget... aka academics. – SvennD Sep 01 '16 at 19:56
  • If you use rsync to keep them in sync and appease the political requirements, switching to the ZFS server as the primary will be easy in the future because the data is already there. – TomOnTime Sep 02 '16 at 12:51
4

How fast is "fast enough"?

You are doing this once per day, so I'd suspect that if it takes 2-3 hours, that is sufficient.

In that case, "rsync -avP" should be all you need. The newest versions handle large directories, deep hierarchies, and doesn't require as much RAM as older versions.

If no files changed, "rsync -a" will be as fast as "ls -lR". You can't get any faster than "ls -lR" because it does an lstat() of every file on the system.

Benchmark "ls -lR" and "rsync -a". If they are slower than you think they should be, look at https://serverfault.com/a/746868/6472 for advice.

If you need something faster than the "ls -lR" benchmark, you'll have to either write something that uses "inotify", or use some kind of block-based system. In particular, using ZFS on both systems would enable you to use the snapshot export/import system that ZFS has built-in.

Community
  • 1
TomOnTime
  • 7,567
  • 6
  • 28
  • 51
  • You double posted, but thanks for your input, I check inotify more closely and all-in-all it seems like a half ass solution to this problem. For the moment I think I'm going to try just rsyncing it until a better solution is setup (perhaps the popular ZFS/ZFS setup suggested here) – SvennD Sep 01 '16 at 20:01
  • @SvennD *I check inotify more closely and all-in-all it seems like a half ass solution to this problem.* I'd venture to say it's not even that good. It's event-driven, as TomOnTime noted. Which means if you base your backups off using `inotify` and miss an event, that data doesn't get backed up. So to ensure data integrity you have to do a state-based backup anyway, again as TomOnTime noted. So you really don't gain much. – Andrew Henle Sep 02 '16 at 12:48
  • We went with TomOnTime's rsync, as seemingly our tests where no representation of the real workload, when we sync only once a month the changes take up way over 10h but, syncing them daily/hourly seems to take way less time then anticipated, in the future, a ZFS<->ZFS might be a better solution, but at this point its not yet practical in our use case. Thanks all ! – SvennD Sep 03 '16 at 10:10
2

I would adopt a 2-part strategy... and at the end I'll suggest a 3rd part that is optional.

Part 1: Use inotify: Write a program that uses inotify to log which files were created, deleted, and modified. Write another program that reads the log, removes any duplicates, and does backups of those files (and deletes the deleted files). This won't be easy. Programming inotify is complex. The log can't be a simple text file since filenames can include newlines. If the system crashes while the log is being written, you'll need to be able to deal with partially-written filenames.

Part 2: Weekly rsync's just in case. Every few days do a "rsync -a --delete" to catch any files that were missed. The solution in part 1 is imperfect. If your program doesn't keep up with inotify, it may miss some files. If the machine reboots, the log of created/deleted/modified files may lose the most recent items. Bugs and other issues may also result in missing some files.

Optional Part 3: After you have this running for a few weeks and you've gotten all the bugs worked out, you'll still find that rsync is occasionally finding files that were missed. I promise you that will happen. inotify is "best effort". So, at this point you you'll realize that maintaining the code in Part 1 and Part 2 is twice as much work as you expected. To solve this problem, throw away the code you wrote in Part 1, because rsync is all you really needed in the first place.

TomOnTime
  • 7,567
  • 6
  • 28
  • 51