7

I've got a server with a LOT of small files -- many millions files, and over 1.5 TB of data. I need a decent backup strategy.

  • Any filesystem-based backup takes too long -- just enumerating which files need to be copied takes a day.

  • Acronis can do a disk image in 24 hours, but fails when it tries to do a differential backup the next day.

  • DFS-R won't replicate a volume with this many files.

I'm starting to look at Double Take, which seems to be able to do continuous replication. Are there other solutions that can do continuous replication at a block or sector level -- not file-by-file over a WAN?

Some details:

  • The files are split up into about 75,000 directories.

  • 99% of the daily change comes from adding new directories; existing files are rarely changed.

There's some other relevant discussion here.

Community
  • 1
Jesse
  • 1,910
  • 3
  • 23
  • 28
  • Is there any possibility of splitting the files into organized subdirectories or are they already split up? – Bart Silverstrim Jan 03 '11 at 15:35
  • What is the change rate of the files, and what process does the changing? Any chance of obtaining a list only of changed files, and syncing those (with a monthly full sync during a maintenance window)? – Matt Simmons Jan 03 '11 at 15:39
  • Bart, Matt -- answers added to the question above. – Jesse Jan 03 '11 at 16:45
  • @Matt: just getting a list of edited files is extremely time-consuming. I was hoping some software that could keep track of recent modifications would be faster. – Jesse Jan 03 '11 at 16:46
  • 1
    Just to be clear, you want to do the replication to increase availability, not to implement the backup, right? – hmallett Jan 03 '11 at 17:03
  • @hmallett: I need to do both. If I could replicate the filesystem on a remote box, I could then take disk images of that box, and have both availability and remote backup. – Jesse Jan 03 '11 at 18:02
  • @Jesse: This question is a few years old now. Did you ever find a good solution for this? What did you use? – StackzOfZtuff Nov 15 '17 at 17:24
  • 1
    @StackzOfZtuff: We ended up using Syncrify. The initial sync took days, but subsequent incrementals were pretty quick. – Jesse Nov 16 '17 at 22:14

3 Answers3

3

1.5 TB would take 30 minutes with http://www.exdupe.com/ :p

... given that your disks are fast enough (exdupe is so fast that it's IO bound, not CPU bound).

And I havn't reached any limits on file count yet. Had millions too.

Edit: Ah, you need a partition/sector based backup and not file system level? It can't do that... Maybe http:// www . drivesnapshot.de/en/ is worth a shot (had to add spaces because of spam protection). Does diff backups and shadow volume copy too (no reboot).

Lasse
  • 31
  • 2
  • I've never seen exdupe before, but am testing it now. Very interesting. Good suggestion. – ewwhite Jan 04 '11 at 06:17
  • The only reason I was avoiding filesystem backup was that so far, any attempt to do anything at the filesystem level with 10M+ files takes forever. If exdupe does millions of files in 30 minutes, filesystem is OK with me. Sadly, I'm writing to a USB drive, though. – Jesse Jan 04 '11 at 13:44
1

Check out Shadowprotect. It's not continuous, but it can be set to do an incremental every 15 minutes. It's pretty awesome software. Add on the ImageManager enterprise portion and it also gives you some great replication abilities for offsite backups.

Jason Berg
  • 18,954
  • 6
  • 38
  • 55
  • Thanks, just talked to them. They can't actually mirror the volume, but it does sound like they could do good things with disk images. – Jesse Jan 03 '11 at 17:02
1

I've had great luck with Doubletake, despite the price. Their "move" product might fit the budget though...

See my answer to a similar question here.

Community
  • 1
JakeRobinson
  • 2,886
  • 17
  • 26
  • 1
    Looks like DT RecoverNow (formerly "Backup") is also a possibility, and much cheaper than DT Availability. – Jesse Jan 10 '11 at 14:40
  • Are the target servers vmware or Hyper-V virtual machines? If so, it looks like the right answer for you! – JakeRobinson Jan 12 '11 at 01:33