0

I need the ability to push images out to multiple servers whenever they're updated. I've looked at Network Filesystems, but they're all but worthless due to their speed. Images can be uploaded to any one of 3 servers, and would then need to be copied to the other 2.

Any suggestions? I'm open to try just about anything.

EDIT: Graphics data (jpg, gif, png, etc). Linux only.

We're currently using rsync. But having it work back and forth is getting cumbersome. It's all local network.

Glen Solsberry
  • 1,506
  • 5
  • 28
  • 35
  • 2
    what os is this running on? – tony roth Dec 07 '10 at 19:48
  • 2
    When you say that network filesystems are worthless due to their speed, do you mean due to overhead? What have you looked at? Some network filesystems are pretty speedy. We have GPFS in production here, and while it's got its own share of nightmares, it's *quite* fast. (Faster than a local RAID 10.) – mattdm Dec 07 '10 at 19:55
  • 1
    By "images" do you mean graphics files or disk images? – mattdm Dec 07 '10 at 19:56
  • Are you saying "images" as in "data files that are graphics", or as in "disk dumps of the OS and applications" ? – mfinni Dec 07 '10 at 19:57

1 Answers1

0

What OS? Are you transferring them across WAN links or within a data center? Absent this information, I'm going to give you a Linux/Unix answer.

rsync is good for handling file transfers of this sort, since it only transfers updates (rather than the entire file) when a file is modified. If you want this to happen automatically when a new image is uploaded, you probably want to modify your web application to kick off the appropriate process.

If you're working in a Windows environment there are Windows versions of rsync available.

larsks
  • 41,276
  • 13
  • 117
  • 170
  • Rsync isn't going to be particularly fast if the images are all-new, or if computing the difference happens to be more work than sending the whole file. Need... more... information! – mattdm Dec 07 '10 at 19:56
  • @mattdm: For GNU rsync the '-W' disables the incremental algorithm and just sends the whole file. – Scott Pack Dec 07 '10 at 20:03
  • @packs: sure. But then we're back to "why is a network filesystem considered slow?" – mattdm Dec 07 '10 at 21:10