0

We want to compress large backup files and push them into the cloud. I'm looking for an application/utility/codebase that will allow us to efficiently compress on the fly as part of the transfer.

I've found that WinRAR is taking 40 times longer to compress our 100G files when compared to Sql Native compression, but it's much better in reducing the file size. I want something that delivers good compression without taking hours to do it.

Our files are Sql Server backups, using a test database of 100G with native sql backup compression, this reduces the file size by a factor of 3, and the backup time changes from 10 minutes to 25. 15 mins of CPU time saves 66G of bandwidth, which we can use for other files being pushed to the cloud.

I've tried using WinRAR on the 100G backup file, and I get a compression factor of 9, which is 3 times better than native sql, and that's great as we can now push 9 times more content into the cloud. But the compression takes 7 hours!

We can't be the only shop to have faced this issue? What's your solution to maximising use of precious bandwidth.

MrTelly
  • 311
  • 3
  • 9
  • You need to test different compression utilities (gzip, bzip2, xz, 7z etc.) with different compression settings and find one which matches your requirements. In general, the better the compression ratio, the longer it takes to compress. We cannot make these tests for you. – Tero Kilkanen Jul 18 '17 at 09:05
  • If you have piles of cpu cycles to spare across multiple cores, there's also the parallel implementations of some of the above pigz and pixz, combine with tar if it's more than one file. There's also a plethora of compression benchmarks on the internet you could use as a starting point. – Kjetil Joergensen Jul 19 '17 at 01:13
  • I would like to take an already tried/tested approach and not re-invent this particular wheel. We can't be the only shop to have faced this issue? What's your solution to maximising use of precious bandwidth. – MrTelly Jul 19 '17 at 08:17
  • 1
    Have you considered Veeam or any other similar backup software that has in-built data compression and deduplication algorithms? – B.J.Goodman Jul 19 '17 at 18:06

0 Answers0