2

I read through some similar question, but still have some confuse about my situation.

My website allow user upload huge number of large images (scan pages of book). My server will auto tiling and saving those images. So each page will become 10 thousands of small tiler (each image is 128px*128px, take 2k space). I have about 100GB images, and right now they already fill up all the inode table.

Here is couple of my though and understanding, please correct me if I am wrong.

  1. mysql blob

    upside: Don't need worry about inode / directory structure any more.

    downside: slow image serving, and may slow down whole database

  2. file system

    upside: faster image serving

    downside: Slow backup speed, extra directory design, need large inode size (this also means I have to reformat my server disk)

  3. mongoDB BinData with base64 (I am not familiar with this, but it seems to be a good option)

  4. Amazon S3

    upside: they take care of everything

    downside: need cause money, and lost full control of those images.

    (UPDATE: I may not allow to use 3rd party provider, but I will still live this here, since it may be a good solution for other people, according to @Niels suggestion.)

  5. Other Magic Great solution.

So I am wonder which is the best in my situation and why. Thanks for help

xzhang
  • 121
  • 4

2 Answers2

1

I would definitely go for S3 unless you have thousands of very active users... And even if you do some donations or advertising should cover the cost. S3 storage is really, really cheap compared to your own server. Storing 100GB of data would cost you about 9USD per month, at high redundancy level. That means mirrored to at least 4 storage locations, with transactional guaranteed durability of 99.9999%, for a cost that wouldn't realistically buy you a single server in 3 years. Traffic could be costly, but you're paying that with every colocation provider, so no real difference unless you get to big numbers.

S3 is internally a highly optimized string dictionary, so capable of storing millions if not billions of related files.

Honestly, save yourself the frustration and be gone with all backup, redundancy and server administration woes for a small price, then focus on keeping the surrounding website performant. Much more fun too.

Also, if bandwidth usage is really astronomic, consider using your webserver as an 'edge' server. In most mass storage scenarios 1% of the content is requested 99% of the time (think Facebook photos). Some scripting that would locally mirror 1GB of the recently requested images, and flushes out by last access time, should be easy to do, and keeps S3 bandwidth costs within the free tier probably.

Niels Keurentjes
  • 259
  • 4
  • 12
  • Thanks for your suggestion, but sadly I just been told I may not allow to use a 3rd part provider on this. Is any other solution? – xzhang Apr 17 '13 at 01:44
1

As far as I'm aware, there is no way to adjust the number of inodes on an existing file system.

If you can recreate the file system, you can use the -N option to specify a higher number of inodes for the new FS

fukawi2
  • 5,327
  • 3
  • 30
  • 51
  • Yes, so I hope can find some better way to avoid resize inode table. – xzhang Apr 17 '13 at 15:56
  • The only other thing I can think of... If you still have free space somewhere, create a new partition and mount that inside your existing store. That's a short-term hack resolution though. – fukawi2 Apr 17 '13 at 23:31
  • So another partition for all image saving ? This may works for me, better than recreate the whole disk. Is this also means saving in database is a really bad idear? – xzhang Apr 17 '13 at 23:56
  • I'm not a fan of binary data in databases, but that's my personal opinion... – fukawi2 Apr 19 '13 at 06:40
  • you could use ReiserFS instead and avoid the inode limit problem altogether – Renan Apr 21 '13 at 00:17