How to access large volume of shared files?

Question

I am part of a group of engineers that works together on simulations; we are having trouble figuring out how to store/access simulation output files.

• Each simulation study generates ~20GB of output files. Presently we run 1-2 studies/week.
• Simulation output files must be read/write accessible from several local computers.
• A study's files may need to be accessible for a few years.
• Our company is small with limited IT budget/support.

In the past we have either saved the simulation output on one of the local computers and shared a directory with the other computers, or saved the simulation output in a network location that we can mount as a drive on the local computers. Both solutions are running out of room.

What approaches make sense for sharing/accessing this volume of data?

Answer 1

Assuming 2 studies a week for 52 weeks, thats 2 Terabytes of data a year. That is a lot of data.

Remote storage, like Amazon S3, would not be a good solution if you need immediate access to old data. However, if you dont mind downloading large amounts of data over time, then it could be a cheap and redundant solution.

For immediate access, you need local storage. A network attached storage (NAS) device would be a good choice. And since you have a limited budget, you can add one device per year. A device like this would work. Set one up, and as you need more storage, you can buy another one. Obviously, you can buy larger capacity devices if you choose.

One thing I would suggest, although it costs more, would be to get a device with RAID support. RAID will give you redundancy in case a drive dies - in other words, you won't lose your data.

Personally, I would go with name brand NAS devices. They cost more, but you are more likely to get better support if something breaks. I shy away from the cheaper models, as I see people have too many issues with them. You get what you pay for.

You also should consider keeping your local copy on NAS devices, and backing up your data to a low cost, slow storage like Amazon's Glacier in the cloud. Backups are your friend.