3

I am developing a web application service and am looking to scale our internal servers.

Our current server has ~1TB of storage capacity. Users create ~10gb of data which is added to our database daily.

As you can see at our current rate, we would only be able to sustain this storage growth for ~100 days. We do not expect the rate of data creation to slow in the near future. Due to the nature of the information we are not looking to use commercial cloud storage (aws, google, microsoft, softlayer etc.)

We would like to build a server infrastructure that can be continually expanded (beyond the limits of 100TB). This would be a gradual process as needed, and would span multiple 4U server racks.

My question is that what would be the standard way to do this without over-complicating our software. I have looked into ZFS and openNAS, but there seems to be limitations with pooling and continually expanding storage.

What is the best way to build a homogenous storage architecture that can be continually expanded to support our storage needs?

Gordon
  • 31
  • 4
  • 4
    The question is very open-ended, and leads to subjective answers. Capacity planning is only 1 part of a storage equation. What about redundancy, fault tolerance, high availability, performance, throughput, retention and for how long? What about backups? What about DR? How much of the data needs to be readily available? What's the budget? There are different solutions (hardware and software) that specialize in different areas, so please be descriptive about what you're trying to do. With limitations in current technologies, there are no full-proof homogeneous methods that will last forever. – CIA Aug 14 '17 at 19:17
  • Understood, however, the throughput, bandwidth, and budget are not directly related to the problem of growing storage, those factors can be adjusted as necessary to suit our needs for increased storage capacity. – Gordon Aug 14 '17 at 19:51
  • 2
    At a higher level, how critical is this data to your business? What's the budget? Rolling your own storage and then having to maintain it is doubly expensive. First, you don't work for free. And then, while you're babysitting your homegrown storage you're not actually providing any value to your employer because you're doing something a commodity storage server does. Any serious, professional analysis of your situation would include looking at simply *buying* a storage system that meets your bandwidth, capacity, IO ops/sec, and reliability requirements. – Andrew Henle Aug 15 '17 at 10:45

1 Answers1

2

Even from my limited experience I should say that your problem is under-specified. For example, you don't talk about load, latency, and bandwidth requirements. Check this Q&A page for more: Can you help me with my capacity planning?

My experience is building and managing 140TB (and growing, 80TB primary + 60 backup) storage for our lab. We use it as a storage for research data (imaging data mostly, some genomics; no databases). System consists of two servers in different server rooms, and was designed to be expandable. Please take what I write with a grain of salt.

Our servers run FreeBSD and ZFS, 4U boxes with SAS expanders. Each box is stuffed with 4-5TB drives. Backup is newer and has 12x5TB drives, configured as RAID6 aka RAIDZ2.

Expandability is achieved via two routes:

  1. Each Z Pool (e.g. 12x5TB) can be expanded by adding extra 12 HDDs in parallel. Trick is that we'll have to add drives to storage in chunks of 60TB, we can't (AFAIK) add 30TB and six months later extra 30TB. If you are interested in ZFS yo ushould read up on terminology (many websites, including Oracle's and FreeBSD's)
  2. Using SAS allows us easily add extra 4U JBOD box with up to 45 drive slots, by connecting to main box's HBA/RAID card via single cable.

So, from what I know:

  • ZFS, when run on server-grade hardware, can be successfully used with almost no supervision for hosting 100TB of data and automatic off-site backups
  • Expansion of ZFS storage can be easily done when using SAS expanders, if bandwidth requirements are met
  • Expand-ability is not "smooth" in a sense that if you configure your zpool in blocks of 20TB you will have to add drives in chunks of 20TB or larger down the road. You can't just buy a 1TB hard drive and plug it in if you run some sort of RAID6/7. In case of RAID10 you can do that because you "block" is essentially 1 or 2 drives

Update

Current system bandwidth limitations are as follows:

  1. Network via copper Ethernet 1Gbps (whole campus) or 10Gbps (inside the rack)
  2. RAID60 arrays using HDDs: n x 160MB/s (in 4x configuration is it ~5Gbps)
  3. SAS3 protocol using 4x12=48 Gbps (connects backplane to HBA using SAS3 4x cables)

As you can see, we are currently limited by 1Gbps ethernet, if we work inside the rack (storage<->analysis server), we become limited by server's ability to swallow 5Gbps streams. Our data is mainly large files (5-500GB), so I am not sure about the IOPS. To improve bandwidth currently we need to install fiber optics in the building (our lab and server rooms are on different floors), then we become limited by the speed of our RAID60 arrays. Then we either have to add extra stripes or build SSD-based storage. This system was built to work instead of USB3-based HDD, which it beats.

aaaaa says reinstate Monica
  • 500
  • 4
  • 16
  • We are less concerned with network, latency, and computational load, as these are things we've worked with before. Good to know that my understanding of ZFS was flawed and I will take more time to look into it. – Gordon Aug 14 '17 at 18:29
  • @aaaaaa Could you please provide some performance measures of your current system? (Write / read throughput limit of storage and of network) Will you be able to simply add new drives or will you need to do something to improve network throughput? – Oleg Rudenko Aug 22 '17 at 08:30
  • @OlegRudenko added few notes, hope it makes sense. To solve speed limitation on 10Gbps fiber optics we will have to add a lot of drives (our single stripe is 6-8 drives, so we need to add 6-8x4 drives to get speed to 10Gbps). If you need bandwidth, you might wanna consider SSDs – aaaaa says reinstate Monica Aug 22 '17 at 21:26