How does FaceBook build servers with 12+ hard drives

Question

From The FB Blog:

... is deployed on top of commodity storage blades. The typical hardware configuration of a 2U storage blade is –

• 2 x quad-core CPUs
• 16GB – 32GB memory
• hardware raid controller with 256MB – 512MB of NVRAM cache
• 12+ 1TB SATA drives

Can someone please tell me exactly what parts they use.

I know that you can get motherboards that can take 2 cpus, I can get motherboards that can hold 16 GB of ram, but how do I connect 12 drives?

Answer 1

twelve 1TB drives? peanuts, think BIG! :)

Texas Memory Systems launches the RamSan-6200 SSD System that offers up to 100 terabytes of Flash-based storage in a 40U rack configuration that can sustain a record 5 million input/outputs per second (IOPS) with 60 gigabytes per second throughput while using a little over 6 kilowatts of power.

all this for a whopping price tag of $4.4m, you hear me, Santa? :)

(maybe RAMinSANe would be a better name)

but if you're OK with only 12 drives, you'll get a nice SATA 12 Drive Enclosure with Controller (speed > 500MB/s) for around $1.000

Answer 2

I hate to dig up such an old question, but these answers need help. There are a number of different 2U cases that feature 12x3.5" hot swap bays (3 vertically x 4 horizontally -- See Supermicro's SC826), , and even some that also have an internal bay or two (http://www.servaris.com/servers_m2100.php). It's dense, but not impossible. If you want to get a little crazy, Supermicro's SC847 supports 45 hot swap drive bays in 4U (no motherboard) or 36 with E-ATX motherboard, and someone makes a case (the RMC5D2) with 48 vertical hot swap bays with room for E-ATX board and I think 7 full-sized cards in 5U.

I believe this is for Facebook's Haystack system for photos, where they use a RAID 6 controller to connect the 12 drives in a 2U rackmount server. You can buy a low profile one that still supports 12 drives or use a riser card. As part of their IPO filing in 2012, they revealed that they're storing "more than 100 petabytes" of photos and videos. Even with newer 4 and 6 TB drives, that's a lot of these machines.

Facebook also uses a 100+ PB Hadoop system to analyze everything that happens on the site (multiple TB/day). It skips RAID disk arrays and makes the entire machine redundant by splitting large files into 64MB chunks and writing each chunk to multiple machines. For this you can get a motherboard with 8 ports and cheaply add 4 non-RAID ports. I'm pretty sure these machines don't have so many drives, but the technology is based on stuff that Google is doing. Google does have racks of 12 drive RAID free machines, and a lot of interesting computer designs. They're almost big enough for custom parts to be a volume order of commodity hardware.

You can watch a video on Facebook's different types of servers here.

Facebook has somewhat recently unveiled the Open Vault storage system that supports 30 drives in a 2U Open Rack (same overall cabinet width, but 21" wide internally vs standard 19"). There are 2 individual 1U trays that pull out of the chassis, each with 15 drives (5 wide x 3 deep) and 2 SAS expanders. A video on YouTube shows a system where each of the 30 3.5" drive bays have been populated with 2x2.5" SSD's, which will give some serious IO power when combined with 12G SAS. According to SSD manufacturers, a single unit could support 5 million IOPS. A cabinet full would support over 90 million IOPS and 170 GB/s bandwidth (interface limited) and 1 PB of SSD storage (as much as 3.2 PB of helium filled spinning disk). If you're just looking to use it for a really sweet NAS, there's an Open Compute server that uses the Open Vault with ARM server boards inside.

Even more interestingly, Facebook is experimenting with Blu-ray for cold storage (the data isn't accessed much if ever). Their prototype system has 10,368 discs that hold 100GB each and stores about 1 PB of data. They have a robotic picker that accesses the discs from a rack that holds 24 magazines, with 36 cartridges per magazine and 12 discs per cartridge. Once they get the bugs out and features tweaked, I expect the 10,000 disc Blu-ray changer will be part of the Open Compute Project.

Answer 3

I think that the original FB post is a bit misleading. If they're using storage blades, those are designed to connect to external storage arrays. The blade itself would run Windows Storage Server for example, but you wouldn't store your data on that blade. You would hook a 2U SAN array to that blade and that array could hold 12 hotswap bays at 3.5" or 24 bays at 2.5".

Answer 4

I work with video and since there wasn't drives of 1TB and above 3 years ago, I searched for a solution similar to yours. You have to have internal or external RAID controller for what you want. I found one company that deals in such stuff, and it is (for me at least) some kind of a paradise for creating HUGE storage systems for acceptable price.

http://www.addonics.com/

For example:

http://www.addonics.com/products/raid_system/rack_overview.asp

Answer 5

As others have mentioned there are many different options to fit 12 1TB drives into 2U. While 3.5 inch drives have to be horizontal to fit in that space, 2.5 inch drives like the Western Digital Scorpio Blue WD10TPVT will fit in 2U vertically. You can get a Supermicro S216 based chassis to do up to 24 2.5 inch drives in 2U, along with two 12 port sata raid controllers (like this 3ware 9550SXU-12) to connect them all.

Answer 6

Look at servers 1U, 2U 3U 4U, those fits in rack in server room.

like those http://www.supermicro.com/products/system/2U/

You can look at Dell, HP, IBM servers and many more.

They can suport around 8 hard drive, some more.

Answer 7

Connecting 12+ disks to a machine isn't difficult -- you get a raid controller with 12+ SATA ports on it, or you use SATA expanders, or a SAS raid controller.

The difficulty is physically fitting 12 3.5 inch drives into 2U; I'm not sure you can turn a 3.5 inch drive vertical with only 2U, and you aren't going to fit 12 drives in 2U without either doing that or buring some disks in the box (making them not hot-removable).