29

The last guy quit (I think a lot of adventures begin this way?) and I've been given the job to build a new machine room. I am coordinating with 2 engineers, campus ITS, campus Facilities Management, a project manager, a construction company and 2 subcontractors (electrical and mechanical). Yay.

Most of these folks have done this before, but I have not. The construction is underway, and everything is fine, but honestly I'm at a total loss about things like where the racks should go, where the floor cutouts should be made, etc. I need to understand things like power consumption, PDUs, CRAC units, BTU generation, cabling, etc. After the construction is finished, I will be in charge of who gets to use the room and what the rules are.

I know there are people who are experts at this stuff, but none of the people involved so far know much (including me). Where do I learn this stuff? Google hasn't turned up much for me.

To summarize, I'm looking for a tutorial reference that answers questions like:

  • Understanding cooling requirements and cooling capacities in a server room
  • Understanding power supply and consumption
  • How to set policy for users in a server room (including things to think about like safety, fire prevention, data security, good custodianship)

I'm a competent software guy, but have very little experience with machines beyond building my own small computers.

Fixee
  • 658
  • 4
  • 13
  • 4
    +1 Sounds like a fun new challenge! If all else fails, there are always contracting companies that will do this for you. – Kyle Smith Jun 09 '11 at 18:01
  • 1
    +1 Great question, as a developer, I am bookmarking this for later reference – lagerdalek Jun 10 '11 at 03:05
  • Can you give some more details on the kind of datacenter you are building? it will have a huge effect on a good answer. What is the square footage of the room? Are there multiple rooms? Is this new construction or renovation? How many racks and servers? Is this a public data center, or private to your company? Are there servers set to go in right away, or are you building this space only for new installs? Are you meant to manage anything in the server or just provide the space for others? I could go on, but more detail would help a lot for whomever tries to answer your question. – Kyle Gibbons Jun 12 '11 at 21:09
  • Also, try doing some searches on "data center" instead of machine room. – Kyle Gibbons Jun 12 '11 at 21:09
  • I managed to find a solution that most people won't have access to: it turns out that someone in my department recently wrote several papers on this topic (building data centers for supercomputers) and is willing to help me on out this relatively simple project. – Fixee Jun 13 '11 at 02:46

5 Answers5

11

You might take a look if you have not come across this site yet (http://www.datacenterknowledge.com/). A ton of white papers across the topics. Some are "marketie," but still good starting points at the very least.

user48838
  • 7,393
  • 2
  • 17
  • 14
  • Good answer, but I'm baffled there isn't some kind of book on this? – DutchUncle Jun 10 '11 at 16:26
  • One of the possibly challenges with books on a broader encompassing topic as this one would be the changing/advancing technologies across each of the specific areas. – user48838 Jun 10 '11 at 18:18
  • Yep, but a good one would give a novice a sound base to start from (ducting, power, cooling, etc...) and some clue about specifying requirements. – DutchUncle Jun 10 '11 at 18:47
  • Same thing with the site, granted a little more "digging" is required, but the trade-off is the possibility of more up-to-date information. – user48838 Jun 10 '11 at 18:58
11

Start with the power. It will pretty easily show you how much of everything else you need.

Get a number, maybe ask the electricians, of how many KW or KVA the project is.

If it's less than say 12 KW, you are talking a medium size build out. Few racks, say five, a hundred or so servers.

100 * (1 amperes) * (120 volts) = 12 000 watts = 12KW

Account for some redundancy in your power. Again, ask the electrician about redundant circuits. Get a drawing if you can. Get UPS.

The cooling will be a function of the power. Depending on the size of your room, you may be able to put cooling right there, called CRAC. The cooling will require more power.

You hit on one last point which comes down to physical security. I've used biometric hand scanners in the past. Another option is a simple proximity card system. You may want video surveillance setup as well as other monitoring of temperature, humidity, and things of that nature.

Good luck.

dmourati
  • 24,720
  • 2
  • 40
  • 69
  • can't you exceed 42kw in a single cabinet if it's fully populated (redundant 1kw PSUs, at theoretical peak draw?) – warren Jun 09 '11 at 18:50
  • @warren, I think you'd have a fire on your hands – dmourati Jun 09 '11 at 18:53
  • @dmourati - understanding that rated draw vs reality is often different, it still seems likely that if you have 10 42U cabinets each fully populated with 1kw-powered servers drawing, say, 30%, that you're going to be running `10*42*1kw*.3 = 126kw`. That's not all that much power, really. – warren Jun 09 '11 at 18:57
  • I think it is your 42KW number that is off. I was specifying, for purposes of discussion, more like 5 racks and 100 or so servers to get to 12. You could go as high as 4.8 KW (KVA really) or 10 but not much more than that. 42KW per rack is not going to work. – dmourati Jun 09 '11 at 19:05
  • 1
    @dmourati - either I'm not understanding power draw, or my math is off if your numbers are right. If you have a dual-CPU device each drawing 115W (peak), that's 330W. Add-in hard disks at ~10W each (say four per server), now you're up to 370W. That's *PER* U of rack space. Add in a 10% fudge factor for unaccounted-for components, and you'r up to 400W *per* U of rack space. A fully-populated 42U rack, therefore, is drawing `42x440W == 18.5KW`. 10 Racks brings you to 185KW. Am I missing something there? Also - we haven't added SAN, network, or other devices yet. – warren Jun 09 '11 at 19:23
  • Try a killawatt: http://www.p3international.com/products/special/p4400/p4400-ce.html The power draw "is what it is" not what the power supplies may say for a "peak" and yes hard disks are included. You're not going to be able to fully populate your racks because you need things like network which take up space. My initial numbers (~12 KW) were intentionally on the low side in terms of density but yours are off the charts high. Mine happen to be coming from a contract I'm negotiating at the moment. To give you an example, go try to source a 18.5 KW rack and see who bites. Then ask for 10. – dmourati Jun 09 '11 at 20:51
  • Then go get some of these: http://www.datacenterknowledge.com/archives/2011/01/07/opengate-targets-higher-density-cooling/ – dmourati Jun 09 '11 at 20:55
  • @dmourati - those racks you linked to are 20-30KW capable. – warren Jun 10 '11 at 02:22
  • Yea, if you are going to try to get to anything even near 18KW, you are going to require some highly specialized and costly cooling/containment infrastructure. Their previous limit was 20KW. Our data center is slab floor with open racks. You see occasional racks up to 12 KW. Most people are 3-5. – dmourati Jun 10 '11 at 06:27
  • When I use APC's [configurator](http://configurator.apcc.com/products/powerstruxure/configurator/PSX_Config.cfm?ISOCountryCode=US), and fully populate a 48U rack (46 servers, 2 network devices), I see power consumption listed at ~24KW – warren Jun 10 '11 at 17:34
  • Still crazy high: http://www.apcmedia.com/salestools/SADE-5TNRK6_R7_EN.pdf – dmourati Jun 10 '11 at 19:03
  • Don't forget 1u for a switch per rack :) – Kedare Jun 20 '11 at 21:56
4

Barroso's book is a great resource; although it doesn't answer every question I have, you can't do much better than a free book by a Google data-center designer.

Fixee
  • 658
  • 4
  • 13
3

My organization is moving into a new office and I found myself referring again and again to The Practice of System and Network Administration. It has quite a bit of information about other topics, but I learned a lot from their discussion of data centers, especially each author's ideal data center.

Helixso
  • 125
  • 1
  • 6
2

A quick browse through Amazon brought up this: it's old, but might give you a solid foundational understanding, so sales people and fast builders don't leave you in a bind:

http://www.amazon.co.uk/Computer-Data-Centre-Design-Constructing/dp/0471825794/ref=sr_1_3?ie=UTF8&qid=1307723182&sr=8-3

DutchUncle
  • 1,265
  • 8
  • 16