8

We're getting ready to move to a new colo facility and I'm designing the rack layout. While we have a full rack, we only have 12U worth of hardware right now:

  • 1x 1U switch
  • 7x 1U servers
  • 1x 2U server
  • 1x 2U disk shelf

The colo facility requires us to front-mount the switch and use a 1U brush strip, so we'll be using a total of 13U of space.

Regarding growth, I'm reasonably sure we'll be adding another 4U in servers, 1-2U of network gear, and 2-4U of storage in the mid-term. Specific questions I'm hoping to get help with:

  • where should I mount the switch? the LEDs are on top...
  • should I group the servers by function with space for adding new machines?
  • as an alternative, should I group servers based on whether they are production or staging?
  • where in the rack should I start? in the middle? at the top? at the bottom? equally spaced?

Here's a silly little ASCII diagram of what I'm thinking right now. Please feel free to tear my design apart, I've really no idea what I'm doing :) Any advice is very welcome.

edit: to be clear, the colo is providing redundant power with UPS and generator, so that's why there's no power gear in the plan, except for the 0U PDU that I didn't diagram.

    42 | -- switch ----------------------
    41 | -- brush strip -----------------
    40 | ~~ reserved for second switch ~~
    39 | ~~ reserved for firewall ~~~~~~~
    38 | 
    37 | -- admin01 ---------------------
    36 | 
    35 | -- vm01 ------------------------
    34 | -- vm02 ------------------------
    33 | ~~ reserved for vm03 ~~~~~~~~~~~
    32 | ~~ reserved for vm04 ~~~~~~~~~~~
    31 | ~~ reserved for vm05 ~~~~~~~~~~~
    30 | 
    29 | -- web01 -----------------------
    28 | -- web02 -----------------------
    27 | ~~ reserved for web03 ~~~~~~~~~~
    26 | ~~ reserved for web04 ~~~~~~~~~~
    25 | 
    24 | 
    23 | 
    22 | 
    21 | 
    20 | 
    19 | 
    18 | 
    17 | 
    16 | -- db01 ------------------------
    15 | +- disks ----------------------+
    14 | +------------------------------+
    13 | ~~ reserved for more ~~~~~~~~~~~
    12 | ~~ db01 disks ~~~~~~~~~~~~~~~~~~
    11 | 
    10 | +- db02 -----------------------+
     9 | +------------------------------+
     8 | ~~ reserved for db02 ~~~~~~~~~~~
     7 | ~~ disks ~~~~~~~~~~~~~~~~~~~~~~~
     6 | ~~ reserved for more ~~~~~~~~~~~
     5 | ~~ db02 disks ~~~~~~~~~~~~~~~~~~
     4 | 
     3 | 
     2 | 
     1 |
bleything
  • 183
  • 7

6 Answers6

8

There are as many ways to do this as there are opnions on the subject. My opinion is to keep it simple: firewalls, routers, and switches go at the top of the rack. The heaviest equipment goes on the bottom and then work your way up from there. Leave a U or two in the middle for a pull-out KVM tray, about waist to chest high, if you plan to put one in the rack.

Trying to populate the rack by function is probably going to lead to frustration. You've got a new file server but no room where the file servers go, etc.

I must admit that I need to keep my OC tendencies in check when I'm figuring out where to put things.

joeqwerty
  • 108,377
  • 6
  • 80
  • 171
  • 2
    Putting "like" machines together without a functional reason to do so serves no purpose and creates make-work later on. In general, always ask yourself if whatever "organization" you're trying to impose has a functional purpose. If it doesn't consider not doing it. – Evan Anderson Mar 02 '11 at 04:15
  • +1 I do the exact same, but put the switches in back (usually in the middle somewhere). – Chris S Mar 02 '11 at 18:27
  • 1
    Thanks! I've realized that you (and others) are right about organization. Not worth it. New plan is to just build from the bottom up. – bleything Mar 03 '11 at 04:27
  • Glad to help... – joeqwerty Mar 03 '11 at 12:16
6

I echo the joeqwerty - Keep it simple, and label things. I do like to keep disks near the cooling and work away, with the switches on top - or using 0 U mountings on the side. You'll want to keep a U below the switch free for the cables running back to the servers (seems silly to force you to front mount a switch, but meh.)

Trying to break things out by function will just lead to insanity. The most that I would do is disks down low, 2U servers on the disks, perhaps some free space for more 2U boxes, and then 1U servers on top.

If you do a good job with the cabling, moving servers in the rack (while not a great plan) is doable.

Good luck!

tsykoduk
  • 413
  • 2
  • 4
3

If your cooling is from below (raised floor) you want your high power draw servers close to the floor. Reverse for overhead cooling.

Make sure to install blanking panels on all unused spaces.

I like to mount the switches in the middle, it keeps the cables a bit more manageable. And push back on that front mounting. The switches mount on the rear, IMHO. Can you imagine tracing a cable from front to back?

I also like to leave the top U for the connecting patch panel (drops.)

Is this a rack they are providing? Do they provide the power strips (PDUs) as well? Do the PDUs require rack units too or are they vertical mount? I'm assuming that you have two PDUs. You should verify that they are powered from two different upstream UPSes. If they are not, that might change your decision on how you are setting up the power.

toppledwagon
  • 4,215
  • 24
  • 15
  • The switch mounting policy is to ensure that hot air blows out the back, I'm not sure they're going to budge on it. I forgot to mention in my main post (but have since added) that we have a 0U PDU. Redundancy is as you describe. – bleything Mar 02 '11 at 01:14
  • I know of more than one switch that supports reversible fan direction. – toppledwagon Mar 02 '11 at 05:08
2

We did this same exact thing two weekends ago, although with half the hardware.

We put all the servers and SAN at the very bottom of the rack, so it was easier to rack them because of their weight and size.

We also put the switches on the back of the rack, so there aren't cables running to the front of the rack, and everything is nice and tidy in the back.

We also had to find a spot for a KVM, monitor and keyboard, which we put right above the servers.

KJ-SRS
  • 984
  • 1
  • 8
  • 11
  • I much prefer front mount switches with a 1u pass through. It keeps things clean, and you can read cable labels and managed the switch from the front. – Doug Luxem Mar 02 '11 at 18:24
2

where should I mount the switch? the LEDs are on top...

We usually mount the switch at the top, because then any excess cable can be coiled up above the racks, hopefully in some sort of horizontal cable management system on top of the racks. It's okay to put cables on top of the racks, as long as it's done neatly and you are careful to not create a rats nest.

Putting the switch at the top means that the LEDs are hard to see unless you stand back and take a look. That's not always practical if you have narrow rows.

The colo facility requires us to front-mount the switch

So the network ports will be in front, and you'll need to loop all 24/48 cables over the top of the rack, and down the back? Sounds like another reason to put the switch at top.

Have you considered a KVM switch and a slide-out keyboard/monitor? Those should go in the middle, at an ergonomically comfortable height.

where in the rack should I start? in the middle? at the top? at the bottom? equally spaced?

Heavy machines should go at the bottom. This will save your back, and will improve maneuverability if you need to fit the big servers into the finicky rails. You never want to be put in a position where you need to hoist a heavy server above your head. Even if you have a server lift, it can be difficult to mount a server onto it's rails, and a low rack location will help you do that.

Stefan Lasiewski
  • 22,949
  • 38
  • 129
  • 184
  • 1
    Yep, ports forward on the switch. Cables come up the back of the rack, through the brush strip, and into the switch. I'll leave space for a KVM, but we don't have one now. – bleything Mar 02 '11 at 01:35
-1

it looks great where is the battery backup? so many servers and equipment and no seriuos battery backup?

miro23
  • 198
  • 2
  • 9
  • 6
    Any good colo will take care of your power for you. Typically they'll have N+1 redundant battery backups plus diesel generators. He's much better off than having a single-point-of-failure-ups. – Jason Berg Mar 02 '11 at 00:49
  • Precisely what Jason said... the colo provides very robust redundant power with UPS and generators. – bleything Mar 02 '11 at 00:53
  • Colo's also hate it when you put UPSes in your own rack. It really buggers up their failover capability. – Tom O'Connor Mar 02 '11 at 01:11
  • 2
    Using your own UPS is usually a matter of fire code. If you do allow you to add one, it needs to be connected to the EPO. – toppledwagon Mar 02 '11 at 05:10