I don’t know about you but I’ve seen some really awful server rooms recently as I’ve visited partners and clients. I also did a co-location install in a datacenter a while back and was surprised at how poorly thought out some of the switch placement and cabling was done by some the customers who shared space in the datacenter.
I thought it might be helpful, therefore, if I shared a few of the tips I’ve learned over the years about choosing the right racks and cabinets for your datacenter, deciding what equipment goes where into which rack, how best to plan and organize your network cabling, and so on. Many of these practices have been learned from trial and error after discovering that my previous choices or solutions were inadequate. I’ve also gleaned other tips and good advice from my conversations over the years with network administrators, datacenter managers, and fellow IT pros I almost bumped into as I wandered aimlessly around the corridors of datacenters looking for a poorly labeled cabinet or a particular server blade. And I’m hoping that some of you who read this will share some hard-earned tips of your own by using the commenting feature at the bottom of this article.
Now while the fundamental building block of a datacenter is probably the cabinet or rack, I’m going to focus in this present article on what usually generates the most frustration and expletives when you’re trying to install or replace something or troubleshoot an issue in a server room or datacenter. Then later in a future article, I’ll talk about what to look for when you’re choosing a cabinet or rack, and also about good and poor placement of devices within your cabinet.
But for now, let’s start with what causes the most headaches for us when we enter our server rooms or datacenters.
Yes, I’m talking about the wiring!
Spaghetti without the sauce
Spaghetti wiring is the curse of administrators who are under pressure to fix a troubling issue in their server rooms or datacenters. While the most obvious example of this is poorly organized Ethernet cabling, there can also be a similar problem involving the placement you choose for the power cords that provide electrical power for switches, routers, and rack-mounted servers in your cabinets.
But let’s focus on Ethernet cabling as that’s where the most obvious problem usually arises. About the worst thing you can do in a cabinet or rack is to use only black cables for connecting your server blades and storage pools to your patch panels and Ethernet Switches. Because if all of your cables are jet black, you have absolutely no visual clues offer help for guiding you when you’re rushing to figure out which switch port is connected to which patch panel port to which server and so on.
“But what if I carefully label each of my nice black Ethernet cables? Won’t that make my job easier?” Well maybe, but then again probably not. There are two basic problems with the “label my cables” approach, or three actually. The first problem is that labeling only works if you always take the time to do it properly. In the real world of the IT trenches, however, time is usually of the essence and so shortcuts are often taken when you’re rushing to meet a deadline. For example, when you replace a failed blade or upgrade a switch you might decide to just re-use the existing cabling to minimize downtime. Then when you’re finished you say to yourself, “I’ll update the labels on the cables later when I have time” or “I’ll just leave the labeling as is and update its description in our documentation later on.” And of course in the busy world of IT the phrase “later on” is a code word for “never.”
Which brings me to the second problem with labeling Ethernet cables: the information on the labels needs to be documented somewhere. That’s because labels are usually small so you generally can’t write lengthy functional descriptions on them. They need to be coded instead of being wordy. For example something like “A3-115-C15” meaning port 15 on patch panel C in room 115 on the third floor of building A. Then you need to document exactly what this cable is used for, what its functional purpose is in your server room or datacenter. For example, is it used for server management? For transporting data? As part of your monitoring system? And so on. And we all know of course how poorly we IT pros tend to be at keeping our infrastructure documentation up to date and accurate, again mostly because of time pressure.
There’s a third reason however that labeling cables frequently doesn’t work very well: heat. If your server room is poorly air-conditioned or your cabinet is overcrowded, hot spots can occur due to the exhaust paths of the power supplies in the devices of the cabinet. If the label on a cable should just happen to be too close to one of these hotspots, the glue on the label can easily melt causing the label to fall off. So you might end up with some unlabeled, shiny (and possibly gooey due to the heat) black cables which can result in costly mistakes when you’re replacing a blade or repurposing a switch.
Tri-color pasta
If black is beautiful but impractical for server rooms, what about using different colored network cables for different purposes? Many network admins I’ve talked to follow this kind of approach. For example, you might decide to use green cables for those that carry network management packets, red cables for those carrying data, gray for those used by your monitoring system, and blue for cables connecting to your network backbone. Or perhaps gray for cables connected to servers, yellow for cables connecting patch panels to switches, pink for cables connecting to printers. There are even cables from some vendors that are both colored and have a stripe along them (for example red with a white stripe) and I’m sure one can think up some innovative use for using them.
While this is a commonly followed approach, it has some drawbacks. First, if you’re a proponent of neat and tidy and like to wrap ties or Velcro strips around your cables to neatly organize them, having a few basic colors like this may not be of much help when you’re struggling under time pressure to solve a problem. For example, if you’ve tied together a bunch green cables in a cabinet, then following each patch cable from panel port to termination point will probably be difficult without making mistakes. Unless you take the time of course to untie your cable wraps, but that’s a pain, isn’t it?
Another problem with coding cable functions by color is that you assume it’s always correct. For example, if yellow represents cables that carry data then when you’re troubleshooting you will automatically assume that the yellow cable you’re holding is carrying data. Which it may be, or it may not if the cable was paneled incorrectly or repurposed wrongly previously.
Ken Chase, the founder of Heavy Computing Inc. in Toronto, Canada, recommends following a different approach, however, and it’s one that I’ve tried myself successfully and really like. Ken suggests one just use a lot of different colored network cables and not assign any specific meaning or function to any single color. “I tend to use as many random colors as possible,” Ken says, “and then I note it in the switch configs.” The advantage this has over dedicated color patterns is that “with no pattern, no dangerous assumptions are made. Instead, you must check with the switch config.” As Ken indicates it’s much easier to follow a single cable through a bunch of tied up cables if the colors are all randomly mixed together like this. “Maxing out a number of different colors leads to easier identification and fewer chances of neighboring cables having the same color.”
Server rooms: Use your noodle
So in the end, maybe spaghetti cabling isn’t so bad after all as long as you use a mix of different colors. Though nobody really likes tri-color pasta since the noodles all taste the same no matter what color they are.
I use NetBox