How should I set up my Fibre Channel (FC) network?

Question

How do FC switches work, and how should I configure mine?

Answer 1

Fibre Channel switches usually connect clients to storage. FC is a protocol that is designed explicitly to transport SCSI commands. In fact, the Fibre Channel protocol is a direct extension of the SCSI protocol. All SCSI commands have a FC equivalent, and FC has a few extra ones that allow for networking. Assuming you have all the physics of your FC network taken care of, the question becomes "How do I configure my FC switch?".

Fabric

A fabric is a switch or a group of switches that are connected to each other (a link between two switches is called an ISL, or "Inter Switch Link"). A fabric has a name, and consists of a set of aliases and zones. What are aliases and zones?

• An alias is a name you can use to avoid typing in WWNs ("world wide names", a unique address for a port on a device. Analogous to a MAC address in a regular network). An alias can contain multiple WWNs, but it's usually best to stick to one. Usually, this means you'll have several aliases for each server and each storage device, like "SQL_0" and "SQL_1" for the first two FC ports of an SQL server. You can choose anything you like, but it'll be easier to manage if you stick to a standard.
• A zone is a list of aliases that can see each other. Typically, you set up a zone for each server containing all of the HBAs and the storage ports it'll be able to see LUNs on. It may seem sensible to put more servers in a zone, however there's always the possibility that a failing HBA could broadcast disruptive traffic to other server ports that might interrupt normal IO. For this reason, you want to ensure that your zones are as small as is reasonably possible. "Single initiator" means zones that contain just one WWN capable of sending SCSI IO - a single HBA.

Each fabric has an active zone set, which is to say a list of zones that are "on". You can create a zone and leave it off, or you can remove a zone from the active configuration without deleting it. You can edit a zone once it's created, even if it's active.

Redundancy

If you have two switches, you'll want to have two fabrics. Each server with two FC ports should have one on each fabric, so if one switch fails, the multipath driver on the server can swap to a path through the other fabric. If your storage has two redundant controllers with two ports each, you'll want to ensure that each controller has a port in each fabric.

Importantly, if you have only two switches, you don't want to run a cable between them. This would, by default, have the effect of merging the fabrics, and generally that's not preferable.

Linking switches

If you have two fabrics already, but need to add switches, you don't need to create more fabrics. You can link the new switches to the old ones with ISLs, and they'll join the fabric. At this point, any traffic that needs to go between two devices on the same switch will be handled by that switch, and any traffic that needs to jump switches will go over the ISL. You will want to ensure that you use enough bandwidth between switches to ensure that the ISL is not the bottleneck, however in the vast majority of FC environments, the switches are never the bottleneck. You can add and remove ISLs without taking down the fabric. If you find yourself growing beyond three or four fixed port switches, you'll probably be able to save some money by upgrading to a type of switch called a "director", which is a chassis that allows you to add port blades.

Adding switches to an existing fabric is conceptually easy, but practically, involves commands you'll probably not use much. You should make sure you have a good grasp of the tools in the manual.

VSANs

Sometimes, it's easier to manage a switch when you split it up. You can do this with a VSAN, which is a virtual switch within a physical one. You give it a list of ports it owns, and those ports act as if they are on a separate switch. Be aware that devices connected to one VSAN can't, by default, connect to devices on the other one.

HBA tools

One of the most important debugging tools you have at your disposal are the tools on the server. Many operating systems have programs written by the HBA manufacturers (like QLogic or Emulex) that allow you to list what devices the server sees. If you don't have an OS that supports your manufacturer's tool, you can always reboot the server and enter the HBA setup from the boot menu. From here, you can see the same information.