5

I'm getting a bunch of brand new condor 3 based switches and these things have a feature I'd like to use called port mirroring. Essentially, a mirror port can be configured to show all traffic being sent between two devices logged into the fabric (like between a storage port and an NPIV WWN, for example).

I've looked into proper SAN taps before through Virtual Instruments' VirtualWisdom, and have experience with a Finisar tap and XGIG for troubleshooting, but this mirror port looks like it can be used to do the same thing. Brocade seem to have no documentation I can find describing how to use the data, though.

My question is what kind of setup would I need to be able to properly use the data from a mirror port? I imagine it could run on a server with an HBA that is plugged into this port?

Basil
  • 8,811
  • 3
  • 37
  • 73
  • By setup, you mean hardware, or software? Also, when you say 'use', what information are you expecting to get from the mirror port? Statistics, specific storage content, auditing, deep packet sniffing for exploits? – Slartibartfast Mar 19 '14 at 04:13
  • I'll obviously prefer software, but if I need hardware, I'd like to know. The use case for this data is mostly performance and troubleshooting. I've seen, for example, badly configured initiators cause RSCNs for all other devices logged into a storage port, and high application measured write latency when the storage is reporting under 2ms. – Basil Mar 20 '14 at 12:38

3 Answers3

2

The following is all I have been able to find on this so far. You probably already have seen this however I am providing it just in case. I am continuing to search for more for you.

M_Port reference in the FOS v7.2.0 Admin Guide: M_Port: A mirror port that is configured to duplicate (mirror) the traffic passing between a specified source port and destination port. This is only supported for pairs of F_Ports. Refer to the Fabric OS Troubleshooting and Diagnostics Guide for more information on port mirroring.

Port mirroring reference in the FOS v7.2.0 Troubleshooting and Diagnostics Guide: Port mirroring: With port mirroring, you can configure a switch port to mirror the traffic between a specific source and destination port. This is only supported between F_Ports. This is a useful way to troubleshoot a problem port without bringing down the host and destination links to insert an inline analyzer. Port mirroring captures traffic between two devices. It mirrors only the frames containing the SID/DID to the mirror port. Because of the way it handles mirroring, a single mirror port can mirror multiple mirror connections. This also means that the port cannot exceed the maximum bandwidth of the mirror port. Attempts to mirror more traffic than what available bandwidth allows results in the port mirror throttling the SID/DID traffic so that traffic does not exceed the maximum available bandwidth. The bandwidth of the mirror port is unidirectional. In general, a host (SID) talks to multiple storage devices (DIDs). Thus, a host does not send full line rate to a single target. A mirror port configured at 4 Gbps can only support up to 4 Gbps of traffic. A normal 4 Gbps F_Port is bidirectional and can support up to 8 Gbps (4 Gbps transmit and 4 Gbps receive) of traffic. If the mirror port bandwidth is exceeded, no credits are returned to the receiver port and thus those devices involved in the mirror connection see a degraded level of performance. Use port mirroring to detect missing frames, which may occur with zoning issues or hold timeouts, capture protocol errors, and capture ULP traffic (SCSI/FICON). This feature cannot be used on embedded switch traffic.


Martin2341
  • 43
  • 3
  • Yep, and I've also got some questions about that bandwidth limitation. Let me know if you find more, and thanks! :) – Basil Mar 25 '14 at 19:19
  • Hi Basil. For the bandwidth limit, it boils down to mirroring multiple ports to a single port and thus as the mirrored ports start hitting higher traffic levels, the single mirror port needs to be able to handle the aggregate of the traffic at that moment. As far as using the data, I understand the idea was more to enable easy setup/implementation of a traffic analyzer without bringing down the port. So, the same traffic is going to both ports, one is the "real" destination and the other can have the analyzer. Does that help? – Martin2341 Mar 26 '14 at 00:26
  • If I have an 8Gb port for analysis, a 8Gb storage port, and two 8Gb server ports, what happens if I'm analyzing all three devices and one server's writing at line speed and the other one is reading at line speed? Does the mirror port put rx traffic and tx traffic onto the same mirrored rx port? If so, that means you need double the bandwidth of the storage port to ensure you don't throttle it. – Basil Mar 26 '14 at 12:49
  • Yes, essentially this is something I am curious about as well but not at the full bandwidth side. Rather, what happens if you had mirror-able data from different monitored ports and it occurs at the same time. This possible contention scenario would be possible even if you were under the total bandwidth total limit. I'll try and get answers for both. – Martin2341 Mar 26 '14 at 20:25
  • Yeah, like if a frame is sent between a SID and DID while the mirror port is busy with another frame. Is this new frame delayed? Or queued? Will it show up on the DID's port before the mirror, if the mirror's port is busier? I know if the mirror port's queue is full, no new traffic will be accepted from a SID. – Basil Mar 27 '14 at 18:47
1

Span Ports or Mirror ports are great ways to mirror the actual traffic between the ports; it can lead to problems when exceeding the aggregate traffic of all ETLs (as Martin2341 mentions). Additionally (you mention Finisar Xgigs) the Span/Mirror ports don't see all the traffic. If you Xgig-Capture on those, you'll see that some B2B primitives are not shared, the SCSI 00 ending the exchange might not show up, and the latency from the actual live links to the mirror/span links is not consistent -- so if you're trying to do timing off that, it'll be all over the map.

Span/Mirror ports tend to require specific setup, and there is a limit as to how many per switch you can do. Some vendors require a license for the number of Span, Mirror, or Diagnostic ports you want to try to enable, and there can be throughput issues exceeding 100 ports mirrored.

We tend to use optical splitters, but they seem relatively new on the marketplace based on support dialogs. Brocade doesn't like them traditionally, but I hear Corning is now providing those, and that gives us a supported (from a "warrantee, guarantee, file a support ticket" standpoint) link from switches to storage that can be tapped for Xgig and similar hardware: the fire hydrants are under the street before the fires happen. It's not free, but a fraction of the cost of a storage port, and sees 100% of the traffic, bit-for-bit, warts and wrinkles and all. Plus, it acts the same whether you later switch from Brocade to Cisco or back.

The Xgig connected to the splitters sees all the traffic as it does when installed inline, but you don't have to down the link to take it out, move it, or manage it (IP, firmware, etc). We tend to recommend tapping all of a VMAX's FAs during a new deployment, if the cost is bourne for the taps. In troubleshooting later, it's worth not having to change-control that link down to tap it, only to have the problem show up whack-a-mole on another link as servers move to another path when you down the link.

-1

I can't speak specifically on Brocade but the usual way to monitor a mirror port or span port as it is called in Cisco is to connect up a box with two NICs. One is the management NIC where you SSH in to the NIC and the other is your monitor NIC where you capture the traffic. The monitor NIC connects to your switch monitor port. You mirror the traffic from the port you want to sniff to the monitor port. You put the NIC in promiscuous mode and then you capture the traffic to pcap or use tcpdump to filter out the packets you want to capture. You can then take the whole pcap file and send it to a workstation for analysis with Wireshark or wade through it with less/cat/grep etc on the box. You can use an old desktop running your favorite Linux distro and two NIC cards for hardware.

mpmackenna
  • 134
  • 5
  • I think that fibre channel switches are a bit different than ethernet. tcpdump probably won't work for FC traffic, for instance, and I don't know if it would even listen on that kind of card (since it doesn't look like a NIC). – Bill Weiss Mar 20 '14 at 17:31
  • My mistake. I read quickly and made some assumptions. You should check with the Brocade community on this inquiry. http://community.brocade.com/t5/Fibre-Channel-SAN/Sniffing-Fibre-Channel-Packets-from-Brocade-Switch-200E/td-p/21680 – mpmackenna Mar 20 '14 at 19:04