Second NIC "not working properly" on diskless Server 2012

Question

I've got (after a considerable fight) a diskless Server 2012 running via the MS software iSCSI initiator being booted via gPXE chained from the NIC's PXE.

However, now that it is booting properly I have another problem (iSCSI HBAs are getting more attractive with every hair-pull). The server has a dual NIC, and Windows is only accepting the NIC that is hooked into the SAN, leaving me with no LAN connection!

Device manager shows both NICs, but the one for the LAN has an exclamation point and the properties indicate "This device is not working properly because Windows cannot load the drivers required for this device. (Code 31)"

Windows obviously does have the right driver, as the two ports are identical and the other is working; furthermore, if I install the same OS to the same hardware but onto a local HDD, it has no problems with the NIC. If I tell it to search for a better driver, it just turns around and says the driver is fine, not surprisingly.

I'm pretty sure I know what's going on here, thanks to the previous chapter in this adventure.

The pre-boot program (gPXE in this case) has to write an iBFT (iSCSI Boot Firmware Table) into memory, which is then picked up by the OS (Windows in this case). This table provides among other things a list of the NICs. For each, it specifies the PCI bus and device #, MAC address, and IP information.

I know from examining its source code (and also by a little tool I developed to dump the iBFT) that gPXE by design/laziness only ever writes one NIC to the iBFT, even though the standard allows about 240 of them. Even if it did write multiple NICs, I'd still be in the same boat because other gPXE/iPXE issues forced me to use the UNDI-only build, which means it doesn't even know about the other NICs.

I am supposing that what's happening here is that Windows is looking at the iBFT and - even though it knows the other NIC exists from its own device management system - is deciding that it can't be used because it's not in the iBFT. I have no idea why it would do this.

Is there some way to coax Windows into using the other NIC even though it is not in the iBFT? Or, is there some iSCSI pre-boot program that actually works right? Or is there an entirely different explanation?

Answer 1

I finally got to the bottom of this and did manage to get it working. In the process, though, I've come to the conclusion that iSCSI boot functionality in Windows, gPXE, and iPXE are all half-baked. I'll share the approach that worked for me in case it helps anyone else, but please be aware of some caveats:

• This is a poor solution. A hardware-based solution, such as an iSCSI HBA, will offer better performance, robustness, and will be far, far easier to set up.

• This solution does not scale well to large deployments, mainly because it requires too much manual labour per diskless server to set up.

• This solution isn't that simple. There may be a simpler solution (other than the obvious, use an iSCSI HBA.) If you know of one, please add it and I'll mark yours the answer if I can duplicate it.

• This solution is an ugly, ugly hack. Use at your own risk!!

Before I go on I want to clarify that any time I say "NIC", I am referring to what Windows considers a single "device", but which might actually only be one of several ports on an actual NIC. This terminology is consistent with both the iBFT standard itself and iPXE/gPXE.

Windows, when booted on its iSCSI initiator, has some very finicky requirements with regard to the iBFT (the table the 'iSCSI boot solution' writes into memory before calling the Windows boot loader, which tells it how to access the iSCSI LU). I've been able to piece together some "gotcha" rules (which may or may not hold in your particular situation):

• If a NIC is not in the iBFT, Windows will not work with it. It will manifest the symptom given in the question.

• The list of NICs in the iBFT has to be sorted in a particular order. I don't have full details as I only had two NIC ports in the test server, on the same NIC. One was PCI 08:04.0 and the other was PCI 08:04.1. If the iBFT listed the NIC at 08:04.1 before the one at 08:04.0, Windows got mad. (Note that there is nothing in the standards requiring a given order.)

• The iSCSI target must be reachable from the first NIC listed in the iBFT. This might require you to switch your SAN and LAN ports, due to the above rule.

• If the first NIC in the iBFT is not the same as it was when Windows was first installed, it will crash and reboot. This may require you to reinstall Windows if your setup wasn't right initially. (I'm not positive what constitutes "same", but a different port on the same NIC definitely wasn't "same".)

• The NIC sections must occur in memory in the same order in which they are listed in the control section, or Windows gets mad. (Note that the standards do not stipulate that the ordering must match - again, this is just Windows being lazy.)

The first rule is the rub. Neither gPXE 1.0.0 nor the Jan. 31, 2013 commit of its successor, iPXE, ever write multiple NICs to the iBFT, even where there are multiple NICs they know about. I've verified this by examination of their source code.

My hacky solution was to get the iPXE source tree, and modify the program such that it writes a second NIC section to the iBFT, corresponding to the other NIC in my server (the NIC I was not booting from.) I just hardwired the MAC address and PCI address. I found it was not necessary to place the IP stuff into the NIC's section - just leave it all zeroed and Windows will assign it later during boot. (Please note that the IP stuff does need to be written for the SAN NIC, but iPXE is already coded to do that.)

By using #define, the actual addresses can be entered into some convenient place rather than having to dig through source code every time you want to change them.

If you make this change, please be aware that the NIC sections have an index byte in their header. The iPXE code doesn't touch these (although it is given in the struct) because it never, ever writes more than one NIC, but if you write a second NIC, you will need to set its index byte to 1 or else Windows will not be happy.

The obvious large downside to this solution is that you have to recompile iPXE for every server, keep these separate versions of iPXE on the TFTP server, and configure the PXE server to issue the different boot programs to each of the servers.

Some knowledge of C programming is needed to make the initial change, along with a Linux distro and the GNU dev tools. The iBFT format is specified here.

I wish I could just post my changes here but I actually ended up changing a very old version which the ipxe.org website tricked me into downloading. (Apparently, they never tag stable releases; I've since learned all releases on the master branch are stable.) I'd rather not encourage anyone to use such an old version.

The newest version still has the same limitation. I'll forward this to their development list so hopefully it gets fixed.