iSCSI target stuck reconnecting after server reboots

Question

I've been having some ongoing issues with my iSCSI connection not reconnecting itself after the server reboots, as shown below:

OS is Windows Server 2008 R2. The iSCSI device is for my SQL Server 2008 R2 instance that is in a failover cluster with another identical server. Both servers exhibit the same problem.

The target is listed as a "favorite target" and requires no authentication. Digests are turned off. The iSCSI target is exposed via a Synology NAS device.

It's also worth mentioning that my file/print servers (same OS) exhibit the same problem. I've tried doing research myself but there hasn't been much help so far. This is a very basic setup and most solutions assume a more SAN-like configuration.

All server except the NAS are on ESXi 6 hosts (one VM in the pair on each host).

Edit: Discovery tab:

Update: Log messages:

Answer 1

EDIT:

Ok, so event 113 is KB972107, which is a fairly generic class of problems, but the "...network stack is not fully prepared" caught my attention. Then we see the more interesting event 103.

See this experts-exchange post with a similar issue...

If I attach them while the server si running, everything appears to function correctly, but if I reboot, I the server will usually hang after login and the drives either slowly appear or not at all.

...and solution which points to the following KB Article.

---

Without any further information, my hunch is that the connection is not being gracefully terminated when the initiator is rebooted. This leaves the connection in an inconsistent state on the target device. When your initiator tries to reconnect, the target still believes the original session is active and the connection hangs, either because one side is simply confused, or because you have a limit set on the number of sessions per initiator.

Go ahead and try this again, watching the state of the connection on the remote server. If my hunch is correct, you'll see the session still active on the NAS side. This could be an issue with your server not terminating the session quickly enough before it's forcibly disconnected, or it could be the result of a very long session timeout on the NAS.

Answer 2

We also use Synology Nas's in most of our clients sites and I was seeing exactly the same issue on a regular basis. I spoke to Synology tech support and they suggested configuring authentication masking. I my case I was authenticating the connection using chap and windows was consistently logging an authentication error. The situation was much worse when upgrading the units as I would disconnect a dozen or so luns and half would generally fail to reconnect.

Since making this change I've gone from seeing this issue on around 100% of reboots to only having seen it on a single machine in the last six months.

I realise you don't have authentication configured, but it may be worth a try configuring the masking on the synology end. You'll need to copy the iqn string from the storage manager while it is connected. You can find it in the iscsi target screen and then go to edit > masking and create a new entry where you can paste the string.

Answer 3

I came across this recently and found that even if you set the mask it would still fail. It seems to be linked with Team NIC's being used.

I found you had to do the following: Remove all existing ISCSI targets for the NAS, including in Volumes & Devices Remove CHAP and use masking instead (IE no access by default but read/write for your imitator you connecting with. Use the autoconnect but don't set any specially NIC's or targets at the top. Normally this is a fix for devices not reconnecting but I found this caused the problem instead. Once connected run the auto configure on Volumes and Devices to make sure its reconnected.

Now you should find it reconnects when you restart.

Thanks, Rob Holmes