I have an issue with Veeam Backup & Replication backups failing because the Veeam proxy servers cannot connect to the ESXi host over port 902 (NFC). The ESX hosts are on VLAN65 and the Veeam proxies are on VLAN60. What is really strange is that my laptop that is on VLAN50, can connect.
The Veeam error is:
31/08/2021 20:25:32 :: Processing WSVR12 Error: NFC storage connection is unavailable. Storage: [stg:datastore-3246,nfchost:host-3243,conn:vcsa.domain.net]. Storage display name: [ESXDS05].
Failed to create NFC download stream. NFC path: [nfc://conn:vcsa.domain.net,nfchost:host-3243,stg:datastore-3246@WSVR12/WSVR12.vmx].
To test connectivity, from the Veeam proxy servers, I run the following PowerShell cmdlet:
PS C:\> Test-NetConnection -ComputerName esx01.domain.net -Port 902
WARNING: TCP connect to esx01.domain.net:
ComputerName : esx01.domain.net
RemoteAddress : 192.168.65.2
RemotePort : 902
InterfaceAlias : Ethernet0
SourceAddress : 192.168.60.203
PingSucceeded : True
PingReplyDetails (RTT) : 0 ms
TcpTestSucceeded : False
I can complete traceroutes and pings.
On the ESXi servers, I have checked that vSphere Replication and vSphere Replication NFC services are enabled on the VMkernel (192.168.65.2).
I use an Untangle NG Firewall that acts as my router. There are no rules between VLAN60, VLAN65 and VLAN50. However, when running the Test-NetConnection cmdlet, I see invalid_blocked in the session list between the Veeam proxy and ESXi server. Researching this error does not provide any further assistance. An Untangle employee wrote here:
Don't worry about it. It is entirely normal and happens all the time. Thats why it isn't logged by default because while we should log it because it happened, its not particularly interesting or noteworthy and can often happen a lot.
- I have added a bypass rule to the firewall, but that has made no difference.
- The Windows firewall on the Veeam proxies is completely disabled.
- There are no restrictions on the ESXi firewall, that I can see.
- The ESXi, VCSA and proxy servers have all been rebooted.
- Backups were working intermittently until a few days ago.
I can't see that there is any problem with DNS, authentication, firewalls, routing or anything else in Veeam's KB1198 as I can connect from VLAN50 to VLAN65 without issue.
If anyone can provide any pointers, further troubleshooting suggestions or ideas on what may be happening, I'd be grateful if you could share.
T.I.A
P.S. Veeam Backup & Replication v. 10.0.1.4854 running on Windows Server 2016 ESXi 6.7 with vSphere.
Update
I have another ESXi host (v. 7.0) that is standalone. It is on the same VLAN65 and Test-NetConnection cmdlet works. In terms of networking, it has a much simpler setup and the management VMkernel does not have replication or replication NFC enabled.
