5

I have installed three Intel X540-T1's in servers running Ubuntu Server 12.04, but all are non-functional, no link lights, no packets sent or received, and no connection on ip4 or ip6 whether set up as dhcp or static. Also, dmesg doesn't detect cable connection or disconnection.

I updated the default ixgbe driver to Intel's latest version (3.11.33) with no change. The ethernet controller is being reported as X540-AT2 (which might be a problem that I can't figure out how to fix), but the subsystem is X540-T1 so I believe that might be intended.

Does anyone have any experience with this that could assist?

ifconfig eth2
    eth2      Link encap:Ethernet  HWaddr a0:36:9f:14:5f:ea
    inet addr:192.168.101.1  Bcast:192.168.101.255  Mask:255.255.255.0
    UP BROADCAST MULTICAST  MTU:1500  Metric:1<br>
    RX packets:0 errors:0 dropped:0 overruns:0 frame:0
    TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000 
    RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)


ethtool -i eth2
    driver: ixgbe
    version: 3.11.33
    firmware-version: 0x8000037c
    bus-info: 0000:08:00.0
    supports-statistics: yes
    supports-test: yes
    supports-eeprom-access: yes
    supports-register-dump: yes


lspci -vvnns 08:00.0
    08:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10 Gigabit X540-AT2 [8086:1528] (rev 01)
    Subsystem: Intel Corporation Ethernet Converged Network Adapter X540-T1 [8086:0002]
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 32 bytes
    Interrupt: pin A routed to IRQ 16
    Region 0: Memory at e8000000 (64-bit, prefetchable) [size=2M]
    Region 4: Memory at e8200000 (64-bit, prefetchable) [size=16K]
    [virtual] Expansion ROM at e8280000 [disabled] [size=512K]
    Capabilities: <access denied>
    Kernel driver in use: ixgbe
    Kernel modules: ixgbe

Edit output of dmesg: 

dmesg | grep ixgbe
    [    1.684782] ixgbe 0000:08:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
    [    1.684800] ixgbe 0000:08:00.0: setting latency timer to 64
    [    1.685138] ixgbe: 0000:08:00.0: ixgbe_check_options: FCoE Offload feature enabled
    [    2.068069] ixgbe 0000:08:00.0: irq 45 for MSI/MSI-X
    [    2.068075] ixgbe 0000:08:00.0: irq 46 for MSI/MSI-X
    [    2.068079] ixgbe 0000:08:00.0: irq 47 for MSI/MSI-X
    [    2.429029] ixgbe 0000:08:00.0: (PCI Express:2.5GT/s:Width x8) a0:36:9f:14:5f:ea
    [    2.620029] ixgbe 0000:08:00.0: eth2: MAC: 3, PHY: 3, PBA No: G54042-004
    [    2.620033] ixgbe 0000:08:00.0: eth2: Enabled Features: RxQ: 2 TxQ: 2 FdirHash RSC 
    [    2.652184] ixgbe 0000:08:00.0: eth2: Intel(R) 10 Gigabit Network Connection
    [    2.652237] ixgbe 0000:10:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
    [    2.652254] ixgbe 0000:10:00.0: setting latency timer to 64
    [    2.652566] ixgbe: 0000:10:00.0: ixgbe_check_options: FCoE Offload feature enabled
    [    3.032057] ixgbe 0000:10:00.0: irq 48 for MSI/MSI-X
    [    3.032062] ixgbe 0000:10:00.0: irq 49 for MSI/MSI-X
    [    3.032067] ixgbe 0000:10:00.0: irq 50 for MSI/MSI-X
    [    3.393034] ixgbe 0000:10:00.0: (PCI Express:2.5GT/s:Width x8) a0:36:9f:10:a8:f2
    [    3.584019] ixgbe 0000:10:00.0: eth3: MAC: 3, PHY: 3, PBA No: G54042-004
    [    3.584023] ixgbe 0000:10:00.0: eth3: Enabled Features: RxQ: 2 TxQ: 2 FdirHash RSC 
    [    3.616178] ixgbe 0000:10:00.0: eth3: Intel(R) 10 Gigabit Network Connection

Edit output of sudo lspci

sudo lspci -vvnns 08:00.0
    08:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10 Gigabit X540-AT2 [8086:1528] (rev 01)
Subsystem: Intel Corporation Ethernet Converged Network Adapter X540-T1 [8086:0002]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 16
Region 0: Memory at e8000000 (64-bit, prefetchable) [size=2M]
Region 4: Memory at e8200000 (64-bit, prefetchable) [size=16K]
[virtual] Expansion ROM at e8280000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
    Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
    Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
    Address: 0000000000000000  Data: 0000
    Masking: 00000000  Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
    Vector table: BAR=4 offset=00000000
    PBA: BAR=4 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
    DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
    DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
        MaxPayload 128 bytes, MaxReadReq 512 bytes
    DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
    LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Latency L0 <512ns, L1 <4us
        ClockPM- Surprise- LLActRep- BwNot-
    LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
    LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
    DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
    DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
    LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
         Compliance De-emphasis: -6dB
    LnkSta2: Current De-emphasis Level: -6dB
Capabilities: [100 v2] Advanced Error Reporting
    UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
    UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
    UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
    CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
    CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
    AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] Device Serial Number a0-36-9f-ff-ff-14-5f-ea
Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
    ARICap: MFVC- ACS-, Next Function: 0
    ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
    IOVCap: Migration-, Interrupt Message Number: 000
    IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
    IOVSta: Migration-
    Initial VFs: 64, Total VFs: 64, Number of VFs: 64, Function Dependency Link: 00
    VF offset: 384, stride: 2, Device ID: 1515
    Supported Page Size: 00000553, System Page Size: 00000001
    Region 0: Memory at 00000000c0e00000 (64-bit, non-prefetchable)
    Region 3: Memory at 00000000c0d00000 (64-bit, non-prefetchable)
    VF Migration: offset: 00000000, BIR: 0
Capabilities: [1d0 v1] Access Control Services
    ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
    ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Kernel driver in use: ixgbe
Kernel modules: ixgbe
Jeff Carr
  • 81
  • 1
  • 10
  • Maybe not helpful, but a serious question: why would you pay for that hardware and run in on Ubuntu, instead of an OS for which you can get vendor support? (SLES, RHLE, even Windows...) – HopelessN00b Nov 09 '12 at 19:05
  • This is one of a number of test servers that will only be up for a couple of days. It has only a very short life span and a single purpose. – Jeff Carr Nov 09 '12 at 20:01
  • Oh, well sorry about your luck. Sounds like the test results are in, and Ubuntu doesn't play well with those cards. Although, since it's listed as a supported OS, I'd get Intel tech support on the phone and make it their problem. Or try to, at least. – HopelessN00b Nov 09 '12 at 20:27
  • Can you add the output of " *dmesg | grep ixgbe* "? If you do the *lspci* as root or with sudo, then *Capabilities: * seems problem. – John Siu Nov 10 '12 at 02:18
  • Did you check the cables? After all you don't have a link according to your link lights. And those should be on even if you don't have any driver installed at all. – lsmooth Nov 10 '12 at 10:33
  • I did check the cables, the cables work just fine on other servers, or on the same servers with other 10 gig NICs. – Jeff Carr Nov 11 '12 at 03:32
  • I can post the dmesg messages on Monday as well as the full lspci output running with sudo. The full output was considerably longer, and so I ran without to make the posting a bit more concise. – Jeff Carr Nov 11 '12 at 03:33
  • Your interface is also missing the RUNNING flag. This usually means the link is down (like the link lights also indicate). Maybe it's a problem considering autonegotiation. You could try reducing the linkspeed to 1Gbps or 100 Mbps on the switch ports and see if you get a link then. If you can really rule out the cables and switches I'd call tech support like HopelessN00b suggested. – lsmooth Nov 11 '12 at 19:21
  • There are no switches involved, I have a bunch of servers with direct connections from one 10 gig NIC to another NIC on an opposing server. All the servers are able to communicate using the older Intel prototype 10 gig cards, but none are able to using the X540-T1s with the exact same configuration and cabling. – Jeff Carr Nov 11 '12 at 19:49
  • What models are the other NICs? – lsmooth Nov 11 '12 at 20:12
  • The other NICs are versions of the 82598EB. These were not the finalized versions however. http://ark.intel.com/products/36918/Intel-82598EB-10-Gigabit-Ethernet-Controller – Jeff Carr Nov 12 '12 at 15:05

1 Answers1

3

It turns out that the prototype NICs won't talk directly to any non-prototype NICs. I'm not sure why exactly. The x540-T1 in Ubuntu work just fine to talk to each other, at least after installing the latest Intel drivers (and possibly before). Thanks for the help everyone.

Jeff Carr
  • 81
  • 1
  • 10