3

Our AFP server is now capping out at 1Gbps over our 10Gbps fiber connection. And only in the client<-server direction (client reading from it).

This is a new occurrence. For years now the clients sharing the 10GbE have gotten to share a full 10Gbps. Now if two clients read (for example) they are only sharing 1Gbps.

This doesn't happen over iperf, only with this protocol and only in the read direction. Write will share 10Gbps still.

Server<->Server at 10Gbps will also use the full speed. And the switches in between are all negotiating 10G Full Duplex still.

I should note: our 10Gbps clients get the full 10Gbps and these things I'm reporting are happening whether there are 1 or 20 clients connected, so its not that the bandwidth is maxed or the hard drives won't do that read.

Any ideas?

Edit:

All of the clients are Macs running OS X 10.10.

The servers are Ubuntu Linux 14.04.3 running Netatalk.

The network is a series of 1Gb clients on two different 1Gb managed switches, each connected to a 10Gb managed switch via 4x10GbE SFP+ in a LAG with LACP. Then the 10Gb switch is connected to two servers, each via 4x10GbE SFP+ in a LAG with LACP.

Like this:

Clients <-> 1Gb Switch <-----> 10Gb Switch <-----> Servers

Clients <-> 1Gb Switch <-----^

<-----> = 4x10GbE connections in a LAG

<-> = 1x1Gb connection

The issue occurs with clients on either 1Gb switch and the clients on each switch are sharing 1Gbps, not 1Gbps across both switches.

The only recent changes we've made are:

1) Added a 10Gb client directly connected to the 10Gb switch (But the problem occurs with our without this client connected).

2) Upgraded clients from OS X 10.9 to 10.10.

Chris
  • 83
  • 5
  • 3
    Details like the operating systems and versions in use, the client operating system versions, the network architecture and noting recent changes to your environment would help. – ewwhite Sep 15 '15 at 01:50
  • Are your clients or server in visualized environment by any chance? – dtoubelis Sep 15 '15 at 03:45
  • @ewwhite details added – Chris Sep 16 '15 at 18:21
  • @dtoubelis they aren't, but good point! – Chris Sep 16 '15 at 18:22
  • Do you have any other AFP server that you can test your clients against to see if the problem is a server-side or client-side issue? Can you downgrade (one of) the clients to 10.9? – wfaulk Sep 16 '15 at 18:47
  • 1
    @wfaulk I'm actually working on the downgraded client test right now. I assumed it wouldn't be the case since this is something with the clients having the *split* bandwidth (could they have that kind of awareness of each other?). I don't have any other 10GbE servers so I can't test that. I wouldn't be able to tell by connecting to one of our 1Gb AFP servers. It *is* happening on *both* of our 10GbE AFP servers though. – Chris Sep 16 '15 at 19:13

1 Answers1

1

I remember this issue was discussed when 1Gbps first emerged and it had something to do with flow control protocol being managed on Ethernet level vs TCP/IP level. I managed to find this link that describes the problem. So the proposed solution is to disable Ethernet flow control on the switch.

dtoubelis
  • 4,579
  • 1
  • 28
  • 31
  • That was indeed the issue. Flow control is still on on the 10Gb switch but on the 1Gb switches is has been disabled. Total speed recovery for all clients. Thank you so much for the clue! – Chris Sep 17 '15 at 04:57