Why am I getting 950 Mbps up but only 360 Mbps down on Gigabit Ethernet?

46

13

I got two desktop computers talking to each other directly. They both have Gigabit Ethernet capable network adapters. That's 1 Gbps or 1000 Mbps. I have them connected with a brand new 10 meter long Cat6 UTP straight cable and I get pretty close to that theoretical maximum. The Windows Task Manager (Network tab) shows 844 - 946 Mbps in one direction. But in the other direction, it only shows about 326 - 365 Mbps.

Local: 192.168.100.152
Remote: 192.168.100.151

The local computer runs Windows 8.1 Pro, and I have it connected remotely to the other computer which runs Windows Vista Ultimate.

Iperf results

I used Iperf to do some testing. I ran the test for 60 seconds each time. I ran the test 10 times for each direction of communication. I have then put together this table with test results to get an average.

192.168.100.152 -> 192.168.100.151          106 MB/s
192.168.100.152 -> 192.168.100.151          107 MB/s
192.168.100.152 -> 192.168.100.151          108 MB/s
192.168.100.152 -> 192.168.100.151          107 MB/s
192.168.100.152 -> 192.168.100.151          107 MB/s
192.168.100.152 -> 192.168.100.151          104 MB/s
192.168.100.152 -> 192.168.100.151          101 MB/s
192.168.100.152 -> 192.168.100.151          108 MB/s
192.168.100.152 -> 192.168.100.151          108 MB/s
192.168.100.152 -> 192.168.100.151          108 MB/s
----------------------------------------------------
Min: 101 MB/s    Max: 108 MB/s    Avg: 106.4 MB/s (851.2 Mbps)

192.168.100.152 <- 192.168.100.151          41.1 MB/s
192.168.100.152 <- 192.168.100.151          41.1 MB/s
192.168.100.152 <- 192.168.100.151          41.1 MB/s
192.168.100.152 <- 192.168.100.151          41.0 MB/s
192.168.100.152 <- 192.168.100.151          41.0 MB/s
192.168.100.152 <- 192.168.100.151          41.1 MB/s
192.168.100.152 <- 192.168.100.151          41.0 MB/s
192.168.100.152 <- 192.168.100.151          41.1 MB/s
192.168.100.152 <- 192.168.100.151          41.1 MB/s
192.168.100.152 <- 192.168.100.151          41.1 MB/s
-----------------------------------------------------
Min: 41.0 MB/s    Max: 41.1 MB/s    Avg: 41.07 MB/s (328.56 Mbps)

My question is, why is it so much slower in the other direction?

Windows Task Manager

This is the network diagram as seen while performing the tests in Iperf.

a b

Pay attention to the diagram in the following two screenshots!

c d

Did you notice how it changed from "1 Gbps" to "500 Mbps" in the upper right corner as I switched from sending data to receiving data. Why did it do that? Is it somehow sensing the other network port as half of 1 Gbps when going one way, but yet full when going the other way around?

File transfer test

I did some more testing with a data file, to get some more realistic disk to disk readings. I created a 1 GB file for this purpose. I used only default Windows file sharing facilities. From the local computer, I connected to the C$ share on the remote computer and dragged and dropped the file back and forth (rope skipping) between them, changing the file name each time. I timed everything to the best of my abilities and this is what I got.

192.168.100.152 -> 192.168.0.151    1073741824 Byte    25 s    40,96 MB/s
192.168.100.152 -> 192.168.0.151    1073741824 Byte    20 s    51.2 MB/s
192.168.100.152 -> 192.168.0.151    1073741824 Byte    16 s    64 MB/s
192.168.100.152 -> 192.168.0.151    1073741824 Byte    16 s    64 MB/s
192.168.100.152 <- 192.168.0.151    1073741824 Byte    11 s    93.091 MB/s
192.168.100.152 <- 192.168.0.151    1073741824 Byte    34 s    30.118 MB/s
192.168.100.152 <- 192.168.0.151    1073741824 Byte    11 s    93.091 MB/s
192.168.100.152 <- 192.168.0.151    1073741824 Byte    11 s    93.091 MB/s

The throughput indicated by the Windows file copy diagram is telling a different story. Here I am downloading two files, one after the other to two different locations on the same disk. The first copy indicated 107 MB/s sustained up to 41%, and the second one indicated 98.9 MB/s sustained up to 87%.

e f

So this is in line with the results I got with the Iperf tool. Now, here is what it looks like when I upload to the remote computer.

g h i

It sustains 103 MB/s up to 73%, and then it poops down to stinking 27.3 MB/s at 82% and then goes up a notch to reach 49.1 MB/s at 93%.

Here are two more funny looking "roller coaster" diagrams.

j k

Update 1 - Link speed

I tried disabling the Wifi adapter on the remote computer. (The Wifi adapter was already disabled on the local computer.) I think this is what Timtech meant by that comment. I had the same thought - that having both wired and wireless adapters enabled at the same time limited the throughput on the wired adapter to the level of the Wifi adapter (adapting to the slowest adapter for compatibility). Because the Wifi adapter (DWA-160 Wireless N in this case) is usually detected as a "52 Mbps" - "104 Mbps" link by the Vista computer.

In the following screenshot, the remote computer is set up as a server and the local computer is set up as a client (192.168.100.152 <- 192.168.100.151).

l

But disconnecting the Wifi adapter on the remote computer did not help my low throughput on my wired connection.

Not only that! In Windows Task Manager on the remote computer, the link speed for the wired adapter (LAN 1) appears as "1 Gbps". If you refer to the screenshots above you can see that it is detected as a "500 Gbps" link on the local computer. So for the same wired connection, Windows Vista says it's a 1 Gbps link, while at the same time, Windows 8.1 Pro says it's a 500 Gbps link... which one is right then?

Here's what it looks like on the remote computer when I set it up as a client and the local computer as a server (192.168.100.152 -> 192.168.100.151).

m

As you can see here, about 95% of the 1 Gbps link is being utilized. That translates to 950 Mbps. It's exactly what I got in the test above. But going the other way around is a whole different story.

Update 2 - Duplexing and MDI-X

As suggested by some of you I had a look at the duplex settings. Both local and remote computer were set to auto negotiation mode, as you can see by the screenshots below.

n o

I tried changing to "1.0 Gbps Full duplex" on both computers. I then did the same type of tests as I did before, using Iperf. With local computer as server and remote computer as client I get around 950 Mbps max. With local computer as client and remote computer as server I get around 360 Mbps.

Here, have a look at these screenshots.

p q

What you see here is the diagram for when I upload and download between the two computers. The higher graph (95 - 98% utilization) is local to remote (upstream 192.168.100.152 -> 192.168.100.151). The lower graph (~ 33% utilization) is remote to local (downstream 192.168.100.152 <- 192.168.100.151).

To try to rule out any Auto MDI-X issues, I had one of those crossover adapters attached to one end of the cable (the local computer).

r

That would certainly make the cable a crossover cable. Hell, I even had it tested with a network tester! It is indeed crossed now (pins 1/3, 2/6)!

So now I have true crossover cable connection between the two computers, and I have manually set "1.0 Gbps Full duplex". Yet I still have the same problem. Any more ideas? Besides upgrading the Vista computer (or reinstalling the 8.1 computer)?

Update 3 - Software or Hardware limitation?

My best guess is that I have two operating systems that are not compatible with each other. They are both Windows systems, but not all Windows systems are made equal. I will have to try using Vista on both, or 8.1 Pro on both and see what kind of throughput I get. That means buying an upgrade. Damn you Microsoft.

Both computers are custom built by the way. Here are some specs.

Local
-----
Gigabyte GA-EP45-UD3R
Intel Core 2 Quad Q9650
Intel P45
Corsair XMS2 DHX TwinX DDR2 PC6400/800MHz 4 GB
Realtek 8111C chips (10/100/1000 Mbit)
WD Caviar Black WD1002FAEX
Windows 8.1 Pro 64-bit

Remote
------
Gigabyte GA-X38-DQ6
Intel Core 2 Duo E4500
Intel X38
Corsair XMS2 DHX TwinX DDR2 PC6400/800MHz 4 GB
Dual Realtek 8111B chip (10/100/1000 Mbit)
WD Caviar Black WD1002FAEX
Windows Vista Ultimate 64-bit

Tonny suggested that the Vista machine might be using a bad Realtek chip. So I dug up these specs. I see now that the Vista machine uses a B revision of 8111 while the local machine uses a C revision of the same chip. Does this mean anything? They are both clearly specified for 1000 Mbit (see above) by the manufacturer. Could it be that 8111B underperforms that much (360 Mbps)?

These particular drives reach 107 MB/s burst rate. That's exactly the number I've seen in the test on the local computer. But even the sustained sequential or random read/write of maybe 55 MB/s does NOT translate to 360 Mbps. That should give me somewhere around 440 Mbps, and not the 360 Mbps I'm getting. So I don't suspect that to be the bottleneck, especially since they both use the same drive model. And besides, file copy operation is one thing, but Iperf is not using disks at all, it only uses RAM memory for the tests.

Update 4 - TCP Checksum Offload

As suggested by Tonny, I tried switching off TCP checksum offloading (for IPv4 and IPv6).

s t

I also switched "Speed & Duplex" back to auto for both computers. But this did not help. I still have the low throughput in one direction, and high in the other direction.

Update 5 - New driver version

I tried updating the driver version on both local and remote to the latest version downloaded from Gigabyte website and Realtek website.

Update path...

On local (RTL8111C):
8.1.510.2013 (2013-05-10, Microsoft)
8.20.815.2013 (2013-08-15, Realtek)

On remote (RTL8111B):
6.241.623.2010 (2010-06-23, Realtek)
6.250.908.2011 (2011-09-08, Realtek)
6.252.1109.2012 (2012-11-09, Realtek)

u v w x

I still got the same lousy throughput in one direction.

Update 6 - CPU utilization

I checked CPU utilization. This should not be an issue. Here are my findings.

On local...

Download: 4 - 10 %
Upload:   4 - 10 %
Idle:     0 -  4 %

On remote...

Download: 24 - 38 %
Upload:   10 - 25 %
Idle:      1 -  6 %

Local (download, upload, idle)...

y z aa

Remote (download, upload, idle)...

ab ac ad

The remote uses much more CPU power, but this is also the one with the slower Core 2 Duo. But it never exceeded the 38 % point during my tests. What's particularly interesting here is that it uses so much more CPU power when downloading (local -> remote) than when uploading (local <- remote).

So with a throughput of 950 Mbps it uses 38% and at 360 Mbps it uses 25 %. Also the core utilization is not balanced, it uses one core more than the other. I'm not sure what conclusion to draw from this. The local computer doesn't display core utilization, so I can't compare it. But CPU utilization is even on the local computer (10 % on download/upload).

Update 7 - New Intel Gigabit network adapter

I have now installed a brand new PCI-Express Gigabit network adapter from Intel as a replacement for the built-in Realtek RTL8111B on the remote computer, which is supposedly too slow on upload. The product number of the Intel adapter is EXPI9301CT. This adapter is supposed to be very good according to reviews I've read. I just want to rule this out as a possible bottleneck.

ae

I've done some testing now with Iperf for Windows and here are the results.

Local (download, upload)...

af ag

Remote (download, upload)...

ah ai

On average, this adapter is actually a little bit slower than the Realtek adapter. I think it has a smaller overhead than the Realtek and as a result of that a more stable continuous throughput. But I still get only around 360 Mbps in one direction, and 950 Mbps in the other, even with this Intel adapter.

local: 192.168.100.152 (win 8, realtek 8111c)
remote: 192.168.100.154 (vista, intel desktop ct)

192.168.100.152 -> 192.168.100.154    113 MB/s
192.168.100.152 -> 192.168.100.154    104 MB/s
192.168.100.152 -> 192.168.100.154    103 MB/s
192.168.100.152 -> 192.168.100.154    104 MB/s
192.168.100.152 -> 192.168.100.154    102 MB/s
192.168.100.152 -> 192.168.100.154    104 MB/s
192.168.100.152 -> 192.168.100.154    101 MB/s
192.168.100.152 -> 192.168.100.154    102 MB/s
192.168.100.152 -> 192.168.100.154    101 MB/s
192.168.100.152 -> 192.168.100.154    104 MB/s
----------------------------------------------
Max: 113 MB/s    Min: 101 MB/s    Avg: 103.8 MB/s

192.168.100.152 <- 192.168.100.154    42.2 MB/s
192.168.100.152 <- 192.168.100.154    41.2 MB/s
192.168.100.152 <- 192.168.100.154    41.1 MB/s
192.168.100.152 <- 192.168.100.154    43.0 MB/s
192.168.100.152 <- 192.168.100.154    42.3 MB/s
192.168.100.152 <- 192.168.100.154    42.3 MB/s
192.168.100.152 <- 192.168.100.154    40.2 MB/s
192.168.100.152 <- 192.168.100.154    40.9 MB/s
192.168.100.152 <- 192.168.100.154    41.3 MB/s
192.168.100.152 <- 192.168.100.154    42.0 MB/s
-----------------------------------------------
Max: 43.0 MB/s    Min: 40.2 MB/s    Avg: 41.65 MB/s

I have no idea why it peaked up at 113 MB/s in the first test run, local to remote. It kept that speed throughout the test run, the graph was almost flat at 113 MB/s. As before, I used a 60 seconds interval for each run. On the next run however it dropped to 104 MB/s.

As you can tell by these values, I still have the same throughput with this Intel adapter as with the built-in Realtek adapter. So I think it's safe to say that it has nothing to do with the adapter itself. So we can stop blaming RTL8111B as being an inferior/lesser chip than the RTL8111C found on the other motherboard. This looks more and more like a software/OS/configuration issue, or all three things at once.

Update 8 - Great results with Ubuntu LINUX

After exhausting all other options I finally decided to run some tests with Linux, and I got great results. I used a Ubuntu Linux 13.10 Live system and Iperf for Linux (version 2.0.5-3) on both local and remote machine. Here are the results.

=======================================================
REALTEK 8111C <-> REALTEK 8111B | IPERF ON UBUNTU LINUX
=======================================================

local: 192.168.100.152
remote: 192.168.100.151

192.168.100.152 -> 192.168.100.151    112 MB/s
192.168.100.152 -> 192.168.100.151    112 MB/s
192.168.100.152 -> 192.168.100.151    112 MB/s
192.168.100.152 -> 192.168.100.151    112 MB/s
192.168.100.152 -> 192.168.100.151    112 MB/s
192.168.100.152 -> 192.168.100.151    112 MB/s
192.168.100.152 -> 192.168.100.151    112 MB/s
192.168.100.152 -> 192.168.100.151    112 MB/s
192.168.100.152 -> 192.168.100.151    112 MB/s
192.168.100.152 -> 192.168.100.151    112 MB/s
----------------------------------------------
Max: 112 MB/s    Min: 112 MB/s    Avg: 112 MB/s

192.168.100.152 <- 192.168.100.151    110 MB/s
192.168.100.152 <- 192.168.100.151    111 MB/s
192.168.100.152 <- 192.168.100.151    111 MB/s
192.168.100.152 <- 192.168.100.151    111 MB/s
192.168.100.152 <- 192.168.100.151    111 MB/s
192.168.100.152 <- 192.168.100.151    110 MB/s
192.168.100.152 <- 192.168.100.151    111 MB/s
192.168.100.152 <- 192.168.100.151    111 MB/s
192.168.100.152 <- 192.168.100.151    111 MB/s
192.168.100.152 <- 192.168.100.151    111 MB/s
----------------------------------------------
Max: 111 MB/s    Min: 110 MB/s    Avg: 110.8 MB/s

Local (download, upload, idle)...

aj ak al am an

As you can see I get the same throughput in both directions when using Ubuntu. Is it because I use the same OS on both machines, or is this something else? Would I get the same throughput if I had the same Windows versions set up on both machines? I don't understand why it would matter if I use a slightly outdated Windows version, namely Vista, on one machine and the latest version on the other?... I mean Vista is still a current and supported OS and backed up by Microsoft. Windows XP is a different story.

But I know they are doing everything they can to kill off Vista. For instance, the latest Office 2013 is intentionally not supported on Windows Vista. I'm sure Microsoft wishes that Vista never happened. Just like they will be wishing that Windows 8.0 never had happened. But I'm usually just as persistent as they are, and I don't upgrade my Windows installs until I absolutely must.

So the question is how to get the same throughput in both directions with two different Windows versions. Windows Vista should be capable of Gigabit speed - it's not a 20 year old OS or something, it's not Windows 95 we're talking about. Vista is a modern OS. I haven't tested running the same Windows version on both machines yet. There might be a difference in TCP implementation or something between the two OS versions. If so, then I will probably be forced to upgrade the Vista machine. Either that or switch to Linux. I'm not prepared to pay more for less. Why would I have to upgrade Windows just to get Gigabit throughput in both directions?...

Update 9...

Cable

I tried reversing the cable. I got the same results as before. I also got a new Cat 6 patch cable and tried that one. The throughput test results were the same. So the cable is not the issue here. I only used pre-terminated/molded patch cables. So the wiring should be correct. But I plan to terminate my own installation cables later on.

FW and AV

As for firewall (FW) and anti-virus (AV), I don't use any third-party FW or AV software. I only have the Windows Firewall and the Security Essentials. I have disabled them both on both machines. The throughput test results were the same as before.

ao ap

aq ar as

LAN Speed Test

I have installed LAN Speed Test Lite 1.3 on the local machine. I believe the test is performed between the memory on local and the disk drive on remote machine. I'm not sure. But it asks for a share path on the remote machine. I used o$ share on remote.

at au av

Upload: 427 Mbps
Download: 420 Mbps

I don't trust these results too much. If you look at the graph you can see that it varies very much throughout the test. The test was a "successive" test, that is: write (upload) test first, and then read (download) test. Obviously if you do a simultaneous upload/download test the overall throughput will be lower. But I'm not interested in such tests. So far I have only been doing "successive" tests with both file transfer tests in Windows (file sharing/smb) and in Iperf.

I haven't done any memory to memory tests with LAN Speed Test because it requires the use of a program called LST Server on the remote, and this program requires registration in order to use it.

Update 10...

Disk drive tests

I used Crystal Disk Mark 3.0.3 to test the disk drives. Here are the results.

Local disk: 118 MB/s read, 113 MB/s write
Remote disk: 70 MB/s read, 69 MB/s write

These are sequential read and write speeds based on 5 runs and a 1000 MB load.

This is the local disk (disk mark, read, write)...

aw ax ay

And this is the remote disk...

az

But I don't get this... these results seem contradictory.

Okey, the local disk can read at 118 MB/s, so that would allow for the reported upload of about 100 MB/s. But the remote disk would not be able to receive it if it is only capable of 69 MB/s write. But by some magic twist I still get little over 100 MB/s upload on average.

Going the other way around makes more sense. If remote disk can read at 70 MB/s, and the local disk can write at 113 MB/s, then the download should not be any faster than 70 MB/s. I get about 40 MB/s download on average. That would seem reasonable.

So I can't conclude anything from these results. I mean the disk drive on the local computer is barely used. It is also the disk that holds the OS, and it is the only partition on that system. While the remote disk is almost full, and it is also partitioned with several partitions. However, it is not used for the OS. I chose drive letter O: for the test here, because this is the partition with most free space.

(Note that I used drive letter C: in the previous tests which is on a completely separate Seagate disk drive that holds the OS on the remote machine. So those readings are not comparable.)

Write caching

With disk write caching enabled I got these results.

Local to remote: 106 MB/s
Remote to local: 42.2 MB/s

I then disabled write caching on all drives on the remote and on the local drive.

ba bb

I did not reboot because no reboot was requested for the changes to take effect. I then got following results.

Local to remote: 106 MB/s
Remote to local: 42.1 MB/s

There was practically no change at all. No reboot, and no reboot was requested.

QOS packet

I then went on to disabling QOS Packet Scheduler for the appropriate adapter on the remote machine, and then on the local machine.

bc bd

Local to remote: 107 MB/s
Remote to local: 41.9 MB/s

No significant change here. Again, no reboot and no reboot was requested.

Jumbo packets

I then went on to enable jumbo packets I used the 4 GB setting because 4 KB is the largest MTU size that's supported on both machines.

be bf

Local to remote: 105 MB/s
Remote to local: 33.3 MB/s

Now here, the upload (local to remote) was not affected, but the download throughput was reduced significantly. No reboot was requested, but I decided to reboot both machines anyway, just for good measure. I then performed the same tests again and got these results.

Local to remote: 117 MB/s
Remote to local: 33.2 MB/s

So the upload is now even faster, but download is still slower than it was before I made these changes, even after the reboot. I would have expected them both to go up a little. What does this mean?

Samir

Posted 2013-11-04T22:29:29.637

Reputation: 17 919

Came across this great question. Simply don't use iperf under windows, it's a bad port. Try an alternative. – Matt H – 2017-01-18T21:47:42.783

Could be the computer's motherboard is limited in internet speed. – Timtech – 2013-11-04T22:56:24.953

@Timtech Ok... it does seem to limit it somehow. Notice that it changes from "1 Gbps" to only "500 Mbps" in Windows Task Manager on the local computer. See the screenshots I just added. Could this be the reason? – Samir – 2013-11-04T23:01:39.493

It could be the cable. Have you tried another Cat6 cable? Also it would be worth checking the duplex settings of the network adaptors to make sure they are communicating at full-duplex. Is this a cross-over Cat6 cable? – Sim – 2013-11-06T01:30:59.273

@Sim Yes, "I have them connected with a brand new Cat6 UTP cable". No, I haven't tried another cable. No, it's not a cross-over cable, it's a straight cable. Should I try a cross-over cable instead? You mean as in duplex mismatch?

– Samir – 2013-11-07T19:54:56.390

The answer to this question is simple you have auto full duplex enabled. Can you covert all your units to the same makes your problem harder to see without pulling out a pen and paper... – Ramhound – 2013-11-07T20:19:16.847

4I think the throughput figures displayed in task manager is auto-scaled based on your actual throughput. It is certainly that way for Windows 8 and mine is currently shown 54 Mbps and in the last few minutes it has been 100MBs and 100kbs (when idle) and various values in between. – sgmoore – 2013-11-07T20:19:55.090

@Ramhound Is it? Is it really that simple? I did have auto negotiation enabled on both ends. But what do I achieve by setting duplex settings manually? I did that, and it didn't help. See update 2. – Samir – 2013-11-07T22:54:27.043

Duplex/mdx setting and cross-over or not is a red herring. Gigabit with AUTO settings on both sides will automatically use full-duplex and do automatic cross-over if needed. If you had a mismatch here performance would be bad in BOTH directions. The slow uploader is the Core2Duo Vista machine with the older motherboard I'm guessing (isn't clear from your description). I think it uses the first generation Realtek which just isn't able to deliver in upload. TCP checksum off-loading in the driver might help. If it is on try switching it off. (Sometimes it works against you.) – Tonny – 2013-11-07T23:08:10.750

@Ramhound What units do you mean? You mean the use of "Mbps"? That's the same thing as Mb/s. What would you like me to use instead? For my own calculations I prefer using MB/s because other units are not very representative for the amount of data I transfer in a normal situation. Generally, in the world we live in today we are dealing with very large files. It would be meaningless to use... say KB/s or even Kb/s, not to mention b/s or B/s or even less so Bauds. For referring to a dialog, I use whatever unit Windows is using. I'm not to blame for Microsoft's choice of units. – Samir – 2013-11-07T23:15:35.790

@Sammy 1 Mbps and 1 MB/sec are not the same thing. One is megabits second the other is megabytes per second – Ramhound – 2013-11-07T23:22:10.593

@Ramhound Yes, that is correct... I know 1 Mbps is 0.125 MB/s and 1 MB/s is 8 Mb/s. So?... your point is? I said that Mbps is the same thing as Mb/s, not MB/s as you are suggesting here. Are you pointing out that the use of lower and upper case B letter is ambiguous? If so, then I will change that. – Samir – 2013-11-07T23:31:36.673

@Ramhound There! I have now changed 851.2 Mb/s to 851.2 Mbps and 328.56 Mb/s to 328.56 Mbps. That's the only place I used "Mb/s". In all other places I used "Mbps" or "Gbps" when referring to bits per second. So the slash is now only used when referring to Bytes per second. Does that make you happy now? :) Just to recap now: slash (MB/s) means Byte per second, and no slash (Mbps) means bits per second. ;) – Samir – 2013-11-07T23:38:58.697

@Sammy - When I first looked through your post I was on a phone and it appeared at first glance that some calculations were in Megabits and the others were in Megabytes. I don't know what you had before ( since I first made the comment it appears you updated the question ) I am sure it makes it more clear. While I know the difference personally I don't like having to do math while also reading :-) – Ramhound – 2013-11-07T23:39:57.973

@Ramhound Ahh... OK. So how does it look now? The only calculations I did was really just the averages in the Iperf tables. The actual readings (in MB/s) were directly brought in from Iperf. You can set Iperf to use a certain format. I used the -f MB option. Simply because I like MB/s and it's what Windows is using and I can easily compare different values. The KB/s used to be like a golden standard, but as we got more bandwidth and higher throughput and larger files over the years, the MB/s has become more relevant than KB/s. – Samir – 2013-11-07T23:45:48.730

@Tonny Yes, you got it right. I mean the "uploader" (a.k.a. remote) is the Vista machine and it runs on a Core 2 Duo (X38 chipset). The local computer runs on a Core 2 Quad (P45 chipset). I don't know the Realtek chips, but yes, the X38 board is a bit older than the P45. So you think it's a hardware limitation on the X38/C2D/Vista side? I will post exact specs. – Samir – 2013-11-07T23:55:54.023

What was the CPU usage during the transfers ? Did the CPU usage go up when the speeds go down ? AFAIK Realtek chips hand off some processing to the CPU, so if the CPU has to do anything while it's transferring data, it will slow down the data transfer. – Lawrence – 2013-11-08T00:46:29.443

12Have you considered booting both machines from Linux live-CDs (two identical boot disks) to rule out software issues, and transferring a file from RAM-disk to RAM-disk to rule out HDD issues? – Moshe Katz – 2013-11-12T03:55:05.617

do you have write caching turned on the hard drives? – Rex – 2013-11-12T06:31:47.197

When launching 'iperf', what does it report on each side as the TCP window size? Also, have you asked the 'iperf' server to guess at the MSS? Is the reported value reasonable? – ewhac – 2013-11-13T02:06:03.277

Just got back to your question. The Realtek B version in the Vista machine is I think the culprit. If I recall correctly from past experience the A & B version max out around 35% of Gigabit, while the C and D versions can achieve around 90-95%. That correlates nicely to your findings. Double check by following Moshe Katz's suggestion. Linux Live-CD or USB stick on both machines and run iperf from there (most Linux versions have iperf on-board as standard). – Tonny – 2013-11-28T22:07:51.147

1I wanted to throw in my 2 cents here... my first: did you disable all virusscanners? my second: did you try reversing the cable to see if your problem also reverses (switching the end of the cable to the other computer and visa versa). If you get fast download but slow upload then the problem is in the wiring. (i.e. your pairs are not twisted together but with other pairs) – Rik – 2013-11-30T01:17:47.500

1@MosheKatz I just did that. You can see by the results I just posted (see update 8) that I got great throughput in both directions with Ubuntu. – Samir – 2013-11-30T20:41:48.403

@Rex Write caching is enabled on both local and remote hard drive. Are you saying I have to disable it in order to improve performance? How much does this really effect hard drive performance? – Samir – 2013-11-30T21:13:55.420

1@ewhac The TCP window size on the local machine usually differs from the window size on the remote machine. The default value is 0.06 MB on local (win 8) and 0.01 MB on remote (vista). Do they have to be the same value? How do I ask it to guess the MSS? Would that be the -m switch ("print maximum segment size")? I usually don't set any of that. Why would I have to? – Samir – 2013-11-30T21:39:50.777

@Tonny I think we can rule out Realtek 8111B as the culprit (see update 8). I got over 90% utilization between revision B and C using Ubuntu LINUX Live system on both machines. – Samir – 2013-11-30T21:42:53.887

@NothingsImpossible Yeah, I kind of got that now (see update 8). The question is, how is it possible to get full blown Gigabit speeds on Ubuntu LINUX but not on Windows Vist/8.1. Any ideas? – Samir – 2013-11-30T21:45:48.857

1@Sammy Did you check if you removed all other protocols from the lan-connection in Windows (e.g. the "QOS Packet Scheduler" and possible "Virtual"-drivers.) It's a long shot but they can get in the way. Also did you disable the virusscanners on both machines (i know, also a long shot but that's where we are now :) – Rik – 2013-11-30T21:59:22.793

2@Sammy wow, this topic is getting big :) I saw you tried Linux on both ends. But did you already try Linux <--> Win8 and/or Linux <--> Win.vista ? If one of them has full speed in both direction you know the other is at fault (which would show in the other test) and you/we can concentrate our efforts on that machine. – Rik – 2013-12-07T01:10:24.020

Why don't you try your test using UDP in both directions and compare the results with your TCP numbers? – None – 2013-12-06T23:55:55.690

Concerning test 10, disk speed. Operating systems today use pre-mapped or fully mapped DMA for external device copying, so my guess is that the data is copied to RAM before it get stored on disk. This would explain higher throughput than what your disk is capable of. – Mogget – 2013-12-09T20:16:20.030

Answers

6

Based on your response:

@ewhac The TCP window size on the local machine usually differs from the window size on the remote machine. The default value is 0.06 MB on local (win 8) and 0.01 MB on remote (vista). Do they have to be the same value? How do I ask it to guess the MSS? Would that be the -m switch ("print maximum segment size")? I usually don't set any of that. Why would I have to? – Sammy Nov 30 at 21:39

The TCP window size represents the maximum amount of data the TCP stack will squirt down the wire blind before stopping and waiting to receive acknowledgements from the remote machine -- in other words, the maximum amount of unacknowledged traffic on the wire. The fact that the TCP window on the Vista machine is much smaller than on the Windows Se7en machine supports the theory that you're not filling the pipe enough before stalling and waiting for ACKs.

Therefore, the first thing I would try is increasing the TCP window size on the Vista machine using the -w argument. 0.01MB is a somewhat unhelpful unit; I'm guessing it's 16KiB. So start with 16KiB and run a test:

iperf -c -w 16K ...

Double the window size and repeat the test:

iperf -c -w 32K ...

Hopefully you should observe a speed increase. Continue doubling the window size until you no longer see significant speed increases.

ewhac

Posted 2013-11-04T22:29:29.637

Reputation: 511

4

As previously stated, you will need to modify the TCP Window size in iperf to get higher throughput over low latency high speed links. Different versions of Windows (or iPerf) may have different default window sizes. Try "-w 256k" to start with on both the client and server.

Can you confirm the arrow direction of your charts? In iPerf, data is pushed from the client to the server (opposite of what I usually think).

You can rule out the hard drives as the cause since iPerf does not touch the hard drives.

I think you already verified that you are running the latest NIC drivers. There should be no need to muck around with manually setting the speed/duplex. Same with jumbo frames -- they are not worth the hassle with modern hardware.

Verify that Large Send Offload (LSO) and Large Receive Offload (LRO) are enabled on both NICs. Some NIC drivers use different names (or multiple options that control this behavior) so you may have to hunt around. Usually the default is to have them all enabled.

Michael

Posted 2013-11-04T22:29:29.637

Reputation: 121

3

I think u need to read a little bit more about how iperf works. If you don't set the correct tcp window size your results will vary a lot. I believe its the -w switch. This website will assist with calculating the optimal TCP window size. You need to know the RTT and bandwith to calculate it. http://www.kehlet.cx/docs/tcpwin.php

Also try other tools like "LAN Speed test" lite which will just do up and down transfers between shares at either end. Its results are quite close in the testing ive done with it. Also make sure u check what your hard drives max r/w speeds are.

PeterJ

Posted 2013-11-04T22:29:29.637

Reputation: 104

Can I use ping command to figure out the RTT? If I ping the remote I get less than 1 ms. But it doesn't tell me how much. Is there some other tool I can use, one with greater precision? I know I can't use 0 ms in the formula. – Samir – 2013-12-06T23:13:07.707

I used a ping program called hrping. As for LAN Speed Test Lite, it was not as good as Iperf. Maybe it gets better once you have the LST Server installed on the other end. But I didn't feel like registering to use it or paying for a license.

– Samir – 2013-12-08T14:23:06.100

3

A couple of point which may help you out.. TCP IP stack is now implemented differently in post Windows 7 releases. I would look closely at my TCP optimisations, there may not be much in it between your two boxes but still worth tweaking some settings on your vista Box.

The use of netsh int tcp set global congestionprovider=ctcp has been depreciated. In order to set or change the congestionprovider the following command must be used:

read the article here: http://forums.speedguide.net/showthread.php?280646-When-will-TCP-Optimizer-support-Windows-8-amp-Windows-Server-2012

netsh int tcp set supplemental custom 300 10 ctcp disabled 50 Then type: netsh int tcp set supplemental custom

For further details about the above command simply type: netsh int set supplemental

To check which congestionprovider your currently using use the following: netsh int tcp show supplemental

But only Win Server 2012 can create custom templates CTCP maybe an issue.

Also Auto-Scaling could be a factor.

Maybe worth taking a look at your drivers maybe in need of upgrade/downgrade. see which ones work best.

Another thought is packet-sizes being written to HDD.. What size are your HDD sectors formatted to 512b ~ 64K it could be caching bottleneck.. Worth consideration when using GBit speeds - or test with SSD disk instead!

Have you looked at enabling Jumbo Packets (if this apply's to your NIC's)

majika

Posted 2013-11-04T22:29:29.637

Reputation: 31

Is it possible to disable window size scaling and have the RWIN value set manually to 64K? – Samir – 2013-12-08T14:56:13.340

I just read that the RWIN value is changed dynamically on modern Windows systems. Is this true? That RWIN can't be a fixed value? – Samir – 2013-12-08T15:16:54.733

2

Great results with Ubuntu LINUX

This sounds quite familiar... you get inexplicably bad iperf performance in Windows, but the same hardware works fine with Linux.

After fighting this battle over and over, I have come to the conclusion that iperf in Windows is flaky. I don't know why... honestly, I've quit caring because I know I can always get sane results from linux.

So if you want to know why you're getting such bad performance, look no farther than the application.

Mike Pennington

Posted 2013-11-04T22:29:29.637

Reputation: 2 273

2

Something to try is to stop the MMCSS service. Yes, you will lose your audio temporarily, but it could be that something is activating it, causing the network activity to be throttled so that the network activity doesn't drown out other activity.

See here for some info about MMCSS and network throttling: http://blogs.technet.com/b/markrussinovich/archive/2007/08/27/1833290.aspx

It can be tweaked as per these instructions: http://support.microsoft.com/kb/948066

I doubt it's the cause in this case, as throughput would probably be lower, but after banging my head against the wall for a long time with my own throughput problems on Vista, I discovered this was the cause. I set NetworkThrottlingIndex to 70 and I finally got the throughput I expected.

Mark Sowul

Posted 2013-11-04T22:29:29.637

Reputation: 2 877

2

One thing you have not tried is removing Remote Differential Compression on Vista.

My thinking is motivated by the extensive CPU use on Vista. It might be that the Vista network driver does not know how to offload this to the network card, while Windows 8 does it much more efficiently.

For full details see the article :
Prevent slow file copying over the network in Vista by disabling Remote Differential Compression.

But in a nutshell, just uncheck Remote Differential Compression in Control Panel -> Programs and Features -> Turn Windows features on and off, then reboot.

image

harrymc

Posted 2013-11-04T22:29:29.637

Reputation: 306 093

0

Previously I chase my tail with exactly the same problem for a while! Slow transfer speeds in one direction, in my case outbound (uplink).

Windows 7 Pro, Celeron J1800 with Realtek Gigabit 8111C builtin lan card. QNAP 453a and MacBook Pro on the other end.

When measured via Iperf3 I was getting 112 mbps with the my Windows 7 set as a client (CPU usage at 25-30%). And only 39-41 Mbps when set as a server, with heavy CPU usage between 50-100%. So bad that the PC would freeze at times of bandwidth testing.

Regular file transfer capped at 45mbps max no matter if I was uploading or downloading files to my NAS or my MAC.

I was getting nothing more than 35-45 megabytes per second. Pretty frustrating!

Ended up being a bad lan card driver. I was obsessed with updating drivers and always updated my drivers when new one become available. Guess what, after several updates my lan card slowed down.

Some of you might say, just delete the old driver and install the new one. Simple, ah? I tried and tried, It did not work for me.

Here is my solution:

Installed windows from scratch with OEM drivers from the manufacturer's website. As well I did the following:

Under Device Manager / Lan card / Advanced settings / Disable everything except FLOW CONTROL.

Under Windows Features, Disable Remote Differential Compression.

Now average speed is between 80-100 Mbps.

Sorry for the poor quality pics.

enter image description here

Gi Cakov

Posted 2013-11-04T22:29:29.637

Reputation: 1

-2

Linus from Linus Tech Tips on Youtube had the same problem, and I apologize but I have forgotten what the solution was. It is a recent (as of 4/20/2016) video, and I may be wrong but I think it ended up being a case where a single thread on a processor is the limiting factor, so if you have older hardware and you are trying to hit 1 Gbps, it is actually the CPU that is the problem, if it offloads the bulk of the work to the CPU, which most on-board NICs do these days. I could be mistaken, it is a recent video of his, I highly encourage you to check his stream. For what it is worth I am running into the same problem on my gigabit internet connection.

Prestaeus

Posted 2013-11-04T22:29:29.637

Reputation: 9

"I have forgotten what the solution was" - not much of an answer then is it? – DavidPostill – 2016-04-21T10:01:44.980

1So take the time to determine what the solution was. This answer is not helpful, you can still be answer banned, for submitting bad answers even if they are marked as a community wiki – Ramhound – 2016-04-21T13:51:05.767