3

I am having very bad performance with NFS between two debian stretch machines, and I can't find a solution. I am using rsync at the server side in the exports file and tcp,hard,intr,rsize=32768,wsize=32768 at the client side as options when mounting. Both servers are VMs running under Hyper V and sharing same internal network switch.

Performace is acceptable with using large files, but with rsync performance is very low. Takes a few seconds to copy a tar file (Size 2 GB) and several minutes to extract the same tar file in the same destination folder. Tar without any special options, just tar xvf .

EDIT

I am using 4 TB Western Digital WD40EFRX SATA drives. These 4 drives are assigned directly to the Hyper-V VM as physical discs. These 4 discs, are made into a RAID 5 ext4 filesystem in the Debian Stretch NFS Server. The test files I am using are 40.000 files of different sizes adding a total of 2 GB. (mostly small size files, images and text and a few jar files).

Both servers are sharing the same Hyper-V External Virtual Switch, connected to an Intel I211 Gigabit Network Adapter

Ping sample from NFS client to NFS server:

PING xyzserver (192.168.1.49) 56(84) bytes of data. 64 bytes from xyzserver (192.168.1.49): icmp_seq=1 ttl=64 time=41.2 ms
64 bytes from xyzserver (192.168.1.49): icmp_seq=2 ttl=64 time=5.48 ms
64 bytes from xyzserver (192.168.1.49): icmp_seq=3 ttl=64 time=29.7 ms
64 bytes from xyzserver (192.168.1.49): icmp_seq=4 ttl=64 time=64.2 ms
64 bytes from xyzserver (192.168.1.49): icmp_seq=5 ttl=64 time=43.0 ms
64 bytes from xyzserver (192.168.1.49): icmp_seq=6 ttl=64 time=11.5 ms
64 bytes from xyzserver (192.168.1.49): icmp_seq=7 ttl=64 time=29.4 ms

EDIT2

I believe it has nothing to do with Debian 9. Even with Debian. Also not only with NFS. We have tested with Windows machines also and with CIFS and performance is quite similar.

Performace when using large files is good (specially with NFS) but with small files (source code, etc.) it can get really slow (even under 1 MB/s).

Any ideas of what could I be doing wrong? (Old question)

We are considering NOT using either NFS or CIFS.

Any idea of what we could do in order to get decent performance? (New question)

Having a fast RAID5 disk subsystem with a fast server and having two VMs in the same server seems ridiculous we cannot find a way to rsync files or untar files in the shared filesystem a decent speed.

Any suggestions of what I could use? (hardware, software)

LincolnP
  • 207
  • 2
  • 7
  • 2
    There's a lot that can be done to tune NFS for some scenarios, but it's really best to tune things from the bottom up and the top down. This equates to making sure NFS isn't performing horribly (ioping tests fo the backing storage VS NFS over a network can give you an idea as to what's going on). Tuning NFS is difficult, because it means tuning the storage layyer under it, tuning the network it uses, and tuning the NFS server/client itself. It should probably be tuned in that order, so you don't attempt to solve lower level problems with higher level tuning. – Spooler Apr 08 '19 at 16:08
  • 1
    What I'm getting at is: Post your storage configuration. What filesystems are in use, all the way down to the physical disks? What networking is in use, all the way down to your physical NICs? and what does your NFS configuration look like? If you post those, something may show up as "obvious"... maybe. But without that, we're just guessing. – Spooler Apr 08 '19 at 16:09
  • How many files? Tens to hundreds of thousands of files is a very large number of IOs to file system metadata, and hits the dentry cache hard. – John Mahowald Apr 08 '19 at 16:17
  • We are preparing two servers with same configuration but instead of sharing with NFS sharing with CIFS. I will post the results as soon as we have them. This might help bringing light to this issue. – LincolnP Apr 08 '19 at 18:38
  • Just finished doing tests with CIFS also. Speed is a bit better for small files, but still unacceptable. – LincolnP Apr 10 '19 at 17:47

1 Answers1

1

We found no solution to these problems performance problems with CIFS and NFS under Hyper-V.

we finally decided to move to PROXMOX and everything is working so much better. Not sure why, probably Proxmox virtualization of Debian since it's also based on Debian is far more efficient that Hyper-V, which most likely is more efficient for Windows virtualization?

LincolnP
  • 207
  • 2
  • 7
  • From your ping output above, it looks like you had some significant problems with latency from the NFS client to server. This would certainly cause small file performance to degrade significantly, and nnot affect large files as much. As to exactly where in that stack the latency was, you would have to run several intermediate tests to isolate it while running on hyper-v virtual networking. You might try proving me wrong by posing an "identical" ping test from the NFS client to your new NFS server hosted on proxmox. A wired LAN connection should really be less than 5ms latent between nodes. – Spooler Apr 15 '19 at 17:34
  • Thanks @Spooler, you are absolutely right. Furthermore, it´s not a wired LAN, it is an internal network (also tested with Private) within the same Hyper-V server, which means there is no routing at all in the LAN. My guess is with the virtual device emulation that Hyper-V is doing makes it really slow, but I cannot confirm. – LincolnP Apr 15 '19 at 18:19
  • That does seem pretty damning for the hyper-v virtual networking stack in this scenario. I don't ever use hyper-v, so I can't speak to any tunables that might be relevant here. But using proxmox is a very acceptable solution imo :) – Spooler Apr 15 '19 at 18:21
  • You might "accept" your answer so this question gets resolved. – Spooler Apr 15 '19 at 18:21
  • Last comment on Windows 10 Virtualization on Proxmox. Works really well. surprisingly fast. – LincolnP Jun 26 '19 at 08:27