31

I am planning my new shared storage infrastructure for a small webserver farm. Therefore i ran a lot of tests with a lot of NAS filesystems. By doing so, i got some unexpected results and would like to know whether somebody here can confirm this.

In short: Samba is hugely faster than NFS and GlusterFS for small file writes.

Here what i did: I ran a simple "rsync benchmark" with a lot of files to compare the write performance for small files. For easier reproduction, i re-ran it just now with the contents of the current wordpress tar.gz.

  • GlusterFS replicated 2: 32-35 seconds, high CPU load
  • GlusterFS single: 14-16 seconds, high CPU load
  • GlusterFS + NFS client: 16-19 seconds, high CPU load
  • NFS kernel server + NFS client (sync): 32-36 seconds, very low CPU load
  • NFS kernel server + NFS client (async): 3-4 seconds, very low CPU load
  • Samba: 4-7 seconds, medium CPU load
  • Direct disk: < 1 second

I am absolutely no samba guru (i think my last contact was with samba 2.x), so i did not optimize anything here - just out-of-the-box config (debian/squeeze package). The only thing i added "sync always = yes" which is supposed to enforce sync after write (but seeing those results..). Without it, the tests where about 1-2 seconds faster.

All the tests where run on the same machine (self-mounted it's NAS export), so no network delays - pure protocol performance.

Side node: As file system i used ext4 and xfs. The above results are with ext4. xfs performed up to 40% (less time) better. The machines are EC2 m1.small instances. NAS export are on EBS volumes, sources (extracted tar) on ephemeral disk.

So here we go: Can anybody explain to me why samba is so much faster?

Also: Is NFS performance with the kernel server supposed to be that horrible (outperformed by the GlusterFS NFS server) in sync mode? Any idea how to tune this?

Thanks, L

Linus Ardberk
  • 311
  • 1
  • 3
  • 3
  • Forgot: NFS measures are the best of a couple, where i played around with rsize, wsize, noatime, noac, udp vs tcp and so on. – Linus Ardberk Mar 21 '12 at 20:12
  • 3
    +1 for research and benchmarks! – Bigbio2002 Jun 12 '12 at 19:28
  • 3
    By the way, did you reboot each machine between tests to clear out the disk read cache? – hookenz Mar 03 '13 at 23:38
  • 1
    We have observed the same differences in CIFS vs. NFS performance during development and testing of SoftNAS. I can confirm that async NFS is much faster than sync NFS, and Samba does edge out NFS for the benchmarks we have run using ATTO benchmark software. –  Mar 03 '13 at 23:09

1 Answers1

5

We use Samba extensively, and I've always found it to typically be faster than NFS in almost all cases (and I only say almost because I haven't compared them enough).

From what I've seen after a couple of packet captures, the SMB protocol can be chatty, but the latest version of Samba implements SMB2 which can both issue multiple commands with one packet, and issue multiple commands while waiting for an ACK from the last command to come back. This has vastly improved its speed, at least in my experience, and I know I was shocked the first time I saw the speed difference too - Troubleshooting Network Speeds -- The Age Old Inquiry

Community
  • 1
Univ426
  • 2,139
  • 14
  • 26