3

We're seeing performance problems on a file system that contains ~500 M files that are relatively small (typically tens of kb) and ~150 K directories. Files access is predominantly reads and writes are fairly rare. Files are mostly stored in a hashed directory hierarchy with ~30K files per directory, but in some cases there can be up to 500 K files in one directory.

The server shares the file system to ~10 client machines.

Getting directory listings is often slow and sometimes also reading files by absolute paths, even locally.

This is a physical server that runs Ubuntu 12.04.4 (kernel: 3.8.0 / x86_64) and we use ext4 on a hardware RAID-6 volume.

What would be a recommended file system setup in this scenario?

  • is there a file system particularly well suited for this case (e.g. ext4, xfs, btrfs, ...)?
  • what kind of RAID configuration (e.g. sw vs. hw, RAID level etc.) should we use?
  • what about the file sharing setup (technology [e.g. NFS vs. competition], configuration, ...)?
marko
  • 131
  • 1
  • 2

2 Answers2

2

When you have a problem like this, you have to:

  • obtain all requirements (latency, bandwidth, redundancy, reliability, security, required features...)
  • analyse the current systems. If they are none, create test environments. Understand how all components work. Understand the current and the expected load.
  • add system monitoring (with graphs) for both production and test systems. Monitor at least CPU usage, network usage and Disk I/O usage.
  • create test servers and load test them. Load test with synthetic benchmarks not only with micro-benchmarks.

Use stable versions of recent OS with latest stable kernel.

Mircea Vutcovici
  • 16,706
  • 4
  • 52
  • 80
  • Thanks. Do you have recommendations for a disk i/o benchmarking tool that you've personal experience with? – marko Jul 02 '15 at 05:51
  • Depends what you want to test - the disk, the disk controller, filesystem, etc... you can use `hdparm`, `dd`, `bonnie++`, `fio`. See: http://wiki.opensfs.org/Benchmarking_Basics and http://serverfault.com/questions/219739/i-o-performance-benchmarking-linux – Mircea Vutcovici Jul 02 '15 at 14:19
0

What would be a recommended file system setup in this scenario?

is there a file system particularly well suited for this case (e.g. ext4, xfs, btrfs, ...)?

I think ext4 is the best choice here in terms of safety. In terms of reading performance RaiserFS(/4) could be better:

https://unix.stackexchange.com/questions/28756/what-is-the-most-high-performance-linux-filesystem-for-storing-a-lot-of-small-fi

http://www.phoronix.com/scan.php?page=article&item=linux_39_fs&num=1

filesystem for millions of small files

what kind of RAID configuration (e.g. sw vs. hw, RAID level etc.) should we use?

You need to:

  • shorten latency / access time
  • reduce number of reads/writes (noatime,nodiratime,realtime,barrier=0)

Every RAID level, which has possibility to do parallel operations will be good choice, for example RAID 10 (Linux software raid) has a lot tuning options.

what about the file sharing setup (technology [e.g. NFS vs. competition], configuration, ...)?

I can't give you numbers here, file sharing is also quite wide term. I've had very good experience with FTP when dealing with lots of files (not FTPS - because negotiating TLS for every transfer takes time), better tunnel the traffic. It seems that the best division between flexibility of the protocol (what u can do) and protocol overhead (how much time it takes, space in the packet) when coping small files.

Community
  • 1
Michal Sokolowski
  • 1,461
  • 1
  • 11
  • 24
  • We'd basically need a distributed file system akin to NFS, but we're open other options if something else would be a better fit in our case. – marko Jun 26 '15 at 20:13
  • Sorry, but it means nothing. What do you need? Block device sharing support, extended ACL's, xattr, compression? It sounds like you didn't analyze it yet. – Michal Sokolowski Jun 26 '15 at 20:49
  • We need to share files across multiple systems. Currently, there's only one system that changes the filesystem at a given time and the filesystem contents need to be readable from two other systems. Compression or ACLs are not needed. – marko Jul 02 '15 at 06:00