1

i'm building an email system and considering some filesystem that support online-resize, safe when power-outage, and does not restrict number of files per directory(or at least can hold some millions file per directory)

I want to know which filesystem is best for this situations. Can you help me,pls? Thanks in advance!

PS: i'm layering for my email storage as:

Layer-3: Filesystem(XFS,JFS,Btrfs,Reiser4,...) on top of drbd(/dev/drbd0)

Layer-2: DRBD(Dual-primary mode) on LVM2's logical volume

Layer-1: LVM2 on physical volume(/dev/sdc,/dev/sdd,...)

Layer-0: physical volume(sdc,sdd,...) is Hardware RAID10(with "Write-cache mode" is enabled) (each "physical volume" actually is 4 HDD)

Another question: Do you see something wrong in my design?

Edited: i'm using Ubuntu 12.04 LTS with 3.2 kernel.

Locke
  • 81
  • 3
  • 7

6 Answers6

4

XFS has been a reliable workhorse for me for many years. The mail system I'm talking about is happily running Cyrus IMAP server with 50 000+ accounts (at peak it was nearly 100 000 accounts) and about 300 000 mail boxes. There are tens of millions of mail files. Everything works smoothly and server load is mostly idle.

But... couple of million of files per directory? What kind of mail system are we talking about? XFS might handle that somehow, but no file system has been designed for that kind of behaviour.

Janne Pikkarainen
  • 31,454
  • 4
  • 56
  • 78
3

My recommendation of that list would be XFS. You didn't provide the Linux distribution information, but assuming CentOS or Red Hat, XFS is now somewhat integrated. It's a mature filesystem and offers online defragmentation and can be grown (not shrinked) on the fly. I rarely hear about JFS... Reiser is a gamble and has lost tremendous mindshare... Btrfs isn't mature enough to trust for this. Is there anything wrong with ext4?

Please also reference these posts:

Production-ready, highly reliable filesystems on Linux: ext4 ext3 XFS or JFS (or ZFS)?

What is the best filesystem for Maildir folders storage?

Community
  • 1
ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • XFS may be easiest. Is there a reason you're not interested in ext4? – ewwhite Mar 12 '12 at 15:30
  • Hi, i'm using Ubuntu 12.04 LTS with 3.2 kernel. As fas as i know, ext4 support offline resize, not online resize. Maybe my info is obsolete?? – Locke Mar 12 '12 at 15:32
  • I read something and find that xfs & btrfs is not safe on power outage, can lose much of data. You could confirm it? Thanks! – Locke Mar 12 '12 at 15:40
  • Make sure your RAID write-cache is battery or flash-backed. Also use a UPS... – ewwhite Mar 12 '12 at 15:44
1

If the lower layers are all redundant and properly cared for, the only considerations in choosing a filesystem are speed and reliability of the abstract filesystem (since one can assume that nothing physical breaks with such redundancy on lower layers).

For those requirements, good old ext3 still wins (with metadata journaling only) - XFS is as indicated not good with OS failures, and ext4 is also not mature enough for these taks - I have had ext4 volumes exhibit critical FS errors in the past, with no logical reason for them.

That said, where does the requirement for "millions of files per directory" come from ?

Email is stored as either mbox or maildir (or, more recently, dbox); none of those require millions of files per directory - far from it, in fact: maildirs hold one logical mailbox folder per directory, and I don't know anybody who has more than a couple thousand files per folder.

mbox is one huge file per mailbox, period, and simply not suited for today's email storage.

dbox, the new kid on the block, is supposed to outperform both mbox and maildirs in the majority of cases, but again, does not store "millions of files" in a single directory.

adaptr
  • 16,479
  • 21
  • 33
  • hi, i use maildir, and normally i receive > 100 mails/day (from colleagues, partners, customers and from monitoring system) --> 50,000 mails/year. I don't assume that my email users will delete emails or reorganize it into folders, we can not sure about it. – Locke Mar 12 '12 at 16:56
  • I'd disagree about XFS's resiliency in system crash situations. I've never lost data on an XFS filesystem throughout numerous crashes and power outages. I think the key is to have a good RAID controller and flash/battery-backed cache. – ewwhite Mar 12 '12 at 16:56
  • @ewwhite: my RAID controller does not have battery-backed. Should i disable "Write-cache mode" instead? – Locke Mar 12 '12 at 17:21
  • @minhcd: a non-battery-backed RAID controller has no purpose. better turn it off and use software RAID. It's usually faster, and far safer. – Javier Mar 12 '12 at 18:56
  • I recheck with my server team and be confirmed that my RAID controller has battery-packed & my servers are allowed to use UPS. Thanks you! :D – Locke Mar 13 '12 at 03:21
0

I found that because of using DRBD's "Dual-primary mode", my filesystem-on-top only have 2 choices: GFS2(from RedHat) and OCFS2(from Oracle). I'm curiously to know which option is more stable on Ubuntu 12.04 LTS...

I heard some problems about OCFS2; I'm not sure OCFS2 v1.6 could be better?

Edited: I decided to go with GlusterFS mirror on top of XFS. Thanks you all!

Locke
  • 81
  • 3
  • 7
0

Some of the setup you list matches a number of email systems I work on. The one large except is that we have been slowly pulling out DRDB wherever possible due to the issues it causes. For 99.99% of the time, it is perfect and fallover works nicely, but those times when it doesn't just cause a whole heap of pain.

As others have commented, the number of mails per directory will slow things down, massively. We've gotten around this by offering automatic archiving solutions. On a very basic level, its a script that connects over imap and then uses imap to move the older mails into an archiving subfolder broken out by dates. For most mailboxes, this allows things to stay relatively clean but still organized.

For the filesystem choice, the tuning parameters make a fairly large difference depending on which mailbox format you choose. With maildir, noatime can make a difference for with some clients although the likes of Dovecot with its built in cache negate this gain these days.

Ryaner
  • 3,027
  • 5
  • 24
  • 32
0

For the LVM setup, see LVM dangers and caveats which covers various points you need to get right for LVM to work well.

Community
  • 1
RichVel
  • 3,524
  • 1
  • 17
  • 23