20GB+ worth of emails in my /home what is a better solution for that?

Question

My email storage requirements are outgrowing anything reasonable with respect to local mail storage. As we speak 99% of my home partition is filled with personal mail in Thunderbirds mail dirs. Needless to say, this is just painful, badly searchable and as history has proven me that backups work, but Thunderbird is capable of loosing a lot of mail very easily.

Currently I have an remote IMAPS server (Dovecot) running for my daily mail, accessible from anywhere, which from my own practice works efficiently up to about 1000 emails. Then some archive directories should be used to move mail around.

I have been looking into DBMail, but I wonder if I make my case worse or better which such solution. None of the supported database employ string deduplication or string compression out of the box, so is this going to help me with 20GB+ mail? What about falling back to a plain old IMAP server? A filesystem like ZFS would support stuff like GZIP transparently, which could help.

Could someone share their thoughts? The 20GB mostly consists of mailinglists, and normal mail. Not things like attachments.

---

To add some clarifications;

• As we speak, my mail is not server side indexed at all - only my new mail arrives at a remote IMAP server. It is all local storage from former POP3 accounts, local mirrored Gmail and IMAP accounts. In my perspective it is not Thunderbird that sucks, its fileformat that sucks.

• Regarding the 1000 mails. On the road I am using Alpine and MobileMail, quite happy with both of them, but some management is required to actually manage the mail. Sieve helps a lot with that, but browing through 10.000 e-mails is not fun, especially not on a mobile client.

• I am quite happy with Dovecot, never had any issues with it. I just wonder if this is the way to go. Or if there are any other better solutions.

• What my question is: what is the best practice solution that allows 20GB+ mails and is -on demand remotely accessible, easy to backup and archive worthy. It doesn't need to be available 24x7.

The final approach I took was installing a local IMAP server (Dovecot), configured it for being my archive, using the following guide: http://en.gentoo-wiki.com/wiki/Dovecot/InstallThunderbird

Answer 1

IMAP, unlike its predecessor POP, is designed so that the "master" home of the email is on the server. Anything that may be stored by the IMAP client (whether it's Thunderbird or a webmail client or anything else) is treated as a cache for what's on the server. As such, it is completely discardable and non-valuable. If Thunderbird doesn't manage it well and corrupts it or loses it, it is of little concern. It will be rebuilt from the information on the server. At worst if it's really corrupt it may need to be deleted. Don't back it up, and don't worry too much about it.

A good well-managed Dovecot server will NOT lose or corrupt your email in any way. (But I can't guarantee that, of course... bugs and disasters are always possible.) I've built several very large email servers using Dovecot.

Your claim that a Dovecot IMAP server "works efficiently up to about 1000 emails" sounds wrong. My personal email is stored on a Docevot IMAP server. I have over 100000 individual emails there, excluding mailing lists. This is using regular Maildir storage on a regular ext3 filesystem. I could upgrade to dbox but there hasn't been a need. It works fine.

20GB is not an enormous amount of storage these days. I don't even think I'd be worrying at all about data deduplication and compression for that amount of data.

Answer 2

I would suggest you to make a backup of your email messages in the maildir format, (a single individual file for each message).

You could save them as either EML or PDF. For security you could compress and encrypt them using the ZIP format, one ZIP file for each message because of the following:

Why single files.

• They are easy to move around, copy and synchronize
• The chances that the files get corrupted are minimal as oposed to storing them in a big monolithic file (mbox, pst , db, dbx )
• You can manage them using only windows explorer

You could save your exported files to a portable hard drive or burn them to DVDs to have an offsite copy.

The Dovecot server supports both the mbox and maildir formats and using the rsnapshoot you can backup the /var/mail folder. The problem of this approach is the time that is needed to download as oposed to only downloading new email messages.

MsgExtract can export messages from IMAP and from Thunderbird and save them as single files with optional ZIP compression. If needed you can export those saved files back to an IMAP server, Thunderbird or Outlook.

(Disclaimer, I am the author of MsgExtract)