8

Assuming I'm using Dovecot and it's maildir format to save and access mails on a server. How do I avoid race conditions while copying a maildir?

I did read some tutorial about backups and using maildir but it did no see anyone writing about this. They just use cp or rsync to copy the directory to another place. Is it impossible that maildir become an inconsistent state by copying or do I need some kind of locking?

EDIT: I want to make regular backups but I don't think that it realy metters for the question. I know that I could stop the mail server (Dovecot and Postfix) but I think it should be possible without doing this. As far as i know maildir support concurrent access by different applications.

JojOatXGME
  • 289
  • 3
  • 9
  • Stop dovecot, and your mta, then make your copy? Are you trying to do this on a live system? Do you have lvm setup to make snapshots, or some other filesystem snapshot tool? What is this copy for? Are you migrating, is it a backup? – Zoredache Feb 19 '16 at 17:38
  • @Zoredache The server is running on a VM. I don't think that there is any snapshot tool in the filesystem. But I would also not know how such tool could ensure consistency. Look at my edit for answers of your other questions. – JojOatXGME Feb 19 '16 at 18:02
  • `how such tool could ensure consistency` - You get all the applications commit changes so your filesystem is consistent, take a snapshot, then backup against the read-only consistent snapshot, and delete when you are done. – Zoredache Feb 19 '16 at 18:15
  • @Zoredache There is something to combine multiple operations to a single transaction in a file systems? I have never seen it before. Does applications like Dovecot and Postfix use it? As far as I know there is no function to use such functionality in modern standard libraries like C++11. I know that filesystems have something like commits to keep the filesystem itself consistent. But it doesn't mean that the maildir is consistent, too. – JojOatXGME Feb 19 '16 at 19:19

2 Answers2

10

I just read the documentation for Maildir in Dovecot and few other documents about Maildir and Maildir++. I hope that I did not miss anything important.

Maildir is designed to work without locks. Most required operations are atomic on modern file systems. This means that you do not have to care much about race conditions like inconsistent read. But there are still some issues if you want to backup a Maildir while the mail-server is running.

Issues

  • Backing up and restoring tmp/ is useless. Each Maildir contains the directories new/, cur/ and tmp/. Directory tmp/ contains mails which are being written to disk at the moment. They are moved to new/ when they are written successfully. This means that files in tmp/ may not be complete yet. Even if the file is complete, the process which was writing the file is not running anymore after restoring the backup. This means if such file is restored, it will never be added to the mailbox and it may never be removed.

  • It may be sensible to exclude dovecot-uidlist.lock from backup. Dovecot uses an extension of Maildir called Maildir++. This extension need locking. Acquiring the lock for reading is not required by the extension but it may be sensible to exclude the lock-file from backup.

  • Acquire dovecot-uidlist.lock or use snapshot of filesystem. Basically, you can just copy the directory but it is possible to miss some mails by race conditions. The reason is that listing and copying the content of a directory (recursively) is not atomic. This means that it is possible to miss an email while creating the backup when a user is changing a tag (e.g. seen/unseen) or moving the mail. To handle such situations, Dovecot acquires the lock-file (same as above) for every action. If you are using Dovecot, you can solve the problem by acquiring the lock before creating the backup. This can be done with /usr/lib/dovecot/maildirlock. Another possibility is to use snapshots of your filesystem. Since creating a snapshot is atomic, the problem does not occur on snapshots.

As a quick summary: If you want to make a backup while Dovecot is running, you should first acquire the lock-file dovecot-uidlist.lock or create a snapshot of you filesystem. Then, you can copy the Maildir. It may be sensible to exclude tmp/ and dovecot-uidlist.lock from your backup.

JojOatXGME
  • 289
  • 3
  • 9
  • 1
    My strategy for the Maildir storage was to keep it in a separate logical volume (via LVM). Then I could snapshot the volume (instantaneous) and then make my backup from the snapshot volume. Never had a problem. – tgharold Mar 18 '16 at 10:19
  • @tgharold Yes. As far as I understand it, you should not have the third poroblem when using snapshots. Every change in `new/` and `cur/` seems to be atomic. The only thing would be that you may backup the lockfile and the files in `tmp/`, too. This is nothing which would brake your backup. – JojOatXGME Mar 18 '16 at 13:19
  • @JojOatXGME When you say "If you want to make a backup while Dovecot is running, you should first acquire the lock-file dovecot-uidlist.lock", how exactly is that done? – TommyPeanuts Apr 15 '20 at 10:08
  • 1
    At the time of writing, the dovecot package included the binary `/usr/lib/dovecot/maildirlock`. This small executable can be used to acquire the lock. – JojOatXGME Apr 16 '20 at 15:40
7

An updated answer to this question is to use dovecot's doveadm or dsync command. example:

dsync -f -u <user> backup maildir:<backup_location>

There is a dovecot-backup shell script which with adds some nice wrapping around dsync to loop over all the users mailboxes, tar up the files, prune older backups and send email alerts if errors occur.

TownCube
  • 155
  • 5
Toushin
  • 71
  • 1
  • 1