1

I have an old server running Sendmail and Dovecot 0.99. There are about 50 email accounts on this server, associated with various domain names hosted on it. The email data is stored in MBOX format on the old server.

I need to migrate these email accounts to a destination server running Postfix and Dovecot 2.x. The destination server uses Maildir format.

Migration of emails themselves is easy using imapsync. HOWEVER, the POP UIDL values are not preserved. This causes any users with POP clients (MS Outlook, etc) to re-download all messages in the inbox as duplicates into their mailbox.

How can I preserve the UIDs?

I have tried dovecot dsync, but the documentation is poor, and I keep getting error messages that don't make sense.

NOTE: We cannot require users to switch from POP to IMAP, or change any settings in their client software. The migration must be transparent to users.

Thank you in advance for any advice!

Ryan Griggs
  • 885
  • 2
  • 12
  • 27

1 Answers1

1

OK I figured out how to do this. Here's the answer, for anyone who may find it helpful. This is a high-level overview. Let me know if you have questions about the details of any particular step.

  1. Create the user's mailbox on the new server. Set the password to a known value. Ensure the username matches the old username.

  2. Copy the user's current hashed password on the old server. Save this string somewhere.

  3. Set the user's password to a known value on the old server.

  4. Force dovecot to build the appropriate mailbox files on the new server: telnet to the new server, port 110 and log in with the username and password, then issue a UIDL command. This should list zero messages.

  5. Telnet to port 110 on the OLD server, log in as the user, and dump the list of UIDs using the UIDL command. Save this list to a file in the Maildir of the user's account on the NEW server. Call it uidlist.old

  6. Make a note of the UIDVALIDITY value on the old server, which, for Dovecot 0.9x will be the first part of the UID when issuing the UIDL command: Example line output from UIDL command:

    1 1234567890.12345 1 = message number 1234567890 = UIDVALIDITY 12345 = Message UID

  7. Modify new server's dovecot configuration to use the same UID format as the old server. Edit the /etc/dovecot/conf.d/20-pop3.conf file and set the pop3_uidl_format to the desired format (For dovecot 0.9x it should be %v.%u)

  8. Stop Dovecot on the new server.

  9. Modify the user's dovecot-uidlist file, changing the Vxxxxxx value to the old account's UIDVALIDITY value. This file can be found in the user's home directory/Maildir

  10. Delete any dovecot.index* files in the user's Maildir.

  11. Restart Dovecot on the new server.

  12. Import the user's emails using imapsync. Include the --useuid option.

  13. Once imported, again telnet into the new server, port 110 and log in as the user. Issue the UIDL command. This forces Dovecot to rebuild the list of messages.

  14. Stop dovecot again on the new server.

  15. Verify that the dovecot-uidlist file on the new server is in the following format: 3 Vx Ny Gz 1 Px.y :z ...

The first line contains the Vx (uidvalidity), Ny (next message UID) and Gz (Global identifier) Subsequent lines contain individual messages. 1 is the message number Px is the uidvalidity value and y is the message ID. z is the message filename.

If the file is NOT in this format, again telnet to the new server port 110 and validate as the user, then again issue the LIST and UIDL commands. This should force Dovecot to rewrite the file in this format.

Ensure the count of messages matches the count of messages from the old account. Sometimes there may be one extra duplicated message. It's probably at the top, and if so, you should see that the filenames are identical. Just delete this line from the dovecot-uidlist file if it's a duplicate.

  1. Use the following PHP script (requires PHP) to replace the Message ID of each message in the dovecot-uidlist file with the correct IDs from uidlist.old. Save the script as uidimport.php in the user's Maildir directory.

    http://pastebin.com/x2vvVD9w (tried to post here but it got messed up)

  2. Make sure there are now blank lines or spurious values in either uidlist.old or uidlist.new. BE SURE there is no trailing blank line at the bottom!

  3. Run uidimport as follows:

    php uidimport.php uidlist.old dovecot-uidlist > dovecot-uidlist.new

This creates a dovecot-uidlist.new file with the correct UID placed in each line.

  1. Stop Dovecot again and replace the existing dovecot-uidlist file with the .new version created above.

  2. Delete any .index* and .log files.

  3. Edit the new dovecot-uidlist file and ensure the Nxxxx value on line 1 is set to the NEXT number after the UID of the LAST message in the list. This number will be used for the next message that arrives.

  4. Restart Dovecot and telnet to the new server port 110. Authenticate as the user then execute the UIDL command and save the output to uidlist.new

  5. Do a diff of uidlist.old and uidlist.new. If NO differences are found, you have successfully duplicated the user's mail while preserving the UID of each message! Congrats!

Following these steps will prevent Outlook and other POP messaging applications from re-downloading the user's existing messages. I have successfully performed these steps on accounts with up to about 7000 messages in the inbox with success!

Feel free to contact me if you have any questions.

Ryan Griggs
  • 885
  • 2
  • 12
  • 27