I have a longstanding situation with a client who has a tremendous amount of email. To make matters worse, the bulk of the message content and attachments are text files. So, we're talking about a lot of headers, small files and a whole bunch of content to index.
The platform the data is currently on is Hosted Exchange 2010. I chose that under the assumption that it was the most robust email solution out there. The hosted platform I am on has some really nice features, such as being able to grab a weekly PST snapshot of every mailbox for backup purposes, which is difficult feat with an IMAP provider.
The worst user is one whose inbox receives about 5000 messages per month and most of them cannot be removed because they contain important legal information about contracts and deals, so the integrity of the threads must be preserved.
To compound the issue further, there are 5 mailboxes that are all subscribed to with Apple Mail via Exchange protocol on 5 computers. They insist on using Apple mail, and on having all those mailboxes available to all the machines, which is legitimate as they need to be able to search correspondence related to deals and contracts across all the accounts at the company to make sure no exchanges get missed when responding to a client. The only viable alternative would be to suggest that they start a different database to hold those threads, which would be a lot of extra work for them and would probably require hiring extra workforce to process.
I have exhausted my knowledge of how to deal with this and the volume of email coming in continues to increase. The email incoming rate in the primary mailbox has doubled since last year.
What I have done to address the problem so far is:
- "chunked out" all server side mailbox folders to a size of less than 5000 headers
- moved non-primary accounts for each user into Outlook 2011 in headers only mode
- removed all non essential communications out of the mailbox (Facebook alerts, promotions, etc.)
- moved content before 2012 to a separate exchange mailbox for the primary user to bring the total mailbox size down to under 7GB for the primary and its archive, all other mailboxes are smaller
- had the hosted Exchange provider move all mailboxes to a new server-side database
- exported all local data to a Mail archive, deleted all Mail data and recreated all accounts
No matter what I do, I am having consistency issues with Apple Mail maintaining sync with the accounts reliably. Symptoms:
- server side folders randomly disappear or lose their content (all is well server side, Mail just fails to display a folder or its content)
- inbox stops updating, loses all its content or reverts to a date in the past
The users insist on continuing to use Apple Mail. Although I am fully aware that there are several issues with this application, that it does not properly implement IMAP or Exchange, has issues maintaining the integrity of its local caches and many other issues, I am still doing my best to try and allow the users to continue using it as they prefer to do. Part of the reason is that I am not certain Outlook 2011, or any other email client, can handle this gracefully. And if I were to insist that the only way this will work well is to switch email clients, I would have to be absolutely sure that the new solution would indeed maintain integrity.
Archiving content locally is not an option as that content would have to be propagated to at least 5 machines and kept in sync with every archive operation, which must happen frequently with the amount of content coming in.
If anyone knows of a sure fire solution for this, something that would resolve this situation once and for all, I am all ears.
I have tried to add the Exchange accounts as IMAP in Mail, which does yield better sync results but aggravates a different problem that is a deal breaker - with that many connections to the same mailbox (5 Mail + 2 iOS devices), the Exchange server runs into too many concurrent IMAP connections issue. This happens regardless of which client I use, I was able to replicate the problem using MS Outlook 2010 and 2007, Thunderbird on Win and OS X, Mail and Outlook 2011. So, at best I can maintain a single IMAP connection without risking connection rejections because of the IMAP concurrency limit. I have also verified that other hosted Exchange implementations suffer from the same limitation by testing against Office 365.
I could move the situation back to an IMAP only provider, but again, I am not sure which provider I would choose who could guarantee better results.
I welcome your words of wisdom.
