5

I'm having the following arrangement. One server compiling emails in php and another server which run postfix (thus I have network latencies).

Each email is about 300Kb and so as to send an email (using Zend email class) it takes about 0.2s . However, if I compress the email text content, the size becomes 30Kb and the latency about 0.06 . The problem is that postfix doesn't support text compression, thus I'm thinking of developing a milter which will decompress emails from the queue.

Can you think of any drawback of this solution or can you suggest any alternative ?

edit: The company I work for sends > 100 000 emails to registered users daily and I'm trying to reduce sending time to minimum.

ptigas
  • 53
  • 4

2 Answers2

2

My preference would be to compress emails BEFORE they reach postfix with PHP:Zlib for example. After all an SMTP server handles email/SMTP traffic not compression/decompression.

ank
  • 700
  • 5
  • 13
  • the problem is that I want to avoid the network overhead. And it seems that compression/decompression time is less than sending 10 times bigger emails to postfix. (Also, check my edit.) – ptigas Apr 26 '12 at 14:37
  • also, people are using milters for spam filtering so it's pretty common to add additional functionality to SMTP servers, no ? – ptigas Apr 26 '12 at 14:44
  • 1
    @ptdigas 1) It is my opinion that the network is seldom the problem in mail delivery. Of much more concern is the mail server getting bogged down by the actual delivery mechanics: dead/busy/misconfigured servers, graylisting, wrong/obsolete users or with overquotas etc etc leading to overcrowded queues on busy machines. Which leads to overcrowded I/O on disks/filesystems. Depending on your setup such an SMTP server may also be taxed with virus/spam scanning and what have you. – ank Apr 27 '12 at 07:31
  • 1
    2) Milters for SPAM filtering is nice but serious/busy machines use external programs to do that even different dedicated hardware, software providing much more flexibility and capability. – ank Apr 27 '12 at 07:36
2

Prove that it is the network that is your bottleneck.

300KB messages over gigabit ethernet have a theoretical transfer time of about 2.5ms each; this indicates that you could send hundreds of them per second between the source and postfix.

However, postfix also has to store these messages in an on-disk queue, which is far slower - storing 100*300KB messages on disk (~30MB) would probably take up to a second or more, several times slower than the network transit time.

Gather accurate data first, then complain that something is a bottleneck.

adaptr
  • 16,479
  • 21
  • 33
  • OK! you are right. iperf reports that the bandwidth is about 700Mbits/sec. Given the disk limitation, the solution with compression wouldn't solve this problem? Milter dequeues emails from the queue, thus it will fetch the compressed email and not the original 10-times bigger. – ptigas Apr 26 '12 at 15:14
  • No, milters are applied before email enters the queue. – adaptr Apr 26 '12 at 15:18
  • We haven't even touched upon your backend **delivery** of these messages: unless your postfix server also has a gigabit *Internet* connection, the inflow speed is largely irrelevant. – adaptr Apr 26 '12 at 15:22