0

I have a client who send out a weekly newsletter to around 3,000 recipients. The newsletter is built using around 50-100 images depending on the template they use (which I recognize is horribly efficient, given the design of the newsletter).

However until recently, the send out caused their web server to become non-responsive since it was a single web server handling the requests for both static assets and PHP, causing the web server to run out of apache children and not be able to serve fast enough. Measuring the web server right after a send out we observed the following:

  • Sample time of about 15 minutes
  • Consistently above 300 requests per second (the vast majority being static assets, very few web site visitors)
  • Peaked around 800 requests per second
  • Approx. 100 recorded newsletter views
  • Approx 50 recorded newsletter clicks (the majority leading to the website itself)

We fixed the issue temporarily with a secondary web server dedicated to serving static assets, with a much leaner apache configuration. This resulted in a newsletter send out that the web site itself barely flinched at.

Our confusion revolves around how such a massive surge of requests for static assets does not match the number of newsletter views and clicks, is this something anybody else has seen, and are there any other ways to deal with such traffic? I imagine reducing the number of static assets and a separate static assets web server should get us most of the way, however I'd really like to be able to explain this phenomenon.

Thanks for any advice!

DanH
  • 817
  • 2
  • 8
  • 23
  • This all sounds very cache'able or CDN'able - is that right? – Chopper3 Jul 09 '12 at 10:07
  • 2
    What phenomenon? People open your spam (triggering asset loads), decide they don't want it, and don't visit your website (or buy from you ever again). Seems pretty straightforward to me. – womble Jul 09 '12 at 10:21
  • @Chopper3 it's very CDNable, yes, but I doubt the client is willing to pay for the service. I suspect we'll end up running 2 webservers, 1 to catch static asset requests, and then pass on page requests to the 2nd web server. – DanH Jul 11 '12 at 03:12
  • @womble Mailchimp is only recording 100 newsletter views, which would surely be triggering if people were viewing the newsletters, whether in spam or not. Unrelated: The site sells no product, newsletter signup is only done directly via the subscribe form, there's no tricking people to subscribe here. – DanH Jul 11 '12 at 03:12

1 Answers1

1

Approx. 100 recorded newsletter views

And how is that recorded? I would count the views by embedding a non-cacheable image in the newsletter - but that's obviously not the case here. You certainly can't rely on SMTP DSNs nor javascript.

But I would still expect that a number of the 'viewers' will be malware scanners and compressing proxies (you should be able to determine from the source ip / user agent).

(BTW wouldn't it have been cheaper to stagger the email delivery or use a CDN rather than implement a new server?)

symcbean
  • 19,931
  • 1
  • 29
  • 49
  • The views and clicks are reported by Mailchimp, which I'm fairly sure inserts a non-cacheable image like you say. The reason another server was used is because it was handy to make use of in a short time frame. – DanH Jul 10 '12 at 04:35
  • 1
    I expect you'll find (with a little bit of digging) that you have prima-facia evidence that mailchimp are not getting an accurate count of email views. – symcbean Jul 10 '12 at 08:45