4

We use capistrano for website deployments and our Apache document root is a symlink to a particular code release. The deployment procedure switches the symlink from the old release to the new release as the final step of the deployment.

We are migrating our webservers from real servers running RHEL 5.6 to Amazon EC2 virtual machines running Ubuntu 11.10 and the new servers are suffering from a problem where Apache doesn't immediately notice the change to it's document root when the symlink is switched. It can take a second or so (and I think I've even seen it take a couple of minutes). It's kind of like Apache has cached the physical path of the symlink for some time.

Does anyone know some Apache settings I could look at to get it to "scan" for changes to it's served files quicker.

Thoughts:

  • I read that the disks on virtual machines are much slower (since they are network attached storage). Perhaps the filesystem cache somehow works differently too? If so, is there anything that can be done?
  • The website runs PHP code. Perhaps there is some PHP config differences between RHEL and Ubuntu? I checked realpath_cache_ttl but both servers have it commented out:

e.g.

; Duration of time, in seconds for which to cache realpath information for a given
; file or directory. For systems with rarely changing files, consider increasing this
; value.
; http://www.php.net/manual/en/ini.core.php#ini.realpath-cache-ttl
;realpath_cache_ttl = 120
  • We do use the APC opcode cache but don't think it's the issue due to experimentation. The PHP code is in different file paths for each deployment and we ensure stat=1.
  • Here is a similar question that is very interesting: 294107 - but doesn't provide an answer for me.
  • One solution would be to reload Apache everytime we modify the document root symlink. I'll do this if we can't find another solution.
Community
  • 1
Tom
  • 4,157
  • 11
  • 41
  • 52
  • Are you using a particular framework or is it from scratch developed app ? – m0ntassar Mar 30 '12 at 17:39
  • A bit of both. Some pages use the Yii framework. The problem occurs on both though. – Tom Mar 31 '12 at 18:24
  • Is the symlink a local file as far as the kernel is concerned? In other words, is it mounted under /, or is it mounted over some other protocol? I ask because I have seen NFS cache do the same thing. – chrishiestand Apr 01 '12 at 23:40
  • From `df -h`, it appears to be mounted under `/` (but it's a EC2 VM so not exactly sure what it's REALLY doing under the hood). We're not using NFS. – Tom Apr 04 '12 at 14:15
  • Have you made any progress on this? I am also using Apache, Capistrano, PHP and Yii and occasionally see this, and this is on a dedicated server. I have an restart task that clears the APC opcode and user cache on deploy, but maybe 1 out of 10 deploys, the site will go on for minutes showing the old version, as if Apache is still pointing to the old location and has not realized that the symlink is changed. For this reason, I also have a deploy:restart_apache task that the devs use to kick over Apache. I am definitely going to play with sync and a graceful restart. – SteveK Mar 13 '13 at 01:10
  • @SteveK - we do `apache2ctl graceful` as part of every employment to ensure the new document root is picked up (and any new Apache config takes effect). It doesn't cause downtime. – Tom Mar 14 '13 at 11:07

1 Answers1

2

If it is an EBS-backed image, note that Amazon uses some pretty aggressive caching behind-the-scenes (i.e. your settings mean nothing) to get EBS performance vaguely in the ballpark for common read tasks. Writes are slower to be committed to disk.

I have enjoyed some success with <command to modify disk> && sync. So it's possible Amazon is trapping the sync system call and forcing EBS to sync as well. Or maybe it's the natural behavior of sync speeding up the process (i.e. pushing in-memory changes to the filesystem immediately also kicks EBS). Unfortunately Amazon is not particularly forthcoming about their EBS tech.

If the problem truly lies with Apache, you don't specify if you are doing anything to try to alert Apache to pay attention... a "apachectl -k graceful" will tell each worker thread to finish what it is doing and exit, then launch a new thread to replace it, in theory one that does not have the content cached. (I doubt the destination of the symlink is cached by Apache, it might be cached in the file descriptor table for the process; sync should solve that.)

Mike
  • 311
  • 1
  • 2
  • Thanks, we are now using `apache2ctl graceful` after each deployment and didn't come across problems. Out of interest, what does the `-k` do? (I couldn't see it in the man page). I'm also thankful for the advice about `&& sync`. – Tom Apr 08 '12 at 20:42
  • apache2ctl can operate directly with single word commands (as you are using it), but also can operate in pass-through mode where you can use any option valid for the httpd binary (called apache2 in Ubuntu). The `-k ` simply invokes `httpd -k `. The two are functionally equivalent; I trust passthrough mode a bit more, but can't justify why that is. :) – Mike Apr 10 '12 at 17:23