How to automate website deployments to production including minifying CSS/JS?

Question

Currently I'm deploying my PHP website from staging to production using the following manual process:

• Minify CSS and JavaScript files using online YUI compressor tool.
• Move the original CSS and JavaScript files into another directory (to back them up) then replace them with the minified CSS and JavaScript CSS files.
• Start copying all the files from my staging server to my Windows PC.
• Delete staging .htaccess and index files so they don't get copied to the production server (these files have specific stuff for the staging environment).
• Open WinSCP and go to the web root on the production server then switch out the main index file for an 'updates in progress' one so if any visitors come to the site then they get a message saying it's down temporarily.
• Then copy from my PC to the production web server using WinSCP (this overwrites all existing files on the server) and takes about 4 minutes seeing there's lots of files.
• Log into the production server with SSH and run about 5 commands to set the proper permissions for various writable directories and make sure everything is still under www-root group. I think the upload process with SSH changes all the existing permissions.
• Switch out the 'updates in progress' index file back to normal index.

All up this process is pretty convoluted and takes about 5-10mins and I have to do it each time I make changes to the website so it has become a chore.

Now is there any way to automate this process using some scripting tool? Or is there a nice deployment tool that people use that can do all this? My staging and production machines are Ubuntu 12.04 server based so potentially I could use bash scripting to do some of the work.

Some improvements I can see that could be made so far would be:

• Add some code to my website so it detects if it's in Staging or Production then uses the original or minified CSS/JavaScript files depending on the environment. This will save me swapping the files around manually.
• Use the commandline YUI compressor and run that from a script file when needed.
• Use something like rsync so it only copies the changed files from my staging server to the production server not overwriting every file

One company I worked for used the Fabric command line tool for automating the deployments and doing this kind of thing. However is this a good option or is there a better tool out there?

Answer 1

Fabric would be ideal for this kind of task. If you're familiar with python you have a lot of flexibility with what you can do.

Capistrano is another option that might be useful, I have used it for Rails deployments and had no issues there. Allows you to easily run shell commands on remote hosts.

Another option might be to use Ant (python way with fabric is much nicer imo).

Are you using source control like git? You could add some of these tasks as post-commit hooks when you push it to your "production" branch.

Rsync would be the quickest and safest option. Make sure you exclude any files which should not be in production (backup files, vim .swp files, etc).

Answer 2

Well I ended up using Fabric and came up with a script that does exactly what I want. In 48 seconds everything is copied to a temporary directory, all files minified then the site shows a 'upload in progress page', everything is rsync'd to the server using SSH then it's put live again.

from fabric.api import *
import datetime

# Format the current date as string 20120908-221521 (8 Sep 2012, 22:15:21)
# This will be used to append to the folder of the deployment directory so there's a backup of what was deployed
currentDate = datetime.datetime.now().strftime("%Y%m%d-%H%M%S")

# Where the files for this deployment are stored
deploymentDir = "/var/www/deployments/mysite-deployment-" + currentDate

# SSH config for production server
env.use_ssh_config = True
env.hosts = ["mysite.net"]
env.user = "root"
env.port = 44

# Test connection to live server
def test():
  run("ls -al")

#--------------------------------------------------------------------------------------
# Main functions - to run use "fab live" from command line to update production server
#--------------------------------------------------------------------------------------
def live():
  backupToDeploymentDir()
  switchToProductionFiles()
  minifyCssAndJavaScript()
  switchToUpdatesInProgress()
  rsyncToProduction()
  switchToLive()  
  print("Deployment " + currentDate + " complete")

# Alias for the clearCachedFiles method - to run use "fab cc" or "fab live cc"
def cc():
  clearCachedFiles()

#-------------------
# Utility functions
#-------------------

# Backup to deployment directory
def backupToDeploymentDir():
  local("cp -a /var/www/staging/ %s" % deploymentDir)

# Remove development cache and log files, switch into production mode for htaccess and front controller
def switchToProductionFiles():
  local("rm -rf " + deploymentDir + "/temp/cache/*")
  local("rm -rf " + deploymentDir + "/temp/logs/*")
  local("rm -rf " + deploymentDir + "/fabfile.pyc")
  local("rm -rf " + deploymentDir + "/nbproject")
  local("rm " + deploymentDir + "/.htaccess")
  local("rm " + deploymentDir + "/index.php")
  local("mv " + deploymentDir + "/live-.htaccess " + deploymentDir + "/" + ".htaccess")

# Minify CSS and JavaScript files
def minifyCssAndJavaScript():  
  # Compress CSS
  local("sudo yui-compressor --line-break 1 -o " + deploymentDir + "/public/css/main.css " + deploymentDir + "/public/css/main.css --type css")
  local("sudo yui-compressor --line-break 1 -o " + deploymentDir + "/public/css/pdf.css " + deploymentDir + "/public/css/pdf.css --type css")
  local("sudo yui-compressor --line-break 1 -o " + deploymentDir + "/public/css/print.css " + deploymentDir + "/public/css/print.css --type css")
  local("sudo yui-compressor --line-break 1 -o " + deploymentDir + "/public/css/reset.css " + deploymentDir + "/public/css/reset.css --type css")

  # Compress CSS external library files
  local("sudo yui-compressor --line-break 1 -o " + deploymentDir + "/public/css/lib/fullcalendar.css " + deploymentDir + "/public/css/lib/fullcalendar.css --type css")
  local("sudo yui-compressor --line-break 1 -o " + deploymentDir + "/public/css/lib/fullcalendar.print.css " + deploymentDir + "/public/css/lib/fullcalendar.print.css --type css")

  # Compress JavaScript
  local("sudo yui-compressor -o " + deploymentDir + "/public/js/ajax.js " + deploymentDir + "/public/js/ajax.js")
  local("sudo yui-compressor -o " + deploymentDir + "/public/js/common.js " + deploymentDir + "/public/js/common.js")
  local("sudo yui-compressor -o " + deploymentDir + "/public/js/default.js " + deploymentDir + "/public/js/default.js")
  local("sudo yui-compressor -o " + deploymentDir + "/public/js/global.js " + deploymentDir + "/public/js/global.js")
  local("sudo yui-compressor -o " + deploymentDir + "/public/js/organisation.js " + deploymentDir + "/public/js/organisation.js")

  # Compress JavaScript external library files
  local("sudo yui-compressor -o " + deploymentDir + "/public/js/lib/fullcalendar.js " + deploymentDir + "/public/js/lib/fullcalendar.js")
  local("sudo yui-compressor -o " + deploymentDir + "/public/js/lib/head.load.js " + deploymentDir + "/public/js/lib/head.load.js")
  local("sudo yui-compressor -o " + deploymentDir + "/public/js/lib/jquery.js " + deploymentDir + "/public/js/lib/jquery.js")
  local("sudo yui-compressor -o " + deploymentDir + "/public/js/lib/jquery-ui.js " + deploymentDir + "/public/js/lib/jquery-ui.js")
  local("sudo yui-compressor -o " + deploymentDir + "/public/js/lib/sha512.js " + deploymentDir + "/public/js/lib/sha512.js")

# Switch out the main index.php file for updates in progress one, before we rsync
def switchToUpdatesInProgress():
  run("mv /var/www/updates-index.php /var/www/index.php")

# Sync all files from deployment directory to live site then run permissions
def rsyncToProduction():
  local("rsync -azvv -e \"ssh -p 44\" " + deploymentDir + "/ root@mysite.net:/var/www")
  run("chown -R root:www-data /var/www")
  run("chmod -R 750 /var/www")
  run("chmod -R 770 /var/www/temp/cache")
  run("chmod -R 770 /var/www/temp/logs")
  run("chmod -R 770 /var/www/temp/sessions")
  run("chmod -R 770 /var/www/library/mPDF/ttfontdata")
  run("chmod -R 770 /var/www/library/mPDF/tmp")
  run("chmod -R 770 /var/www/library/mPDF/graph_cache")

# Clear cached files from the production server, useful if the PDF or JSON cache output has changed
def clearCachedFiles():
  run("rm -rf /var/www/temp/cache/*")
  print("Cached files cleared")

# Switch out the updates index.php file for the live one which will put the website live
def switchToLive():
  run("rm /var/www/index.php")
  run("mv /var/www/live-index.php /var/www/index.php")