- Initial installation of operating system( installing centos 5 or ubuntu server manually 8 times seems like a tedious task that just begs for automation)
Setup PXE(Preboot Execution Environment) to use kickstart files that automate the process of installing the OS
- Centralized Configuration Management and Software Updates for Host and possibly Guest(virtualized) servers
Puppet is great tool for managing all of the config files across the cluster and pushing them out to them and then using a common scm tool like git, mercurial or svn to track changes made to files.
- Hardware, Services and Network Monitoring
Nagios and/or Ganglia are well suited tools for monitoring your hardware and to some extent determining incident causes.
It should be noted that learning to use these tools and most importantly getting all these elements configured and in place take some time but once they are setup it makes scaling out much easier and cuts down on response times for certain issues (like reinstalls for example or applying sweeping changes like security updates and software installs). Most importantly though with tools like puppet it makes it much easier to keep all of your machines identical in function.
