I'd like to have the ability to log and record various resources: CPU, memory, disk IOPS, etc of a machine so I can test its capabilities.
The purpose of this software is to test my own applications and memory usage of them.
I mainly use Ubuntu, but I also admin servers with Centos4/5 as well.
I am also interested to know what large providers use for the job.
If you need to monitor machines that area attached to a network, take a look at this post. It's very, very good.
But, if you intend to monitor a standalone machine (as you said), you could use SAR. It's a tool to collect and report system activity. This post and this post can help you to understand the big picture. This post is also very good.
Most monitoring systems are designed for exactly that - measuring the behaviour of a live system over a long period of time. There are lots of tools available for doing this - Bob's answer here gives links to some of the stuff available.
However there are a couple of problems with this.
First these are not really setup for testing purposes - you're going to have to think about how you reconcile these metrics with your testing.
Secondly, while the standard system metrics can give a good pointer to how performance is constrained - they don't actually measure transactional performance. The important thing is to deliver the data the user requests promptly - not to keep the CPU usage below some arbitrary value - so you need to think about how you measure actual end-to-end performance.
The other thing to bear in mind, is that application tuning is a very empirical art - you're going to have to colelct and visualize a lot of data - most of the monitoring tools aren't really very good for this.
