Two recommendations:
iotop is a handy program that shows you real-time I/O usage in a format like the top
command. This should help you identify the script/binary thats doing all the I/O. I think this may already be installed on newer kernels, but yum may provide it on RHEL if you don't see it.
lsof
is also helpful with I/O troubleshooting; it lists any open files.
UPDATE
If there is "constant" disk activity, then iotop
should tell you which process(es) are the culprits. Regarding I/O transaction history: I don't believe IO history is logged anywhere in most Linux distros, but you can add a cron job.
As root, open up crontab with crontab -e
and add these two lines:
* * * * * /bin/date >> /var/log/iotop; /usr/local/bin/iotop -bot --iter=3 -q >> /var/log/iotop
* * * * * /bin/date >> /var/log/lsof; /usr/bin/lsof -b -w >> /var/log/lsof
That dumps iotop
and lsof
output to a log every minute. Add a logrotate script for each to keep the files from taking up all your space, e.g.:
someuser@myhost:~> cat /etc/logrotate.d/lsof
/var/log/lsof {
rotate 3
weekly
compress
missingok
notifempty
}
I think lsof is what I want, I'll try it to see if it shows me the open files. Is there anyway to get a tail of io operations including deletes? – JTew – 2012-05-28T20:43:20.167
Looks like you figured it out, but I added some info the my answer if you find it helpful. – Banjer – 2012-05-29T12:41:43.187