Why is kjournald extremely active on a mostly quiescent filesystem?

Question

Note: This is a "documenting my process" question that I'll be writing an answer to. Happily, serverfault already helped me solve this, because writing up my situation so others would be able to assess it caused me to realize what was happening.

1. I'm using the ext3 filesystem on a Debian server.
2. Despite very little activity happening on my main filesystem with respect to files, there's tremendous activity from kjournald (seen via iotop).
3. This activity comes in periodic bursts that raise my overall average writes to about 2 MB/sec (which is of great concern to me because I'd like to get some SSDs and that rate is actually enough to seriously threaten the very generous write endurance of current models).
4. I've already made the filesystem in question mount with noatime,nodiratime.
5. I've already increased the filesystem's journal commit interval from 5 sec to 300 sec.

What is going on? (Spoilers: it was a userspace thing. I'm writing this mainly to highlight the possibly-counterintuitive underlying issues.)

Answer 1

See, what happened is, the main application that runs on this server manages an extremely large, well-populated directory tree, and writes files in that tree with somewhat suboptimal ownership and permissions. Because it's fairly obnoxious to get that application to change that, and the files need their ownership and permissions fixed up reasonably quickly (some delay is fine, but not a lot), I set up a cron job to, every minute, throw a mass chown -R and chmod -R at the large, well-populated directory tree. Everything seemed to keep running fine while that was going on, so I said, enh, it's overkill but it works, I'll live with it.

However. As it turns out, when you do a chown or chmod, it registers journalable ext filesystem metadata regardless of whether any change took place. So nothing or next to nothing was actually changing on the filesystem, but immense amounts of metadata were generated, which then hammered the hell out of the disk when the journal committed. Oops.

So I changed the chown and chmod to find jobs that actually look for files that need to be changed before changing them, and average writes went from 2 MB/s to maybe 50 kB/s. Yay.