Identifying files that changed in the last hour

Question

Right now I use python to figure out files that have been modified in the past hour. This is really slow on my network (~50000 files - and checking each one's timestamp). I have a custom script to which I pass this list - it does some transfers/logging in/etc on a remote server.

I want to speed up the file list generation time (it takes ~15-20mins to just figure out the file list). Any suggestions?

One thing that might be helpful is that the network has a netapp filer. That filer creates those .snapshot dirs. Can I somehow hook into the filer (through the API?) and figure out a list that changed recently?

If you're not familiar with the NetApp API, suggestions on how to generate a file list (fast!) with unix commands would be great! (BTW, this is a network filesystem - so there will be multiple machines making changes).

Answer 1

Just use find:

find /path/to/dir/ -mmin -61

Relevant options:

   -mmin n
          File's data was last modified n minutes ago.
   Numeric arguments can be specified as

   +n     for greater than n,

   -n     for less than n,

   n      for exactly n.

So, -mmin -61 means "find files that have been modified less than 61 minutes ago", in other words, those that have been modified in the last hour.

You might want to use these options as well, they will speed up the search but I don't know if they are appropriate, that will depend on your setup:

• -maxdepth 1 : Don't descend into subdirectories.
• -type f : Look only for regular files, no directories etc.

I just ran this command on my laptop (i7,2.6GHz) in a directory containing 78353 randomly generated files (meaning the modification dates are also random). It took less than one second to return a list of 51 files modified in the last hour.