I have a client with 150 Linux servers spread about various cloud services and physical data-centres. Much of this infrastructure is acquired projects/teams and pre-existing servers/installs.
The client is largely about image processing, and many of the servers have large SAN or local disk arrays with millions of jpeg/png files.
There is a configuration management agent on each box, I can see that many disks are 100%, some are pretty empty, and there is a lot of duplicated data.
The client now has access to a CDN. But at the moment just enumerating what is possible is a daunting task.
Are there any tools to create useful indexes of all this data?
I see tools like GlusterFS for managing these distributed filesystems, and Hadoop HDFS
I am wondering whether I can use the indexing tools of these systems without actually implementing the underlying volume management tools.
What should the starting point for generating an index of potential de-duplication candidates?