0

Is there any tool with which I would be able to track any changes happening recursively inside a particular folder?

I have used inotifytools and fswatch but both are hitting the server performance because the folder I am watching is having a size of 1.5TB + . It is an EFS attached to an EC2.

Can someone suggest how can I avail one folder watcher without affecting the performance? My initial thought was to capture the data on a daily basis in files and push the file to S3 bucket. But it is hitting the performance badly.

Ranadip Dutta
  • 123
  • 8
  • What kind of changes do you wish to track? What kind of files are you storing? How many files? Is there a deep subdirectory structure? What kind of information on the changes do you need the tool to report? – Leo Jul 05 '18 at 07:00
  • @Leo: All are flat files. Majorly zip,txt,tar.gz,csv and so on. Yes, there are 100 + sub directories. I am trying to get only the creation, deletion, modification,userID(who did the particular change), and timestamp. – Ranadip Dutta Jul 05 '18 at 13:34
  • [Auditing](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security_guide/chap-system_auditing). What do people always try to reinvent the wheel? – Andrew Henle Jul 05 '18 at 13:54
  • This may not be an ubuntu problem, more likely to be an EFS problem. inotify is pretty efficient, and isn't impacted by the amount of data in a directory, only by the number of entities whose metadata is being tracked. So there may be some metadata inefficiency in the use of EFS, and I would guess performance impact using a local file system to be significantly different. – Jonah Benton Jul 05 '18 at 14:04
  • @JonahBenton: May be you are right. But how can I achieve it now. Do you know any paid tool or any product which can serve me or can help me out on this? – Ranadip Dutta Jul 05 '18 at 14:33
  • @ranadip in the context of the current architecture, I would look for EFS throughput/latency tunables. I assume they are there, but haven't used EFS for a long time. Can also look at file system metadata cache tuning. This solution- EC2 instance with an EFS volume, with different user accounts making changes in a single file system tree- sounds hard to manage, but the actual problem isn't described in the question. So there may be other approaches. – Jonah Benton Jul 05 '18 at 14:49
  • @AndrewHenle: I will check the auditing – Ranadip Dutta Jul 06 '18 at 10:15

1 Answers1

2

I believe the audit daemon is what you are looking for:

https://linux.die.net/man/8/auditd

Once you get your rules setup aureport -m gives you reporting on modifications to whatever file or directories you configure.

You'll also want to be mindful of disk space because audit logs can build up pretty quickly. You'll want to either run jobs to ship those out or regularly compress and rotate them.

frontsidebus
  • 496
  • 2
  • 7
  • I will check this. I am sceptic about the RAM utilization and not storage. – Ranadip Dutta Jul 06 '18 at 10:14
  • No Doesn't suit the performance..:( It is taking the almost 30GB - 50GB RAM depending upon the day and the operation. I mounted S3 bucket separately for logs. That removed the storage constraint; but server performance is badly impacted. – Ranadip Dutta Jul 10 '18 at 05:17
  • I am accepting this because I am not getting much option for this but this can help me to some extent. – Ranadip Dutta Jul 11 '18 at 13:38