0

Basically I have a file system that is accessible both by NFS and ssh. I need a copy of this on my local. But I want to pull only those files that I need (basically cache them on demand). Since I'll be using this as a read-only fs the cache is equivalent to a permanent copy (even the remote won't change). So, is there an application that does this out of the box or should I be writing my own FUSE implementation for this? The problem is the FUSE implementation would be for Mac (although I have a Linux container running which can do this too.)

Samarth S
  • 1

2 Answers2

1

Mount the NFS share.

If desired, read only. Either server side for the export, or locally with the rdonly mount option.

Only transferring files on demand is inherent for NFS, you have to open a file to get its contents. Caching, both client and server side, keeps subsequent accesses fast. If for some reason you wish to pre-warm cache, just read the files.

If not NFS, syncing the entire volume to local would be simple. Remote is not changing. But not meet the requirement of only have a local copy on demand.

NFS has the advantage of already being implemented. As file system for macOS or Linux, and as a file share in your environment already. See how it performs before trying the (in my opinion, premature) optimization of writing your own caching file system.

John Mahowald
  • 30,009
  • 1
  • 17
  • 32
  • But I wat it to be populated on demand too. It should very similar to a cache system. Except that the cache is big enough to store the full copy and there will be no need for a cache update ever. – Samarth S May 18 '20 at 11:43
  • NFS has in memory caching, and is already implemented. See edit. – John Mahowald May 18 '20 at 21:23
0

If the number of relevant files is not too big then you could configure inotify for those files. After a file has been written (and the file descriptor has been closed) you can start rsync for that file.

Hauke Laging
  • 5,157
  • 2
  • 23
  • 40
  • they are of the order 67k – Samarth S May 07 '20 at 21:09
  • @SamarthS That may be too much. But you can reduce the necessary resources by watching only the directories: `inotifywait -m -r root_watch_dir/` That output can be piped to a program that checks whether the respective file is on your list and calls `rsync` in that case. – Hauke Laging May 07 '20 at 22:16