Caching a file system over the internet

Question

I store all my files on my private server (DSL line, /28 network but still slow, especially the up-stream which is 512kBit). This works well for my smaller files, all my emails, Zarafa etc.

But for larger files (especially Fotos) it is too slow. Furthermore, I want to share the fotos with family. In the LAN this is already done with Samba, each user has a different user (LDAP).

For speeding up internet access, I have a vServer with fast connection but not so much space (10GB).

It would be great if this could be used to speed up the access from internet to the files.

One plan would be to access the files over Samba/NFS over an OpenVPN tunnel, add some cache and then use ownCloud or WebDAV on the vServer.

The caching system would need to support:

1.) Store all accessed files in the cache (10GB) and take it from the cache if they exist. A small meta request could/should be done first to check if the file changed on the master (just modification date)

2.) Files written should be stored into the cache (which is fast) and sequentially updated on the master. When a file is opened for writing, the file might be locked on the master to avoid conflicts

3.) The cache should be always full (10GB) and survive reboots/disconnects. When new space is required a method like LRU or NRU should be used to remove old entries from cache.

I can hardly imagine that there is no solution for that - but I was not able to find one. Any ideas how to solve this?

Answer 1

Try cachefilesd and fs-cache. I have no experience with them but their description match your requirements.

Answer 2

A cloud storage system such as DropBox, Google Drive, SugarSync, etc., could meet most of your requirements depending on the number and sizes of files. The tricky item is how to deal with locking/conflicts.

Answer 3

Well, there always remains the fact that you have to get it from your pc to that virtual server, and that's always going to be slow.

Want to really speed it up, upgrade your internet access to a faster upload, or make sure the files arrive at your vServer without passing through your own computer.

This could be accomplished if you use things like VNC, Teamviewer, logmein, etc to access and work on your server and do all the work there. But you'll likely to run out of diskpace quickly, because you don't have all the tools available you have on your computer, so you can't quickly edit a photo, so you'd need to install the photo editor there too, and that takes space.