User data is stored on two fairly large (>1 PB) OpenStack Swift storage clusters. Let them be Cluster A and Cluster B.
In addition, there are several PoPs that need to interact with that data. Servers in these PoPs are effectively diskless, meaning no user data is stored on them or ever downloaded to them. PoPs can be grouped into general world regions (e.g. North America, South Africa, Central Europe et al.).
Some PoPs are quite a long distance away from Swift endpoints of any of the clusters, introducing an undesirable latency. In order to somewhat mitigate this, I want to setup a caching gateway server in each of the regions, which will cache r/w requests to the nearest cluster.
Currently, clients in any of the PoPs access user data by a permanently mounted swift virtual file system, which is a FUSE module that mounts Swift Object Storage as a block device (more or less). However, svfs isn't all that stable in the first place and in the future, clients should access the cache servers via NFS.
This is a diagram of one branch of the desired architecture:
+------------------+ +------------------+ NFS +------------------+
| Cluster A | SVFS | Region 1 Cache +----------> R1 PoP a Client |
| +----------------> | | |
|Persistent Storage| |Ephemeral Storage+----+ |Generates R/W Load|
+-----------------++ +------------------+ | +------------------+
| |
| +------------------+ | NFS +------------------+
| SVFS | Region 2 Cache | +-----> R1 PoP b Client |
+-----------------> | | |
|Ephemeral Storage| |Generates R/W Load|
+------------------+ +------------------+
I am familiar with the basics of setting up NFS and svfs.
The question is: How can I setup the caching server to use all available resources (a designated cache partition, RAM) to cache as aggressively and as much data as possible before writing to the svfs mount point? Basically it comes down to: How can I cache a directory in linux?
If possible, reads and writes should be consolidated and block sizes in FUSE requests should be at least 128k if possible to maximize throughput and minimize latency if the cache needs to write to the cluster.
Addendum 1: I've switched the cluster mount module from svfs to S3QL on a few of the servers. S3QL's caching has improved performance a bit. I will try to get some performance data for completeness.
