How can I use one lvmcache "cache pool LV" for multiple origin LVs?

Question

I have a shiny new computer with a not-that-large SSD (120GB) and two gigantic spinning-rust HDs (3TB each). I want to partition it as follows:

• swap (~128GB), operating system (~128GB), and data (all the rest) partitions striped across the HDs
• tiny boot partition on the SSD
• all the rest of the SSD's space is used for a cache pool
• the cache pool should be shared among all of the HD partitions, except maybe not the swap

I tried to do this with lvmcache and ran into the following problems:

• If you just create a cache pool in the obvious way (as described e.g. here) you can only assign that to cache a single origin LV. So I could have it for the OS or for the data, but not both.

• I tried to work around that by creating a "thin pool" spanning all of the HDs, thinking that I could cache the entire thin pool, then assign OS and data LVs out of that, but got told that it is not possible to cache a thin pool. (The lvmthin manpage doesn't make a hell of a lot of sense, it is possible that I did that part wrong.)

I am out of ideas. Can anyone suggest how to make this work? Note that I am not married to lvm, if bcache+plain MD (for the striping) can do this, or some other tool I don't know about, then that would be Just Fine Too.

Answer 1

You can create cached volume and convert it to thin-pool.

cache pool (cache data + cache metadata) -> cached volume (cache pool + original big slow volume) -> cached thin-pool

In this example vg0 is your volume group with physical volumes /dev/small-fast-disk and /dev/big-slow-disk. All logical volumes must be in the same volume group.

create cachepool volume (cache data and cache metadata in one command)

lvcreate --type cache-pool --name cachepool --size 128G vg0 /dev/smal-fast-disk

create cache volume

lvcreate --type cache --cachepool vg0/cachepool --size 3T --name thincachevol vg0 /dev/big-slow-disk

convert cached volume to thin volume

lvconvert --thinpool vg0/thincachevol

Answer 2

On a mainline kernel using the same cache for multiple origin devices is NOT supported by dm-cache (which is what lvmcache uses underneath) due to races accessing metadata. See Re: [dm-devel] dm-cache: can the same cache be used with multiple origin devices? for a 2013 comment from a developer.

However, for completeness the original dm-cache developer has an abandoned testing repo supports "multiple source devices on the same cache device" (see https://groups.google.com/forum/#!topic/dm-cache/q-lM1t438PU for details) but this code is not in mainline kernels.