14

Related to this question about using an SSD for system and HDD for data, except I would like my system to do this automatically...

Is it possible to have several layers of storage and push items automatically between them, using preferably free, open-source software?

I know of hugely expensive enterprise-class solutions like the EMC SAN->EMC Centera automatic archiving, but I was wondering if this sort of staged storage is possible to do automatically.

It would be nice to be able to have several layers in this : Memory->SSD->HDD->slower HDD or tape or some other archive solution.

Are there any filesystems which can do this automatically? (ZFS, Btrfs, HAMMER?)

Any Unix-variants are fine, as I'm interested in how this works and whether it's likely to be portable to Linux or other flavours (BSD etc).

Community
  • 1
David Gardner
  • 1,499
  • 2
  • 13
  • 25

9 Answers9

14

Well, ZFS uses a storage layering called Hybrid Storage Pool (HSP):

  1. Layer: memory
  2. Layer: SSD-based read-caches (L2ARC) and write-caches (separate intent log aka slog).
  3. Layer: (cheap) harddisks

With HSP its easy to automatically benefit from the advantages of SSDs compared to a harddisk-only solution. A system using HSP can be both faster and cheaper than the latter. See this link for some nice examples and more details.

I think there are plans regarding hierarchical storage management (HSM) for ZFS (see for example the Automatic Data Migration (ADM) OpenSolaris project) but I don't know its current status.

user9517
  • 114,104
  • 20
  • 206
  • 289
knweiss
  • 3,955
  • 23
  • 20
  • This is just what I was thinking of, thanks! (Now to help on the Linux port... ;) – David Gardner Jul 23 '09 at 08:30
  • 1
    I would not call this tiering but caching. In the sense that caching means keeping a copy of frequently used blocks, tiering *moving* the data to slower/faster storage. I would not call the intent log in ZFS a real write cache if I understand it correctly. – mrossi May 20 '16 at 07:46
5

Check out the LVM based "lvmts" (LVM Tiered Storage) solution this guy is cooking up:

https://bbs.archlinux.org/viewtopic.php?pid=1140640#p1140640

Pretty cool.

Shane
  • 51
  • 1
  • 1
5

TIER seems to answer your needs It is a linux kernel module that can create a tiered storage. It seems to learn by itself the pattern and optimize the placement into the storage

http://www.lessfs.com/wordpress/?p=776

Dolanor
  • 173
  • 2
  • 8
  • Just saw the question is _old_ ... Sorry ^^' – Dolanor Aug 08 '12 at 15:33
  • 4
    Revitalising old questions is better than having then rotting with old and out-of-date information on them! – David Gardner Aug 09 '12 at 10:57
  • I'm worried that this guy wants to reduce redundancy for the sake of extra capacity (especially when SSDs are comparatively small and unreliable). – Tobu Nov 06 '12 at 10:35
  • Also worried that he suggests raid as the solution, when experimental software needs backups most. – Tobu Nov 06 '12 at 10:54
  • Here's a bit of more positive feedback though: http://permalink.gmane.org/gmane.comp.file-systems.ceph.devel/8316 – Tobu Nov 06 '12 at 10:55
1

Some relevant links. YMMV.

http://code.google.com/p/fscops/ -- "Online Hierarchical Storage Manager (OHSM) is the first attempt towards an enterprise level open source data storage manager which automatically moves data between high-cost and low-cost storage media.".

http://www.tack.ch/unix/dmapi/ -- XFS + DMAPI under Linux

http://jfs.sourceforge.net/ -- JFS + DMAPI under Linux

sendmoreinfo
  • 1,742
  • 12
  • 33
  • The OHSM project looks like an interesting one to follow. Only ext2 and the 2.6.30 kernel is supported for now, but I expect if it works out well then this will gain more filesystems. – David Gardner Jan 22 '10 at 11:41
  • That project looks abandoned, most recent activity was in 2009. – sendmoreinfo Sep 12 '11 at 22:05
1

RAID controllers offer some of these features.

"Leverages SSDs in front of HDD volumes to create high-capacity, high-performance controller cache pools"

http://www.lsi.com/channel/products/storagesw/Pages/MegaRAIDCacheCadeSoftware2-0.aspx

dmourati
  • 24,720
  • 2
  • 40
  • 69
1

SAM-QFS is Sun's existing product and was open-sourced last year. It's CDDL, so you could only directly port it to *BSD.

TRS-80
  • 2,564
  • 17
  • 15
1

Answering my own question with something I just found:

I was just updating the kernel and looking at the new stuff that has been added, and there is now a 'CACHEFILES' option which allows for caching (usually remote) filesystems to a local filesystem. I guess I could use this to cache a slower storage mechanism (HDD) to a faster one (SSD), at least for one level of hierarchy.

David Gardner
  • 1,499
  • 2
  • 13
  • 25
  • Did you try this solution? How does it perform? – skolima Nov 24 '09 at 16:17
  • 1
    Nevermind - I found out myself that CacheFS for now only supports NFS and AFS. – skolima Nov 24 '09 at 16:22
  • Using network filesystems for this isn't a bad idea, they tend to mature much faster than kernel patchsets. I wonder if Ceph or DRBD have useful modes for tiered storage. – Tobu Nov 06 '12 at 10:48
0

btier (7 years old)

autofs (recent, file based)

thistleknot
  • 161
  • 5
0

LVM2 snapshots come to mind... but you can't really do more than a single snapshot.

dyasny
  • 18,482
  • 6
  • 48
  • 63