17

I am trying to create an encrypted, growing-as-needed file system in with Linux. I am familiar with LUKS and cryptsetup.

I can create an empty file:

fallocate -l 512M /root/image

I can create a LUKS container on it:

cryptsetup -y luksFormat /root/image

And then "open" it:

cryptsetup luksOpen /root/image luksvolume

At this point, I can just make a file system on it:

mkfs.ext4 -j /dev/mapper/luksvolume

This is all fine and dandy. However, it doesn't address the "grow-on-demand" part of the question.

The idea is that copying a 2Gb file on the encrypted file system will "expand" the image so that it's big enough to contain the file.

Is it even possible to do?

Merc
  • 719
  • 1
  • 6
  • 16
  • Why not make the filesystem the right size in the first place and what problem is it you are trying to solve? – Matthew Ife Dec 12 '15 at 00:42
  • 3
    Sometimes you don't know how big you need a file system to be. The problem is having one file on an encrypted file system, and being able to add staff to it without having to 1) Worry about space run out 2) Having TONS of unused space. Plus, being able to copy that one encrypted file over somewhere else and remount it. – Merc Dec 12 '15 at 04:10

1 Answers1

22

Yes! It looks like it's possible. Let's check how it can be achieved. Note that this doesn't create a true grow-on-demand filesystem, as when the filesystem reaches the max size of the sparse file, it will report 'out of space' errors if more data still needs to be written.

Initially, I was investigating Thin Provisioning, a well-known technology to save storage space in virtualization scenarios. Unfortunately, in common Linux use-cases, it seems to be available only with LVM. As this seems a bit outside the scope of your question, I searched for something else.

The second concept I investigated is Sparse File. This is exactly suited to your question and... my initial doubt was: "OK. I can create a Sparse File. But what happens when I initialize it as a LUKS container? Will such initialization allocate all the available space? If not, what will happen when I will initialize the filesystem in such a container? Will an mkfs.ext4 allocate all the available space?". As I had no answer, I decided to try. So, let's see what happened.

Let's start from my current system, where I have only 3.3G of free space within the /repository filesystem:

root@iMac-Chiara:~# df -h /repository
File system     Dim. Usati Dispon. Uso% Montato su
/dev/sda3       275G  258G    3,3G  99% /repository

Let's create a 10G sparse-file within such a filesystem, with:

root@iMac-Chiara:~# dd of=/repository/file_container.img bs=1G count=0 seek=10
0+0 record dentro
0+0 record fuori
0 byte (0 B) copiati, 0,000119606 s, 0,0 kB/s

and let's verify that... it's really a sparse file:

root@iMac-Chiara:~# ls -lh /repository/file_container.img 
-rw-r--r-- 1 root root 10G dic 12 19:48 /repository/file_container.img

OK. So we have a 10G file, in a filesystem that previously had 3.3G of free space. How much free space still I have?

root@iMac-Chiara:~# df -h /repository
File system     Dim. Usati Dispon. Uso% Montato su
/dev/sda3       275G  258G    3,3G  99% /repository

Still 3.3G. Nice. Sparse-file are really... sparse-file ;-) Let's step ahead, by creating a LUKS container within such a 10G file and... let's see if we run out of space:

 root@iMac-Chiara:~# losetup /dev/loop0 /repository/file_container.img
 root@iMac-Chiara:~# cryptsetup -y luksFormat /dev/loop0

 WARNING!
 ========
 Ciò sovrascriverà i dati in /dev/loop0 in modo irreversibile.

 Are you sure? (Type uppercase yes): YES
 Inserire la passphrase LUKS: 
 Verify passphrase: 
 root@iMac-Chiara:~# cryptsetup luksOpen /dev/loop0 secretfs
 Inserire la passphrase per /dev/loop0: 
 root@iMac-Chiara:~#

So now I have an opened secrets container defined on top of my 10G sparse-file stored in a filesystem having only 3.3G of free space.

How much free space still I have?

 root@iMac-Chiara:~# df -h /repository
 File system     Dim. Usati Dispon. Uso% Montato su
 /dev/sda3       275G  258G    3,3G  99% /repository

Wonderful! Still 3.3GB. Our encrypted container required mostly no space!

Let's check if everything is OK or if there are something strange with our setup:

root@iMac-Chiara:~# cryptsetup status secretfs
/dev/mapper/secretfs is active.
  type:    LUKS1
  cipher:  aes-cbc-essiv:sha256
  keysize: 256 bits
  device:  /dev/loop0
  loop:    /repository/file_container.img
  offset:  4096 sectors
  size:    20967424 sectors
  mode:    read/write

Everything seems OK so let's start using such a container to store something. Let's begin by creating an EXT4 filesystem inside it:

root@iMac-Chiara:~# mkfs.ext4 /dev/mapper/secretfs 
mke2fs 1.42.5 (29-Jul-2012)
Etichetta del filesystem=
OS type: Linux
Dimensione blocco=4096 (log=2)
Dimensione frammento=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
655360 inodes, 2620928 blocks
131046 blocks (5.00%) reserved for the super user
Primo blocco dati=0
Maximum filesystem blocks=2684354560
80 gruppi di blocchi
32768 blocchi per gruppo, 32768 frammenti per gruppo
8192 inode per gruppo
Backup del superblocco salvati nei blocchi: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: fatto                           
Scrittura delle tavole degli inode: fatto                           
Creating journal (32768 blocks): fatto
Scrittura delle informazioni dei superblocchi e dell'accounting del filesystem: fatto

root@iMac-Chiara:~#

It looks like it worked, as there was no track of "out of space". Let's check:

root@iMac-Chiara:~# df -h /repository
File system     Dim. Usati Dispon. Uso% Montato su
/dev/sda3       275G  258G    3,2G  99% /repository

Uhm.... so something happened. We lost something like 100M of space but.... it's an expected behaviour: the creation of the EXT4 filesystem DO require the writing of lots of metadata. So it's normal that some space have been used by the creation process.

Is it a "working" EXT4 filesystem?

root@iMac-Chiara:~# tune2fs -l /dev/mapper/secretfs
tune2fs 1.42.5 (29-Jul-2012)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          e63321c3-cee7-478d-a6af-cbdcaf1be1f7
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              655360
Block count:              2620928
Reserved block count:     131046
Free blocks:              2541265
Free inodes:              655349
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      639
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Sat Dec 12 19:58:05 2015
Last mount time:          n/a
Last write time:          Sat Dec 12 19:58:05 2015
Mount count:              0
Maximum mount count:      -1
Last checked:             Sat Dec 12 19:58:05 2015
Check interval:           0 (<none>)
Lifetime writes:          131 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:           256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      c8b3bf1b-9f05-4267-85d3-2ecfdbaa6dc3
Journal backup:           inode blocks

Yes! It looks ok.

So now we have an EXT4 filesystem written inside an opened LUKS container defined on top of a 10G sparse-file stored within a 3.3G filesystem.

Let's see if everything works correctly, by allocating space "on-demand".

Let's start by writing 500M of dummy data to the encrypted FS

root@iMac-Chiara:~# mkdir /mnt/temp
root@iMac-Chiara:~# mount /dev/mapper/secretfs /mnt/temp
root@iMac-Chiara:~# dd if=/dev/zero of=/mnt/temp/random_data.bin bs=1M count=512
512+0 record dentro
512+0 record fuori
536870912 byte (537 MB) copiati, 2,35214 s, 228 MB/s
root@iMac-Chiara:~#

Have we been succesfull in creating the file?

root@iMac-Chiara:~# ls -lh /mnt/temp/random_data.bin 
-rw-r--r-- 1 root root 512M dic 12 20:09 /mnt/temp/random_data.bin

It looks so.

What happened to our real filesystem?

root@iMac-Chiara:~# df -h /repository
File system     Dim. Usati Dispon. Uso% Montato su
/dev/sda3       275G  259G    2,5G 100% /repository

Uau! We "lost" sligthly more than 500M. That's good, BTW, as the physical space is really allocated on demand!

Let's store another 2GB file:

root@iMac-Chiara:~# dd if=/dev/zero of=/mnt/temp/another_random_data.bin bs=1G count=2
2+0 record dentro
2+0 record fuori
2147483648 byte (2,1 GB) copiati, 25,6539 s, 83,7 MB/s
root@iMac-Chiara:~#

What happened?

root@iMac-Chiara:~# ls -arlh /mnt/temp
totale 2,6G
-rw-r--r-- 1 root root 512M dic 12 20:09 random_data.bin
drwx------ 2 root root  16K dic 12 19:58 lost+found
-rw-r--r-- 1 root root 2,0G dic 12 20:13 another_random_data.bin
drwxr-xr-x 8 root root 4,0K mag 29  2015 ..
drwxr-xr-x 3 root root 4,0K dic 12 20:12 .
root@iMac-Chiara:~# df -h /repository
File system     Dim. Usati Dispon. Uso% Montato su
/dev/sda3       275G  261G    484M 100% /repository
root@iMac-Chiara:~#

Really nice. What happens if we delete a file?

root@iMac-Chiara:~# rm /mnt/temp/random_data.bin 
root@iMac-Chiara:~# sync
root@iMac-Chiara:~# ls -arlh /mnt/temp
totale 2,1G
drwx------ 2 root root  16K dic 12 19:58 lost+found
-rw-r--r-- 1 root root 2,0G dic 12 20:13 another_random_data.bin
drwxr-xr-x 8 root root 4,0K mag 29  2015 ..
drwxr-xr-x 3 root root 4,0K dic 12 20:14 .
root@iMac-Chiara:~# df -h /repository
File system     Dim. Usati Dispon. Uso% Montato su
/dev/sda3       275G  261G    484M 100% /repository
root@iMac-Chiara:~#

As expected, with sparse-file the behaviour is exactly like thin-provisioning: once allocated, storage-space cannot be claimed back when file are deleted. But this, in general, is OK. Don't you?

So at this point, the answer to your question should be complete. Right?


Addition:

Let's see what happens when the underlining storage gets full:

root@iMac-Chiara:~# dd if=/dev/zero of=/mnt/temp/a_third_random_data.bin bs=1G count=2
2+0 record dentro
2+0 record fuori
2147483648 byte (2,1 GB) copiati, 26,7142 s, 80,4 MB/s
root@iMac-Chiara:~#

What? it looks like it succeded! How this have been possible? Let's check!

root@iMac-Chiara:~# ls -arlh /mnt/temp
totale 4,1G
drwx------ 2 root root  16K dic 12 19:58 lost+found
-rw-r--r-- 1 root root 2,0G dic 12 20:17 a_third_random_data.bin
-rw-r--r-- 1 root root 2,0G dic 12 20:13 another_random_data.bin
drwxr-xr-x 8 root root 4,0K mag 29  2015 ..
drwxr-xr-x 3 root root 4,0K dic 12 20:17 .
root@iMac-Chiara:~#

Uhm... It looks ok. Are we sure?

root@iMac-Chiara:~# df /repository
File system    1K-blocchi     Usati Disponib. Uso% Montato su
/dev/sda3       288110208 275070448         0 100% /repository

we have run out of space! Without any error!

Even if it would be nice to investigate what really happened... I'm going to leave this to your curiosity and/or troubleshooting skills of other ServerFault members ;-)

Have fun!


BTW: I've tested all of the above, here:

root@iMac-Chiara:~# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=13.04
DISTRIB_CODENAME=raring
DISTRIB_DESCRIPTION="Ubuntu 13.04"
root@iMac-Chiara:~# uname -r
3.8.0-31-generic
root@iMac-Chiara:~# dpkg -l cryptsetup-bin
[...]
ii  cryptsetup-bin             2:1.4.3-4ubuntu2   amd64              disk encryption support - command line tools
root@iMac-Chiara:~#
Damiano Verzulli
  • 3,948
  • 1
  • 20
  • 30
  • I noticed that you have to be root for those commands to work. Is that always the case for sparse files? – Merc Dec 13 '15 at 03:06
  • No.Sorry. They should work as normal user as well, given proper write permission on the main folder. – Damiano Verzulli Dec 13 '15 at 08:38
  • Thanks for this great answer. Leaves me with a question and a worry. Worry: Pretending to have successfully written that second 2GB file when there was really no space for it? Troublesome... What happens when you try to read it back (with sha1sum or something)? Question: Are there ways to back up a sparse file across the network that keeps it sparse (i.e. only actually copies the parts that are used)? – Thilo Dec 13 '15 at 11:47
  • I was tempted to futher investigate but.... unfortunately I was short on time and, indeed, it's definitely a space valid for another different SF question. Anyway, it can be easily avoided by not overbooking your overall storage: I mean, you can create sparse-files but... so to have the maximum total allocatable space fitting in your phisical disk. Don't you? If, instead, you're searching for "overbooking" solutions.... than maybe something else should be investigated (LVM?) – Damiano Verzulli Dec 13 '15 at 12:39
  • @Thilo I'm also curious what would happen if you tried to read the file that silently overflowed. `rsync` has a `--sparse` option that should create sparse files on the destination disk. – localhost Oct 26 '16 at 13:52
  • That was so helpful I used it in my latest script! published on github and gave credits, please let me know if any issue. https://github.com/jupiter126/blortchzj – user1747036 Dec 14 '17 at 01:57
  • `cryptsetup -y luksFormat /dev/loop0` fails for me with `Cannot format device /dev/loop0 which is still in use.` – Michael Feb 18 '20 at 21:22
  • I suspect the reason why you did not run out of space when creating the last file, is that you created a sparse file in the sparse file-filesystem,... so to say. – leonixyz Oct 30 '21 at 13:26