Linux LUKS and choice of filesystem

Question

Which filesystem displays the best performance when running on a LUKS encrypted partition? Does it make sense to use f2fs when the LUKS partition is on an SSD, or does the device-mapper abstraction render f2fs's SSD optimizations useless?

Answer 1

There is lots of advice, but very few empirical results for the efficiency of using encryption.

I found one such study : SSD Linux benchmarking: Comparing filesystems and encryption methods. In the table below, the measures are real/user/sys and the best results are colored green while the worst are colored in red:

click for a larger image

Based on these results, the author has decided on using dm-crypt in aes-xts-plain mode with 128 Bit keylength and btrfs with ssd alignment and compress=lzo.

Another useful article is The Performance Impact Of Linux Disk Encryption On Ubuntu 14.04 LTS. It studied the CPU usage during installation on the three configurations of stock install options (no disk encryption), full disk-encryption method using LUKS on LVM and eCryptfs-based home directory encryption. The results are summarized in the graph below, and the conclusion is that the install without encryption averaged 26%, while the LUKS on LVM and eCryptfs encryption both averaged about 30~31%, so are both almost equivalent in performance.

click for a larger image

An article that has no empirical results but lots of good advice is LinuX, SSDs and disk encryption. I recommend reading this article, and here is the main advice given:

1. For SSD partition you should at least add the mount options noatime and nodiratime to suppress the bookkeeping of access times for files and directories.
2. Set up ramdisks using tmpfs for temporary files.
3. Change the disk scheduler to use noop or deadline.
4. Enable device-level write cache
5. Parameter the browser so as not to use a disk cache.

Answer 2

I did some testing with a low-to-middle end system. Results below.

In conclusion, it seems that encryption doesn't render a filesystem's optimizations useless. As you can see in the table, for dmcrypt/LUKS (AES256), the F2FS filesystem was faster than EXT4 in almost all scenarions in which it was faster by default (without encryption, that is). Somewhat surprisingly, it also provided a noticeable benefit in cases where it woulnd't without encryption - namely, the Bonnie++ Sequential Output (Per char) and the Flexible I/O Tester Sequential Read tests.

For this same scenario it was also slower in two tests (Sequential Write - Bonnie++ and FIO), but not much slower anyway. Your mileage may vary.

I didn't test EXT4 with AES-128bits because of limited time since it'd not be essential for this answer.

Note: I observed very big, weird deviations in testing with AES-128bits. Up to 15% variations in most tests, and even 37% (!) in one case. Not sure why. I didn't use the system for anything else for the duration of the tests. The system (root) is itself encrypted with AES-256, so I can think of an hypothetical explanation involving CPU pipelines/opcache, but... Really can't tell just yet. (This also happens when testing from a live CD, without ever unlocking the AES-256bit partition, so that's not the reason).

Deviation was kept under 3,6% (1,4% typical) everywhere else. I'm adopting a 4% error margin. Thus, differences smaller than 4% should be ignored for these results.

Test setup:

SSD: Kingston SV300S37A/120G
Mainboard: ASUS Sabertooth 990FX R2.0
CPU: AMD FX-6350 @ stock
Kernel version: 4.11.3
IO Scheduler: CFQ
All tests were run using an 80GB secondary partition.

Table of results in ASCII format (differences deemed "irrelevant" are omitted):

                                                                                                                |+===================================+| 
                                                                                                                ||            % Change               ||
                                      |+=====================+=====================+=================+|         ||--------+--------+--------+--------||
                                      ||   (no encryption)   |       AES-256       |    AES-128      ||         ||   F2FS / EXT4   |Crypto / NoCrypto||
|+============+========================+----------+----------+----------+----------+------+----------||=========||--------+--------+--------+--------+|
|| Test suite | Test Method            | EXT4     | F2FS     | EXT4     | F2FS     | EXT4 | F2FS     || Unit    ||nocrypto| aes256 | aes128 | aes256 ||
|+------------+------------------------+----------+----------+----------+----------+------+----------++---------++--------+--------+--------+--------+|
|| Manual     | Read                   | 390.6    | 391.59   | 320      | 325.6    | -    | 345.36   || MiB/s   ||        |        | -11.81 | -16.85 ||
||(cp and dd) | Write (zeros)          | 501.96   | 517.17   | 96.9     | 96.7     |      | 112.16   || MiB/s   ||        |        | -78.31 | -81.30 ||
||            | Write (random data)    | 100.44   | 97.99    | 91.8     | 89.5     |      | 97.64    || MiB/s   ||        |        |        | -08.66 ||
||--------------------------------------------------------------------------------------------------------------||--------|--------|--------|--------||
||            | Seq. Output – Per char | 80.68    | 83.76    | 63.56    | 80.59    | -    | 75.56    || MiB/s   ||        | +26.79 | -09.79 |        ||
||            | Seq. Output – Block    | 498.92   | 492.42   | 104.74   | 101.13   |      | 90.9     || MiB/s   ||        |        | -81.54 | -79.46 ||
|| Bonnie++   | Seq. Output – Rewrite  | 196.4    | 198.99   | 74.69    | 70.8     |      | 70.27    || MiB/s   ||        | -05.21 | -64.69 | -64.42 ||
||            | Seq. Input - Per char  | 86.93    | 86.04    | 84.01    | 81.25    |      | 87.84    || MiB/s   ||        |        |        | -05.57 ||
||            | Seq. Input – Block     | 352.57   | 355.99   | 286.36   | 289.24   |      | 304.5    || MiB/s   ||        |        | -14.46 | -18.75 ||
||            | Random seeks           | 9452.9   | 9102.2   | 8142.3   | 8224.8   |      | 7431.4   || ops/s   ||        |        | -18.36 | -09.64 ||
||--------------------------------------------------------------------------------------------------------------||--------|--------|--------|--------||
|| Fs-mark    | 1000 files, 1 thread   | 193.7    | 236.6    | 93.6     | 103.7    | -    | 73.9     || files/s || +18.13 | +10.79 | -68.77 | -56.17 ||
||            | 5000 files, 4 threads  | 310.1    | 348.8    | 90.9     | 99       |      | 91.6     || files/s || +11.10 | +08.91 | -73.74 | -71.62 ||
||--------------------------------------------------------------------------------------------------------------||--------|--------|--------|--------||
||            | Random read            | 56.77    | 69.86    | 58.79    | 63.51    |      | 61.93    || MiB/s   ||        |        |        |        ||
||            |                        | 14188    | 17461    | 14695    | 15874    |      | 15479    || IOPS    || +23.07 | +08.02 | -11.35 | -09.09 ||
|| Flexible   |                        |          |          |          |          | -    |          ||         ||        |        |        |        ||
|| I/O        | Random write           | 59.91    | 78.99    | 52.2     | 63.75    |      | 67.75    || MiB/s   ||        |        |        |        ||
|| Tester     |                        | 14973    | 19745    | 13046    | 15935    |      | 16934    || IOPS    || +31.87 | +22.14 | -14.24 | -19.30 ||
||            |                        |          |          |          |          |      |          ||         ||        |        |        |        ||
||            | Sequential read        | 284.02   | 247.57   | 95.14    | 95.17    |      | 95.98    || MiB/s   ||        |        |        |        ||
||            |                        | 71001    | 61889    | 23781    | 23788    |      | 23991    || IOPS    || -12.83 |        | -61.24 | -61.56 ||
||            |                        |          |          |          |          |      |          ||         ||        |        |        |        ||
||            | Sequential write       | 94.06    | 100.77   | 100.63   | 96.56    |      | 86.93    || MiB/s   ||        |        |        |        ||
||            |                        | 23512    | 25188    | 25153    | 24137    |      | 21728    || IOPS    || +07.13 | -04.04 | -13.74 | -04.17 ||
|+------------+------------------------+----------+----------+----------+----------+------+----------++---------++--------+--------+--------+--------+|
_