Does eCryptfs prefetch and/or buffer data?

Question

I have been experimenting with using eCryptfs as the underlying file system for MySQL data to mimic transparent data encryption for MySQL. I ran some tests to measure the overhead of decryption when MySQL reads data and I am getting strange results.

I prepared a table of size 7.5 GB and ran a full table scan (SUM) query. I have two scenarios -

1. When the underlying fs is ext4 and no encryption, the average time of the query on multiple runs - 22.5 seconds
2. eCryptfs mounted on top of ext4 -
   • When table is created for the first time the average run-time of query is 21 seconds (Since it's lesser than unencrypted one, is there some prefetching involved?)
   • Clear the MySQL buffers and re-run the query, the average run-time now is 9.6 seconds (due to eCryptfs buffering?)
   • If I umount my directory and remount it but don't recreate the data, the average time comes around 17 seconds. This is again mysterious. Did eCryptfs store some unencrypted meta information, when it decrypted for the first time?

I ran these tests multiple times (clearing MySQL buffers every time) and I got consistent results. Can someone please explain or point me to a resource, that explains this behavior? Is it possible to disable this (for the tests)?

Answer 1

I asked the same question on eCryptfs' mailing list. This is the thread -

http://comments.gmane.org/gmane.comp.file-systems.ecryptfs.general/294

One of top contributors, Tyler Hicks replied to the query, copied here (in case the link goes dead)

==BeginMessage==

Your test results don't make sense to me at first glance.

In the majority of kernel versions, eCryptfs uses write-through caching. There were a few kernel releases where a write-back cache was implemented, but it caused some problems and that patch was reverted.

For the SUM query that you're doing, I'd expect it to be all reads, so write-back or write-through shouldn't matter much. There would be a layer of caching of the encrypted pages in the lower filesystem and then another cached layer of the decrypted pages at the eCryptfs level, so it is certainly different than when you are just using plain ext4.

You talked about clearing the MySQL buffers and the eCryptfs page cache (by unmounting and remounting eCryptfs) between tests. There is still the lower filesystem page cache which isn't being cleared. Maybe that has something to do with it, but I doubt that is the full story here.

Are you using innodb_flush_method=O_DIRECT? eCryptfs doesn't implement direct I/O, so I'm not sure how MySQL would handle that on top of eCryptfs vs. ext4, which does do direct I/O.

This is something that I'd have to dig into more to make any sense of it. What you're seeing is definitely odd.

==End of Message==