21

I have a system running linux that must run unattended for long periods of time. The system uses industrial CF card for storage. Most of the time there are no writes to flash, although every now and then some configuration data/settings can be modified. The system must be resistant to power failures.

I would like to use ext4 for this. What is the best way to configure ext4 for this kind of setup? Bearing in mind that:

  • Performance is not a problem at all (especially write performance)
  • Upon power loss, the system should always boot in a clean state, even if that means that data written in the last few seconds is lost
  • If it is possible to avoid fsck, then all the better.

(I am aware of this related question: Prevent data corruption on ext4/Linux drive on power loss)

Community
  • 1
Grodriguez
  • 242
  • 1
  • 2
  • 15

4 Answers4

13

I've worked in building a system for automation on boats, and there was a prerequisite: in every moment the power could go down and everything must boostrap again correctly.

My solution was to build a Gentoo-based initramfs system, with only a rw folder for application and configurations (this is the approach used by every router/firewall vendors). This solution add an additional layer of complexity when dealing with system upgrades, but assure you that the system will ALWAYS boot.

Regarding your specific question, you should keep EXT4 journal enabled for having faster fsck (of a few secods), use the data=journal mount option, lower the commit option or use sync option to keep buffers always empty.

Refs: http://www.kernel.org/doc/Documentation/filesystems/ext4.txt

Giovanni Toraldo
  • 2,557
  • 18
  • 27
  • Good! If the application doesn't write too much data you should be happy with the sync option. – Giovanni Toraldo Feb 07 '12 at 08:30
  • 1
    The better place to look at is the Linux kernel documentation: http://www.kernel.org/doc/Documentation/filesystems/ext4.txt Enable data=journal and commit=nrsec for minimizing any potential data loss ( * == default ) – Giovanni Toraldo Feb 07 '12 at 08:46
  • The timed commits are definitely helpful - I believe you can only get down to 1 second intervals (albeit with a MAJOR performance penalty for write-intensive ops), but if you can't afford 1 second of data loss you have bigger problems ;) – voretaq7 Feb 08 '12 at 06:04
  • @voretaq7: I don't really worry about losing a few seconds worth of data (this was stated in my question). I worry about filesystem corruption and booting up in a clean state (if possible, avoiding fsck). – Grodriguez Feb 08 '12 at 07:38
  • 2
    one of the major positive effects of journaling is that recovering from a crash is a matter of replaying the latest uncommitted changes, that is really faster than checking the entire volume for inconsistencies. If this is your major problem, then you should go with the default EXT4 and be happy. – Giovanni Toraldo Feb 08 '12 at 15:12
  • 2
    @Grodriguez "losing" data can be anything from "File isn't there anymore" to "Why is there a chunk of the kernel inside my database?" -- It all depends on what is "lost" :) – voretaq7 Feb 08 '12 at 19:30
  • Gentoo? Seriously? – Tom O'Connor Feb 10 '12 at 10:50
  • This Gentoo flavour exactly: http://en.gentoo-wiki.com/wiki/TinyGentoo – Giovanni Toraldo Feb 10 '12 at 18:27
13

I will preface this by saying that as far as I'm concerned, EXT (in all of its incarnations) is a pretty awful filesystem -- I have seen more "interesting" cases of filesystem corruption in the relatively small number of Linux/EXT{2,3,4} systems I've administered than I have in the relatively large number of Not-EXT filesystems I've had occasion to use.
If at all possible try to pick a more robust filesystem. You'll thank yourself when the inevitable happens.

That being said and all my personal biases out in the open and pushed aside, EXT4 does have three features I can think of that might help you out:

  • Journaling
    EXT4 can be a Journaled filesystem, if you want it to be. Enable the journaling feature (and specifically set the data-journaling mode to journal via tune2fs or as a mount option).
    This incurs a performance hit as all data must be written out to the EXT journal before it gets "committed" to the filesystem (every write basically happens twice) but it ensures you can always recover as far as a journal replay will get you without any problems.

  • SYNChronous Mounts
    When safety is paramount mounting a filesystem with the sync option is always a good idea. This forces all writes to disk immediately - again this is a performance hit, but a good idea if you expect power failures or random strangers yanking the CF card out.

  • Limit writable filesystems as much as possible This one isn't EXT specific, but the all-too-common Linux philosophy of "just create one big root partition and dump everything into it" is, quite frankly, stupid. Create a proper filesystem structure (/, /var, /usr, /home, etc...), and mount as many of the filesystems read-only as possible.
    This used to be common advice for unix systems for the sake of security, but in your case it has an added benefit: You can't corrupt a filesystem if you can't write to it.

voretaq7
  • 79,345
  • 17
  • 128
  • 213
  • The functionality of fully synchronous mounts is *not* equivalent to simply calling `sync` after every write -- Synchronous mounts will not (or at least should not) return from a filesystem write call until the data is on disk. Calling `sync` will flush all pending writes, but there is still a window (however short) between when the write returns and your call to `sync` returns during which data may not be yet be written out to disk. – voretaq7 Feb 07 '12 at 08:15
  • 1
    Which filesystem do you recommend? Can you quantify your experiences? – Mark Wagner Feb 07 '12 at 08:36
  • @embobo my experiences are entirely anecdotal: I've never stress-tested the EXT family of filesystems, but one incident that sticks out in my mind is when I had a Squid server suffer from "Where did all my inodes go?!?" -- the filesystem got stomped on, and somehow subsequently marked clean, but every inode was somehow left in a claimed-but-never-referenced state. The fsck to fix that particular mess was positively EPIC (we wound up just making a new FS). That was the day I lost all confidence in the EXT family of filesystems. – voretaq7 Feb 07 '12 at 20:38
  • @Grodriguez Re: Journaling, the three options are `data=journal` (what I described above), `data=ordered` (metadata is journaled. Data is committed to disk before metadata is committed to the filesystem), and `data=writeback` (which is effectively no journaling/protection of data - *Bad Things* can happen after a crash, like junk in the middle of files). I believe `ordered` is the default on most Linux distros these days... – voretaq7 Feb 07 '12 at 20:41
  • 3
    In addition to "Limit writable filesystems as much as possible": In debian wiki is a guide to do exactly this with many examples on daemons which need special treatment. It should be valid for most other distris, too: http://wiki.debian.org/ReadonlyRoot – krissi Feb 08 '12 at 10:02
7

EXT4 doesn't sound like the best choice for your system; I would suggest looking at a log-structured filesystem. These work by treating data as a constant stream of write updates against a virtual stream, with a pointer that points the latest 'head'. Updates occur by writing data and metadata to the storage, then updating the pointer. In the case of a crash after the write but before the pointer update the latest data is lost but the filesystem is consistent.

Two candidate filesystems are LogFS and NILFS. Both are available in the mainline Linux kernel.

Steve Smith
  • 286
  • 1
  • 3
1

I'm intrigued about the device your building. You're after the reliability of an embedded device while using a filesystem that isn't really suited.

Ext4 (and family) is a fine general purpose filesystem with (I guess) many billions of hours of use on varied hardware and use cases. However, what your asking for doesn't really fit with ext4. The pointers from voretaq7 and Giovanni will help get the best out of using ext4 if you have to, but the real answer is to use something more suited to your requirements. Steve has given you a couple of options. If you keep pulling power from an ext4 FS you'll eventually get a mess.

If this is a one off system that you're building you should make the choice to use something more suitable or accept that there will be problems at some point. It might only be 1 power outage out of 100 or 1 out of 1000. That could be good enough for you to take the risk and the device could likely run for a long time (years) without any manual intervention.

If this is a product that you intend to widely deploy/bring to market you have the choice to use something more suitable. Or you take the business decision to support a percentage of the devices that will brick every year and either need replacing or manual intervention to recover them.

goo
  • 2,838
  • 18
  • 15