41

I am trying to expose a hierarchy of home directories to a number of FreeBSD jails. The home directories are configured such that each is a unique ZFS dataset. The jails are used for development work and hence are created and destroyed on a regular basis.

My first thought was simply to use nullfs to mount /home inside the jail, but nullfs doesn't provide any way to access subordinate filesystems.

My second thought was to export the directories via NFS and then run the automounter daemon (amd) inside each jail. This would have Just Worked...if it were possible to perform NFS mounts inside a jail. But it's not.

My third thought was to run amd on the host and provision nullfs mounts into the jails...but amd support for nullfs doesn't exist.

My fourth thought was to go back to exporting the directories using NFS, because of course amd works with NFS, right? Unfortunately, rather than mounting a directory on the target mountpoint, amd likes to mount things in a temporary location (/.amd_mnt/...) and then create a symlink...which, of course, is useless inside the jail environment.

So maybe you could use nullfs to expose a subdirectory of /.amd_mnt to the jail? No! This brings us back to my first attempt, in which we find that there is no way to access subordinate filesystems using nullfs.

And then my head exploded.

Is there a good solution for what I'm trying to do? A bad solution would be to run a script after booting the jail that would create multiple nullfs mountpoints for each home directory, but this is pretty clunky -- it would need to be run periodically to take into account new directories or removed directories. So basically I would have to write a bad automounter.

There must be a better way. Help me, Serverfault, you're my only hope!

UPDATE 1: It occurred to me that I might be able to solve part of the problem with pam_mount, although this would at best be incomplete. Also, it's not clear from the documentation whether or not pam_mount can automatically create the target mountpoint. If it requires the mountpoint to exist a priori, this solution wouldn't be any better than the bad automounter I've already proposed.

UPDATE 2: As discussed in the answers below, setting VFCF_JAIL on the NFS filesystem does allow jails to perform NFS mounts. Unfortunately, the automounter continues to behave that are unhelpful, and when run in a jail appears to be very good at getting wedged in such a way that a system reboot is necessary to remove the process entry.

larsks
  • 41,276
  • 13
  • 117
  • 170
  • Bind mounts or [polyinstantiation](http://www.ibm.com/developerworks/linux/library/l-polyinstantiation/) would be perfect for this. – Ignacio Vazquez-Abrams Jun 16 '11 at 13:34
  • 1
    And if you're willing to port those features over to FreeBSD that would be awesome. Seriously, the behavior of the standard Linux automounter would be perfect for this, too, since it actually mounts filesystems on the target mountpoint, rather than using symlinks. – larsks Jun 16 '11 at 13:40
  • 4
    I blame a severe lack of time. And perhaps just a little bit of a lack of motivation... >.> – Ignacio Vazquez-Abrams Jun 16 '11 at 13:42
  • It was worth a shot :). – larsks Jun 16 '11 at 13:42
  • 15
    +1 Solely for the awesome title. :) – Shauna Jun 16 '11 at 17:12
  • I don't know if it would fit your environment, but for me the scripts that setup the jail (copying files and whatnot) also mount any common file systems. And on the flip side, the script that tears it all down also unmounts. – Chris S Jun 17 '11 at 00:57
  • The problem here is that the list of filesystems changes regularly...and because each filesystem is a distinct ZFS dataset, you have to create a mountpoint for *each one of them*. You can't simply add /home to /etc/fstab.. Neither the stock jail scripts nor tools like ezjail deal with this at all. – larsks Jun 17 '11 at 01:17
  • 4
    It concerns me how specific your particular method of execution is – Mark Henderson Jun 17 '11 at 01:22
  • @larsks - I was referring to the question title, sorry – Mark Henderson Jun 17 '11 at 01:37
  • Ah, see, I've been up too long. And my forehead hurts from beating it against this problem. I should give it a rest. – larsks Jun 17 '11 at 01:38
  • 2
    Oh, @Wilshire, you're no fun. – larsks Jun 17 '11 at 18:22
  • easier to search in the long run, besides I thought you might like the boost to the top of the active questions to get some more eyes. ;-p – Wilshire Jun 17 '11 at 18:28
  • There is that happy result, it's true, and it will be somewhat more obvious in someone's search results. Now all we need are some answers... – larsks Jun 17 '11 at 18:30
  • Did you get to a solution to this problem? Btw, thanks for the laughs.. many +1's. – pauska Jul 28 '11 at 18:44
  • I did not. I have thrown up my hands in defeat. – larsks Jul 28 '11 at 19:58

1 Answers1

26

Why, hello there Lars! That's a fascinating question you've asked, and after some research I may have found an answer for you.

According to this and other posts out there, it may be possible to set the VFCF_JAIL attribute on the NFS filesystem provider, which would in theory allow jails to perform NFS mounts. This may, in turn, allow one to run amd inside the jail...which would neatly solve this issue.

I'll try rebuilding the kernel this evening to see how things work out. It's not necessarily the greatest solution (since it means you need to make sure this change persists across future kernel updates), but it would be interesting if it works.

And remember...

           ___________    ____                                           
    ______/   \__//   \__/____\                             
  _/   \_/  :           //____\\                             
 /|      :  :  ..      /        \                         
| |     ::     ::      \        /                             
| |     :|     ||     \ \______/     Don't try to rebuild the
| |     ||     ||      |\  /  |        kernel remotely because
 \|     ||     ||      |   / | \         you know you're just
  |     ||     ||      |  / /_\ \        going to hose the server.
  | ___ || ___ ||      | /  /    \    
   \_-_/  \_-_/ | ____ |/__/      \   
                _\_--_/    \      /   
               /____             /  
              /     \           /    
              \______\_________/     
larsks
  • 41,276
  • 13
  • 117
  • 170
  • 6
    Thanks, Lars; I wish I had taken your advice, because now I can't access the server and I'll have to wait until I get home to try this out. – larsks Jun 17 '11 at 19:52
  • 10
    +1 for the LOLs. Also consider this a right of passage - Driving out to your server in the middle of the night to boot KERNEL.OLD (or KERNEL.GENERIC - You do keep an up-to-date GENERIC on your systems, right? If not, you should start.) teaches many lessons... – voretaq7 Jun 17 '11 at 20:36
  • I keep a net-bootable copy of [mfsbsd](http://mfsbsd.vx.sk/) handy, too, which has saved me from a number of boot-loader related problems. – larsks Jun 17 '11 at 20:45
  • 1
    Setting `VFCF_JAIL` made NFS mounts work, but the automounter continues to bedevil me. – larsks Jun 19 '11 at 12:47
  • Automount is not suitable for production use. (Not just on FreeBSD - In my experience it sucks everywhere!) – voretaq7 Jun 20 '11 at 16:31
  • 1
    Eh, I've been using automount for home directories for better than a decade in various environments and I've been happy with it. Mind you, this is the Linux (and Solaris) automounter, not amd. – larsks Jun 20 '11 at 19:39
  • KVMoIP FTW, mamma jamma. – Wesley Jul 28 '11 at 18:52