Removed ZFS files reoccurring after remount

Question

I have a nasty situation with some of my ZFS filesystems (see list below). Files I removed therein keep reappearing after a remount (such after a reboot). One such filesystem is /root. I get an mount error because the /root directory is not empty. Fair enough, I do a rm -rf /root and then a zfs mount vol/HOME/root. Everything is fine until the next reboot, when I get the same error again, because the whole directory is in the same state as before the previous mount.

I have the same problem with /var which was so bad, I left it in the / directory without its own filesystem.

By the same logic, I should have the same problems with /home or /tmp. But I don't.

A scrub did not find any errors. What kind of effect is this and how can I correct the situation?

zfs list

NAME                               USED  AVAIL  REFER  MOUNTPOINT
vol                                310G  1.10T   198K  none
vol/GENTOO                        9.72G  1.10T   198K  none
vol/GENTOO/usr-portage            1.25G  1.10T  1.25G  /usr/portage
vol/GENTOO/usr-portage-distfiles  8.47G  1.10T  8.47G  /usr/portage/distfiles
vol/GENTOO/usr-portage-packages    151K  1.10T   151K  /usr/portage/packages
vol/HOME                          20.0G  1.10T  20.0G  /home
vol/HOME/root                     17.6M  1.10T  17.6M  /root
vol/ROOT                           279G  1.10T   198K  none
vol/ROOT/gentoo                    279G  1.10T   279G  /
vol/TMP                            562M  1.10T   562M  /tmp

Answer 1

ZFS root is often a bad idea... but maybe you can chatter +i / (when the filesystem is unmounted) to prevent writing to the empty mountpoint.

See: Prevent the possiblity of writing data to an unmounted mount point directory

Answer 2

I second @eewhite in saying ZFS root is not a good idea. Anyway, try setting xattr=sa on your ZFS filesystem and, after remounting it, check if something changed.

Also have a look here: it can be related to your problem, perhaps.