A server restart produced this output on the console:
ZFS: i/o error - all block copies unavailable
ZFS: failed to read pool zroot directory object
qptzfsboot: failed to mount default pool zroot
FreeBSD/x86 boot
ZFS: i/o error - all block copies unavailable
ZFS: can't fild dataset 0
Default: zroot/<0x0>
boot:
I booted the host from the usb livecd and mounted the /etc directory under /tmp to enable ssh access:
ifconf -a # get available i/f names
ifconf em0 inet 192.168.216.46
route add default 192.168.216.1
hostname vhost06.internal
mkdir /tmp/etc
mount_unionfs /tmp/etc /etc
echo 'PermitRootLogin yes' >> /etc/sshd_config
passwd
Changing local password for root
New Password:
Retype New Password:
service sshd onestart
There are no zfs pools available to import:
root@vhost06:~ # zpool status
no pools available
root@vhost06:~ # zpool list
no pools available
root@vhost06:~ # zfs list
no datasets available
gpart
shows this geometry:
gpart show
=> 40 15628053088 ada0 GPT (7.3T)
40 1024 1 freebsd-boot (512K)
1064 984 - free - (492K)
2048 16777216 2 freebsd-swap (8.0G)
16779264 15611273216 3 freebsd-zfs (7.3T)
15628052480 648 - free - (324K)
=> 40 15628053088 ada1 GPT (7.3T)
40 1024 1 freebsd-boot (512K)
1064 984 - free - (492K)
2048 16777216 2 freebsd-swap (8.0G)
16779264 15611273216 3 freebsd-zfs (7.3T)
15628052480 648 - free - (324K)
=> 40 15628053088 ada2 GPT (7.3T)
40 1024 1 freebsd-boot (512K)
1064 984 - free - (492K)
2048 16777216 2 freebsd-swap (8.0G)
16779264 15611273216 3 freebsd-zfs (7.3T)
15628052480 648 - free - (324K)
=> 40 15628053088 ada3 GPT (7.3T)
40 1024 1 freebsd-boot (512K)
1064 984 - free - (492K)
2048 16777216 2 freebsd-swap (8.0G)
16779264 15611273216 3 freebsd-zfs (7.3T)
15628052480 648 - free - (324K)
=> 40 15628053088 diskid/DISK-VAGWJ6VL GPT (7.3T)
40 1024 1 freebsd-boot (512K)
1064 984 - free - (492K)
2048 16777216 2 freebsd-swap (8.0G)
16779264 15611273216 3 freebsd-zfs (7.3T)
15628052480 648 - free - (324K)
=> 40 15628053088 diskid/DISK-VAGWV89L GPT (7.3T)
40 1024 1 freebsd-boot (512K)
1064 984 - free - (492K)
2048 16777216 2 freebsd-swap (8.0G)
16779264 15611273216 3 freebsd-zfs (7.3T)
15628052480 648 - free - (324K)
=> 40 15628053088 diskid/DISK-VAHZAD2L GPT (7.3T)
40 1024 1 freebsd-boot (512K)
1064 984 - free - (492K)
2048 16777216 2 freebsd-swap (8.0G)
16779264 15611273216 3 freebsd-zfs (7.3T)
15628052480 648 - free - (324K)
=> 40 15628053088 diskid/DISK-VAH3PXYL GPT (7.3T)
40 1024 1 freebsd-boot (512K)
1064 984 - free - (492K)
2048 16777216 2 freebsd-swap (8.0G)
16779264 15611273216 3 freebsd-zfs (7.3T)
15628052480 648 - free - (324K)
=> 1 30240767 da0 MBR (14G)
1 1600 1 efi (800K)
1601 2012560 2 freebsd [active] (983M)
2014161 28226607 - free - (13G)
=> 0 2012560 da0s2 BSD (983M)
0 16 - free - (8.0K)
16 2012544 1 freebsd-ufs (983M)
=> 1 30240767 diskid/DISK-00241D8CE51BB011B9A694C1 MBR (14G)
1 1600 1 efi (800K)
1601 2012560 2 freebsd [active] (983M)
2014161 28226607 - free - (13G)
=> 0 2012560 diskid/DISK-00241D8CE51BB011B9A694C1s2 BSD (983M)
0 16 - free - (8.0K)
16 2012544 1 freebsd-ufs (983M)
How do I recover from here?
<------ end of original question
I have made some progress and managed to import and mount one root dataset - iocage. This is a boot on zfs system but I cannot find the dataset that contains the root filesystem so I cannot get at /var/log to see if anything is there:
mkdir /tmp/zroot # /tmp is a writable file system
zpool -f zroot # force the zpool import
zfs set mountpoint=/tmp/zroot zroot # mount the imported pool in a writable fs
zfs mount -a # find and mount all the datasets
ll /tmp/zroot
total 12
drwxr-xr-x 9 root wheel 11 Feb 27 13:09 iocage/
Fortunately, all of the absolutely critical stuff is in /zroot/iocage
as the host simply acts as a platform for the jail. However the absence of the root dataset is bothersome to me.
A zpool status
showed zroot with no errors.
I next transferred the iocage dataset to another system using zfs send
.
zfs snapshot -r zroot/iocage@vh6iocsend1
zfs send -R zroot/iocage@vh6iocsend1 | ssh 192.168.216.45 zfs receive zroot/iocagev6
This took a while but it has completed successfully.
Now I need to get the problem host started. This host was restarted yesterday at noon without a problem. I do not recall running freebsd-update fetch
, but even if I had there was nothing for fetch to deliver as the system was already at 12.1p2
I still need help getting the host to boot.
<----------
Additional notes:
I was able to mount the entire zpool
using the altroot
option of zpool import
:
- Boot into the live cd shell.
- Import the zfs pool(s) but do not allow
import
to auto mount any datasets:zpool import -o altroot=/tmp/altroot -N -a
. - Mount the root
/
dataset first:zfs mount zroot/ROOT/default
. - Now mount the remaining datasets:
zfs mount -a
.
The entire zroot
pool's file system is now accessible at /tmp/altroot
.
I have used this with zfs send
to move the contents of /var
to another host. Actually, I sent the entire pool.
However, the original system still will not boot.