I have a RAID-1 BTRFS setup with two 8 TB hard drives. Due to unclean reboots, I cannot mount them anymore, even with recovery flags:
rockpro64:~:# mount -t btrfs -o rootflags=recovery,nospace_cache /dev/sda /srv/dev-disk-by-uuid-7aac1145-798b-4941-91e2-d6bada05c748
mount: /srv/dev-disk-by-uuid-7aac1145-798b-4941-91e2-d6bada05c748: wrong fs type, bad option, bad superblock on /dev/sda, missing codepage or helper program, or other error.
After doing some research, I found someone else with a similar problem who was able to solve their issue by changing the uuid with btrfstune -u /dev/sda. So I ran that command and now my drives seem to be stuck in a changing fsid situation. In order to finish changing the uuid, I'm supposed to run that command again, but running that command having both drives connected, gives me the following errors:
rockpro64:~:# btrfs check --readonly /dev/sda
Opening filesystem to check...
ERROR: Filesystem UUID change in progress
ERROR: cannot open file system
rockpro64:~:# btrfstune -u /dev/sda
WARNING: it's recommended to run 'btrfs check --readonly' before this operation.
The whole operation must finish before the filesystem can be mounted again.
If cancelled or interrupted, run 'btrfstune -u' to restart.
We are going to change UUID, are your sure? [y/N]: y
Current fsid: 7aac1145-798b-4941-91e2-d6bada05c748
New fsid: 7aac1145-798b-4941-91e2-d6bada05c748
Set superblock flag CHANGING_FSID
Change fsid in extents
parent transid verify failed on 2500093132800 wanted 562468 found 562371
parent transid verify failed on 2500093132800 wanted 562468 found 562371
parent transid verify failed on 2500093132800 wanted 562468 found 562371
Ignoring transid failure
parent transid verify failed on 728970788864 wanted 562440 found 562429
parent transid verify failed on 728970788864 wanted 562440 found 562429
parent transid verify failed on 728970788864 wanted 562440 found 562429
Ignoring transid failure
parent transid verify failed on 728779194368 wanted 562441 found 562431
parent transid verify failed on 728779194368 wanted 562441 found 562431
parent transid verify failed on 728779194368 wanted 562441 found 562431
Ignoring transid failure
parent transid verify failed on 728779210752 wanted 562441 found 562431
parent transid verify failed on 728779210752 wanted 562441 found 562431
parent transid verify failed on 728779210752 wanted 562441 found 562431
Ignoring transid failure
parent transid verify failed on 729093505024 wanted 562440 found 562429
parent transid verify failed on 729093505024 wanted 562440 found 562429
parent transid verify failed on 729093505024 wanted 562440 found 562429
Ignoring transid failure
parent transid verify failed on 728622661632 wanted 562441 found 562430
parent transid verify failed on 728622661632 wanted 562441 found 562430
parent transid verify failed on 728622661632 wanted 562441 found 562430
Ignoring transid failure
parent transid verify failed on 728977375232 wanted 562440 found 562433
parent transid verify failed on 728977375232 wanted 562440 found 562433
parent transid verify failed on 728977375232 wanted 562440 found 562433
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=2500078845952 item=164 parent level=1 child bytenr=728977375232 child level=1
ERROR: failed to change UUID of metadata: -5
ERROR: btrfstune failed
It seems that the device journal and the data on the disks are out of sync. I tried to btrfs rescue zero-log but this does not work because a filesystem uuid change is already in progress:
rockpro64:~:# btrfs rescue zero-log /dev/sda
ERROR: Filesystem UUID change in progress
ERROR: could not open ctree
So I tried to disconnect the hard drive which causes the errors above and ran the command again, in order to put it out of its change fsid mode and mount it. But this causes btrfstune to segfault.
rockpro64:~:# btrfstune -u /dev/sda
warning, device 2 is missing
WARNING: it's recommended to run 'btrfs check --readonly' before this operation.
The whole operation must finish before the filesystem can be mounted again.
If cancelled or interrupted, run 'btrfstune -u' to restart.
We are going to change UUID, are your sure? [y/N]: y
Current fsid: 7aac1145-798b-4941-91e2-d6bada05c748
New fsid: 7aac1145-798b-4941-91e2-d6bada05c748
Set superblock flag CHANGING_FSID
[1] 2764 segmentation fault btrfstune -u /dev/sda
So, I'm pretty much stuck right now. I cannot finish changing the fsid, because the second hard drive has errors. I also cannot zero the journal to fix those errors, because there is an ongoing fsid change. I'm going to use btrfs recover overnight in order to backup the data on the disks but I'd really like to fix the issues.
Do you have any ideas what I can try next? If required, I can also provide dumps from the superblocks if required (which look okay except they have the CHANGING_FSID flag). I already thought about hex editing the CHANGING_FSID flag away... Any suggestions on this?
