BTRFS won't mount and can't replace a disk

0

Here's my situation, I have a NAS that I use as a backup server, as well as another Linux machine that had a full backup as well. The NAS has three 2TB drives, so I decided to try BTRFS with RAID5 to see how well that was working. I mirrored both and everything was well. Later I decided to upgrade the OS on the second machine, so I deleted the folder with the second backup, (the data is still potentially there). Thinking I had plenty of time with the NAS backup in place, I didn't get around to making the second backup, (this happened within the last 2 weeks). Now my NAS has a bad hard drive, which I replaced already, and I can't get the BTRFS system to mount up. I started reading this website and ran the -repair option, which seemed like a good idea at the time, but later in the thread it's pointed out that it is not.

So now I'm at a full stop so I don't damage anything further. dmesg states that BTRFS can't mount the device because a replace operation is ongoing, but btrfs replace status /mnt says that it was never started.

My questions:
1) Is it still possible to recover and backup my BTRFS drive?
2) If so, what information do I need to provide to get this underway?
3) If not, are there any tools available right now that can restore my other backup? I think I formatted the drive from ext4 to xfs.

Thanks in advance.

Edit: I forgot to mention that I did start the replace initially by using the information here. And that it had to be restarted due to a power failure.

Edit 2: Would it be possible to have the RAID5 downgrade from 3 drives to just 2 without the other drive in there?

Edit 3: Getting a kernel bug when I let the system just sit there:

[  685.809945] ------------[ cut here ]------------
[  685.810386] kernel BUG at /build/linux-a2WvEb/linux-4.4.0/fs/btrfs/raid56.c:1917!
[  685.810833] invalid opcode: 0000 [#1] SMP 
[  685.811279] Modules linked in: crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper input_leds cryptd snd_hda_codec_realtek serio_raw snd_hda_codec_generic snd_hda_codec_hdmi shpchp snd_hda_intel k10temp snd_hda_codec snd_hda_core snd_hwdep snd_pcm i2c_piix4 snd_seq_midi snd_seq_midi_event snd_rawmidi nuvoton_cir rc_core snd_seq snd_seq_device snd_timer snd mac_hid soundcore kvm_amd kvm irqbypass nfsd auth_rpcgss parport_pc nfs_acl lockd ppdev grace lp sunrpc parport autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear pata_acpi hid_generic usbhid hid amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt r8169 fb_sys_fops mii drm pata_atiixp
[  685.813359]  ahci libahci fjes
[  685.813876] CPU: 1 PID: 2553 Comm: btrfs-transacti Not tainted 4.4.0-36-generic #55-Ubuntu
[  685.814400] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./FM2A85X-ITX, BIOS P1.50 05/04/2013
[  685.814931] task: ffff8800db2cb700 ti: ffff8800d3a54000 task.ti: ffff8800d3a54000
[  685.815462] RIP: 0010:[]  [] __raid_recover_end_io+0x475/0x520 [btrfs]
[  685.816028] RSP: 0018:ffff8800d3a57720  EFLAGS: 00010213
[  685.816568] RAX: ffff8800d3ce5700 RBX: ffff8800d0c66000 RCX: 0000000000000000
[  685.817112] RDX: 0000000080000000 RSI: 0000000000000000 RDI: 0000000000000003
[  685.817658] RBP: ffff8800d3a57760 R08: ffffea00035596c0 R09: 0000000000000018
[  685.818207] R10: ffff8800d0c1d9a0 R11: ffffffff81ccf5ea R12: ffff8800d0c1d9b0
[  685.818759] R13: 0000000000000003 R14: ffff8800d0c1d9a0 R15: ffff8800d2f87000
[  685.819316] FS:  00007f108f26b700(0000) GS:ffff88011ec80000(0000) knlGS:0000000000000000
[  685.819876] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  685.820436] CR2: 0000000000ef5000 CR3: 0000000001e0a000 CR4: 00000000000406e0
[  685.820999] Stack:
[  685.821557]  0000000000000000 0000000000000010 0000000100000001 0000000000000000
[  685.822128]  0000000000000010 ffff8800d2f87000 0000000000000003 ffff8800d2f87000
[  685.822695]  ffff8800d3a577b0 ffffffffc04169d5 ffff8800d3a57770 0000000000000000
[  685.823258] Call Trace:
[  685.823832]  [] __raid56_parity_recover+0x1e5/0x200 [btrfs]
[  685.824410]  [] raid56_parity_recover+0xf6/0x160 [btrfs]
[  685.824990]  [] btrfs_map_bio+0xe3/0x320 [btrfs]
[  685.825557]  [] ? kmem_cache_alloc+0x187/0x1f0
[  685.826137]  [] ? btrfs_bio_wq_end_io+0x28/0x70 [btrfs]
[  685.826720]  [] btree_submit_bio_hook+0xff/0x110 [btrfs]
[  685.827305]  [] submit_one_bio+0x6d/0xa0 [btrfs]
[  685.827886]  [] read_extent_buffer_pages+0x193/0x2f0 [btrfs]
[  685.828464]  [] ? free_root_pointers+0x60/0x60 [btrfs]
[  685.829040]  [] btree_read_extent_buffer_pages.constprop.51+0xac/0x110 [btrfs]
[  685.829621]  [] btrfs_read_buffer+0x2c/0x30 [btrfs]
[  685.830199]  [] read_block_for_search.isra.34+0xe4/0x380 [btrfs]
[  685.830781]  [] ? unlock_up+0x172/0x1a0 [btrfs]
[  685.831363]  [] btrfs_search_slot+0x381/0xa00 [btrfs]
[  685.831946]  [] lookup_inline_extent_backref+0xed/0x5a0 [btrfs]
[  685.832532]  [] ? set_extent_dirty+0x32/0x40 [btrfs]
[  685.833114]  [] ? btrfs_alloc_path+0x1a/0x20 [btrfs]
[  685.833696]  [] __btrfs_free_extent.isra.70+0x113/0xd30 [btrfs]
[  685.834285]  [] ? btrfs_merge_delayed_refs+0x66/0x650 [btrfs]
[  685.834873]  [] __btrfs_run_delayed_refs+0xaab/0x11f0 [btrfs]
[  685.835452]  [] ? lock_timer_base.isra.22+0x54/0x70
[  685.836038]  [] btrfs_run_delayed_refs+0x7d/0x2a0 [btrfs]
[  685.836627]  [] ? btrfs_wait_pending_ordered+0x22/0x90 [btrfs]
[  685.837214]  [] btrfs_commit_transaction+0x4d2/0xa90 [btrfs]
[  685.837801]  [] transaction_kthread+0x229/0x240 [btrfs]
[  685.838390]  [] ? btrfs_cleanup_transaction+0x570/0x570 [btrfs]
[  685.838970]  [] kthread+0xd8/0xf0
[  685.839548]  [] ? kthread_create_on_node+0x1e0/0x1e0
[  685.840132]  [] ret_from_fork+0x3f/0x70
[  685.840718]  [] ? kthread_create_on_node+0x1e0/0x1e0
[  685.841304] Code: 75 d4 89 45 d0 e9 94 fe ff ff 8b 5d d0 4c 89 f1 be 00 10 00 00 89 da ff 15 c9 dc eb ff 8b 45 d4 89 5d d4 89 45 d0 e9 72 fd ff ff  0b 4d 8d a7 98 00 00 00 4c 89 e7 e8 ea 75 41 c1 f0 41 80 8f 
[  685.842653] RIP  [] __raid_recover_end_io+0x475/0x520 [btrfs]
[  685.843314]  RSP 
[  685.843980] ---[ end trace d312a2207e01141e ]---

Blerg

Posted 2016-09-18T16:22:48.443

Reputation: 1 094

Answers

0

Reminder: btrfs’ raid5 mode is highly unstable, especially for recovery.

The best solution for you would be to use the rescue tools (man 8 btrfs-restore) and to save whatever data you can.

Damien Clauzel

Posted 2016-09-18T16:22:48.443

Reputation: 235