2

I have a CentOS server with two RAID arrays. The OS array and a DATA array. Long story short, recently had tons of trouble with the OS and had to reload CentOS (was 5.7 and went to 6.5). Everything is working with the OS portion now. However, I'm having an issue with my DATA array.

The array shows up as /dev/sdb but I used to have a partition (/dev/sdb1) where all my data is stored. The server doesn't see /dev/sdb1 anymore so my best guess is the partition table is somehow messed up. When I print in parted, it lists the partition type as loop:

NON-WORKING SERVER:
Model: Adaptec DATA (scsi)
Disk /dev/sdb: 59.9TB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End     Size    File system  Flags
 1      0.00B  59.9TB  59.9TB  xfs

I have another server that is an exact duplicate of this one and it appears correctly in parted and the /dev/sdb1 is visible:

WORKING SERVER:
Model: Adaptec STORAGE (scsi)
Disk /dev/sdb: 59.9TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
 1      17.4kB  59.9TB  59.9TB  xfs          primary       

Is there any way for me to fix the partition so that I don't destroy my data? I desperately need to keep the data but just can't seem to figure out if there is a way to fix the partition and/or why it's showing up as loop.

Thank you for any help!!

ADDITIONAL INFORMATION:

fdisk -l /dev/sdb:

Disk /dev/sdb: 59914.8 GB, 59914783293440 bytes
255 heads, 63 sectors/track, 7284224 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xf0b62000

blkid:

/dev/sda1: UUID="9de0d036-a86f-4557-8e09-a5ccd33c66be" TYPE="ext4" 
/dev/sda2: UUID="VYyo4L-lkdG-GivI-a6eM-pFfX-TOoV-E1LxcF" TYPE="LVM2_member" 
/dev/mapper/VolGroup-lv_root: UUID="18bbe93e-9b67-4343-8d71-71bd087ab145" TYPE="ext4" 
/dev/sdb: LABEL="Data" UUID="dfda2895-d1cd-4b3e-8453-e5c51c093260" TYPE="xfs" 
/dev/mapper/VolGroup-lv_swap: UUID="d71f193f-acd4-4aea-8d11-be2acd4575f3" TYPE="swap" 
/dev/mapper/VolGroup-lv_home: UUID="223b2be8-f9e9-4671-bc84-e5aa5f73b697" TYPE="ext4" 

parted /dev/sdb unit s print (NON-WORKING SERVER):

Model: Adaptec DATA (scsi)
Disk /dev/sdb: 117021061120s
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End            Size           File system  Flags
 1      0s     117021061119s  117021061120s  xfs

parted /dev/sdb unit s print (WORKING SERVER):

Model: Adaptec STORAGE (scsi)
Disk /dev/sdb: 117021061119s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End            Size           File system  Name     Flags
 1      34s    117021061086s  117021061053s  xfs          primary       
Jason
  • 371
  • 1
  • 7
  • 19
  • Did you try mounting the partition as normal? What is the output of `fdisk -l` and `blkid`? – ewwhite Oct 26 '14 at 19:54
  • @ewwhite Thanks for the reply! When I try to mount as normal (/dev/sdb1), it tells me /dev/sdb1 does not exist. I also added the output of fdisk and blkid. Thank you! – Jason Oct 26 '14 at 20:09
  • What happens if you mount `/dev/sdb /data`? - Show the `dmesg | tail` output. – ewwhite Oct 26 '14 at 20:11
  • @ewwhite - It mounts the volume and shows it as empty. dmesg | tail Adding 6160380k swap on /dev/mapper/VolGroup-lv_swap. Priority:-1 extents:1 across:6160380k NET: Registered protocol family 10 lo: Disabled Privacy Extensions ADDRCONF(NETDEV_UP): eth0: link is not ready e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready eth0: no IPv6 routers present XFS (sdb): Mounting Filesystem XFS (sdb): Ending clean mount usb 6-1: USB disconnect, device number 2 – Jason Oct 26 '14 at 20:17

2 Answers2

1

So someone overwrote the entire volume with a new, empty XFS file system, and your problem is that you need to find the superblock of the old one, because the superblock contains a random number that is used in checksum calculations to determine whether a particular block is part of the current file system or of an earlier iteration (otherwise, fsck on a fresh file system would stumble over the old data).

My first approach would be to create read-only loop devices with an offset, and see if I can mount them:

The standard offset is 1MB, so try first with

losetup -r -o 1M /dev/loop0 /dev/sdb
mount -r /dev/loop0 /mnt

and check if that is the file system you want. If that isn't the correct offset, you can search, with something like

d=`losetup -f`
for i in `seq 1 4096`
do
    until losetup -r -o ${i}k $d /dev/sdb
    do
        :
    done &&
    mount -r $d /mnt && break
    losetup -d $d
done

The inner loop is required because there seems to be a race condition that sometimes causes a "device or resource busy" during the loop device setup.

At the end of that loop, you should have the file system mounted -- verify that it is the one you want, and $i is the offset of the partition in kilobytes.

Then, when you have found the partition, you can clear the area in front of it to get rid of the wrong superblock, create a new partition table with the correct offset for the partition, and then run a file system check.

It is likely that there will be extensive damage on the file system if it truly has been formatted, but the offset should make at least something recoverable.

Simon Richter
  • 3,209
  • 17
  • 17
0

Dodgy.

The loop partition type actually means that there is no partition table.

A partition table basically tells the OS where to look for a partition start and where it should stop when creating a filesystem. So there is no magical thing about the table and if you have the correct information, if you simply write it to the according position, your disk should be as good as... ...er... New... Well, you get the picture... ;)

Here is what I would do:

  1. Create a backup of the current partition table of the problem host, although it is empty, using dd if=/dev/sdb of=/root/sdb-parttable.bin bs=512 count=1
  2. Then I'd use fdisk to find out the start sector and the end sector on the other server, put them down,
  3. Create a new partition using fdisk on the problem server (without doing anything else!) using the exact values from the second server.
  4. Write that partition table to disk
  5. Try to mount /dev/sdb1
  6. If this doesn't work, use dd if=/root/sdb-parttable.bin of=/dev/sdb bs=1 count=64 skip=446 seek=446 to restore the old partition table.

Disclaimer: This is a bit of a tricky operation. The information in this answer was given in hope that it might be useful and without any warranty... If you have any means backing up the disk on a byte level before taking these steps: do so!

  • Thank you VERY much for the reply! Do I need to use parted instead of fdisk since it's a 59TB volume? I did a fdisk -l on the working server and got `/dev/sdb1 1 267350 2147483647+ ee EFI GPT` so assuming the start is 1 and end is 267350? Thank you!! – Jason Oct 26 '14 at 21:15
  • Starting at sector 1 wouldn't be good. Please add the output of `parted /dev/sdb unit s print` of both servers to your question. – Markus W Mahlberg Oct 26 '14 at 21:30
  • Thank you again for the help. Original post updated with output requested. – Jason Oct 26 '14 at 21:37
  • I am not sure wether my solution will work. There may be a few sectors unreadable. Problem is that those might be crucial ones. Obviously on the non working server the partition table was simply omitted, but the same end sector was given. Have to think about that. Will get back to you tomorrow. – Markus W Mahlberg Oct 26 '14 at 21:59
  • REALLY appreciate it. – Jason Oct 26 '14 at 22:00
  • Please give some sort of contact possibility in your profile. LinkedIn preferred. – Markus W Mahlberg Oct 26 '14 at 22:01
  • My email is in my profile but not sure if it displays to other users. I'm actually not on LinkedIn (I know, I need to get with it) but I can put my email here and then delete the comment after you have it? – Jason Oct 26 '14 at 22:04
  • Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/18178/discussion-between-markus-w-mahlberg-and-jason). – Markus W Mahlberg Oct 26 '14 at 22:05