18

Its a relatively common problem when something goes wrong in a SAN for ext3 to detect the disk write errors and remount the filesystem read-only. Thats all well and good, only when the SAN is fixed I can't figure out how to re-re-mount the filesystem read-write without rebooting.

Behold: 

[root@localhost ~]# multipath -ll
mpath0 (36001f93000a310000299000200000000) dm-2 XIOTECH,ISE1400
[size=1.1T][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 1:0:0:1 sdb 8:16  [active][ready]
\_ 2:0:0:1 sdc 8:32  [active][ready]
[root@localhost ~]# mount /dev/mapper/mpath0 /mnt/foo
[root@localhost ~]# touch /mnt/foo/blah

All good, now I yank the LUN out from under it.

[root@localhost ~]# touch /mnt/foo/blah
[root@localhost ~]# touch /mnt/foo/blah
touch: cannot touch `/mnt/foo/blah': Read-only file system
[root@localhost ~]# tail /var/log/messages
Mar 18 13:17:33 localhost multipathd: sdb: tur checker reports path is down
Mar 18 13:17:34 localhost multipathd: sdc: tur checker reports path is down
Mar 18 13:17:35 localhost kernel: Aborting journal on device dm-2.
Mar 18 13:17:35 localhost kernel: Buffer I/O error on device dm-2, logical block 1545
Mar 18 13:17:35 localhost kernel: lost page write due to I/O error on dm-2
Mar 18 13:17:36 localhost kernel: ext3_abort called.
Mar 18 13:17:36 localhost kernel: EXT3-fs error (device dm-2): ext3_journal_start_sb:   Detected aborted journal                      
Mar 18 13:17:36 localhost kernel: Remounting filesystem read-only

It only thinks its read-only, in reality its not even there.

[root@localhost ~]# multipath -ll
sdb: checker msg is "tur checker reports path is down"
sdc: checker msg is "tur checker reports path is down"
mpath0 (36001f93000a310000299000200000000) dm-2 XIOTECH,ISE1400
[size=1.1T][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:0:1 sdb 8:16  [failed][faulty]
 \_ 2:0:0:1 sdc 8:32  [failed][faulty]
[root@localhost ~]# ll /mnt/foo/
ls: reading directory /mnt/foo/: Input/output error
total 20
-rw-r--r-- 1 root root     0 Mar 18 13:11 bar

How it still remembers that 'bar' file being there... mystery, but not important right now. Now I re-present the LUN:

[root@localhost ~]# tail /var/log/messages
Mar 18 13:23:58 localhost multipathd: sdb: tur checker reports path is up
Mar 18 13:23:58 localhost multipathd: 8:16: reinstated
Mar 18 13:23:58 localhost multipathd: mpath0: queue_if_no_path enabled
Mar 18 13:23:58 localhost multipathd: mpath0: Recovered to normal mode
Mar 18 13:23:58 localhost multipathd: mpath0: remaining active paths: 1
Mar 18 13:23:58 localhost multipathd: dm-2: add map (uevent)
Mar 18 13:23:58 localhost multipathd: dm-2: devmap already registered
Mar 18 13:23:59 localhost multipathd: sdc: tur checker reports path is up
Mar 18 13:23:59 localhost multipathd: 8:32: reinstated
Mar 18 13:23:59 localhost multipathd: mpath0: remaining active paths: 2
Mar 18 13:23:59 localhost multipathd: dm-2: add map (uevent)
Mar 18 13:23:59 localhost multipathd: dm-2: devmap already registered
[root@localhost ~]# multipath -ll
mpath0 (36001f93000a310000299000200000000) dm-2 XIOTECH,ISE1400
[size=1.1T][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][enabled]
 \_ 1:0:0:1 sdb 8:16  [active][ready]
 \_ 2:0:0:1 sdc 8:32  [active][ready]

Great right? It says [rw] right there. Not so fast:

[root@localhost ~]# touch /mnt/foo/blah
touch: cannot touch `/mnt/foo/blah': Read-only file system

OK, doesn't do it automatically, I'll just give it a little push:

[root@localhost ~]# mount -o remount /mnt/foo
mount: block device /dev/mapper/mpath0 is write-protected, mounting read-only

The hell you are:

[root@localhost ~]# mount -o remount,rw /mnt/foo
mount: block device /dev/mapper/mpath0 is write-protected, mounting read-only

Noooooooooo.

I have tried all sorts of different mount/tune2fs/dmsetup commands and I cannot figure out how to get it to un-flag the block device as write-protected. Rebooting will fix it, but I'd much rather do it on-line. An hour of googling has gotten me nowhere either. Save me ServerFault.

cagenut
  • 4,808
  • 2
  • 23
  • 27
  • 3
    hmm, couple of questions *'Its a relatively common problem when something goes wrong in a SAN'* why is your SAN so unreliable, I'd check that out first? Have you tried just unmounting with umount, and then mounting it again? Is there a good reason why you need to do a remount?. I usually only need to remount my root filesystems after maintainace. – The Unix Janitor Mar 18 '10 at 23:00
  • umount bounces on open file handles, which are often from processes you'd much rather have exit sanely. – cagenut Mar 19 '10 at 15:47
  • I have a similar issue where after a SAN issue VMs disks are read only and attempting to remount causes the same error in the OP. VMs are on esxi 4.1 with fibre channel storage. Reboot of VM fixes the problem. I dont personally think that this is anything to do with multipath. Surely there must be a way to fix without rebooting, especially since some services (apache) tend to keep running on a read only FS. – Will Dec 26 '12 at 21:22
  • I came here looking for a solution to my own problem (which is different, a corrupt disk). I smiled instead. +1 for "The hell you are" – user1207217 Mar 23 '13 at 08:51
  • I have the exact same issue as this, but I'm using LVM. Same lvdisplay would give me a "read failed after 0 of 4096 at 449197309952: Input/output error" until I did a "multipath -r", then LVM started displaying everything right without errors. I still can't get the partition to remount, though. Can't unmount either, says device is busy. If I shut down all processes using the device, I can unmount and then remount successfully, but I'd prefer just being able to remount the device read-write, as I should be able to... – mpontes May 15 '13 at 10:06

7 Answers7

6

I just recently ran into this problem and solved it by rebooting but after further investigation it appears that issuing the following command might fix it.

echo running > /sys/block/device-name/device/state

I think you might want to look at look at section 25.14.4: Changing the Read/Write State of an Online Logical Unit in this document, however, I recommend rebooting.

Desperatuss0ccus
  • 252
  • 1
  • 4
  • 9
specialKevin
  • 61
  • 1
  • 1
  • Thanks Kevin. (Un)fortunately the problem is long gone so I can't test but this looks like the most promising option. – cagenut Feb 09 '12 at 17:11
  • 3
    In a similar issue I have experienced /sys/block/device-name/device/state was already set to 'running' and the above command did not resolve the issue. – Will Dec 26 '12 at 21:24
3

Try using:

mount -o remount,rw /mnt/fo
Desperatuss0ccus
  • 252
  • 1
  • 4
  • 9
  • I know FreeBSD, not Linux. But for fBSD it's `mount -rw /mnt/foo`, so this one looks the most right to me. – Chris S Mar 19 '10 at 00:27
  • 1
    I have never had this work in the scenario outlined in the question. Once the disk is marked read-only due to errors, it has always taken a reboot for me. – Alex Mar 19 '10 at 00:52
  • 1
    I'll edit this into the OP, but Alex is right here, the problem appears to be below the filesystem: [root@localhost ~]# mount -o remount,rw /mnt/foo mount: block device /dev/mapper/mpath0 is write-protected, mounting read-only – cagenut Mar 19 '10 at 15:37
  • 1
    Have you tried unmounting the partition and remounting it? I've had data errors before with a drive, unmounting (or remount,rw) has fixed it for me. This was with SATA drives (and older EIDE/SCSI) However, in your situation, I am wondering if the issue is that the drive channel needs to be reset. I'm wondering if HDIO_DRIVE_RESET sent through ioctl somehow. blockdev can be used to force rereading of the partition table which might do it. IDE exposes this with hdparm -w, perhaps with your FC drives, you've got a way to send the ioctl to the channel. –  Mar 20 '10 at 00:38
2

I am a fan of preventing the issue in the first place. Most enterprise UNIX boxes will retry filesystem operations like forever. You as an administrator need to do some homework before tuning your MPIO configuration. If your application should wait until the device return to a usable state, then here is a solution. In your /etc/multipath.conf make sure that the device type you care about has a setting for "no_path_retry" set to "queue". Setting this will cause failed I/Os to queue until there is a valid path. We have done this for our EMC Symmtrix/DMX boxes to work about hiccups under certain conditions drive/controller/srdf path failures/recovery. When you want to fail the device manually during a failure it gets more complicated as you will need to use tools like dmsetup to flush/fail I/Os or temporarily change the multipath.conf file and rescan devices....etc.

This approach has saved our bacon countless times and is our standard for hundreds of boxes on a multicabinet/multivendor SAN with replication for disaster recovery.

Just thought I might share with you all. Take care.

TomF
  • 21
  • 1
2

I had the some issue, which I resolved using hdparm with the -r option on subdrives of logical, multipath devices.

-r Get/set read-only flag for the device. When set, Linux disallows write operations on the device.

Drifter104
  • 3,693
  • 2
  • 22
  • 39
c4f4t0r
  • 5,149
  • 3
  • 28
  • 41
1

Do you think it's related to the section in this document titled Why does the ext3 filesystems on my Storage Area Network (SAN) repeatedly become read-only?

It's quite an old article, and is talking about fibre channel, but it may be related to your problem.

Desperatuss0ccus
  • 252
  • 1
  • 4
  • 9
The Unix Janitor
  • 2,388
  • 14
  • 13
  • Yep, its not that exact specific bug since I'm running much newer versions than those they reference, but all sorts of similar situations can cause it. The world of fibre-channel, hbas/hba-firmware/hba-drivers, array firmware, switch firmware, fabric design, device-mapper/multipathd config, lvm, and ext3 is just plain a lot of moving parts. Work on enough environments and you'll see this scenario caused by a grab bag of similar but not identical problems. The question at hand is, how to recover/remount without rebooting. – cagenut Mar 19 '10 at 17:35
0

File system corruption? Try:

dumpe2fs /dev/c/c | grep Filesystem\

If clean with errors, then you need to scan and clean.

Desperatuss0ccus
  • 252
  • 1
  • 4
  • 9
codycook
  • 1
-4

Linux simply doesn't cope well enough with medium-large scale SANs. You MUST give it some care and fine tune the IO timeouts and multipath timeout handling, they're all pretty much at desktop-ready defaults.

(Remember "rejecting IO to dead device"?)

darkfader
  • 73
  • 1
  • 1
  • 1
    You really need to backup statements like "Linux doesn't cope with SANs" and "desktop ready defaults" with references and hard facts. – Chris S Oct 17 '11 at 12:40
  • 1
    Default disk IO timeout of 30seconds? The above thread? The note from RedHat (outdated as it may) stating they cannot handle a "State change notification" gracefully, the way it would be intended. That Redhat by default put the multipath bindings in a location (/var/lib) that would not accessible at the load time of the multipath driver? That you can't recursively hot-disable a PCI hotplug hba and temporarly automagically take all dependant LUNs offline until it has been replaced. That it has no multithreaded HW init and takes "a while" to come up with >1k luns. Udev, being a shell script... – darkfader Oct 17 '11 at 15:18