2

I have a CentOS 5.x Linux guest running on a XEN (Citrix) server, and from time to time it mysteriously goes into the read-only mode.

I've checked the file-system for errors, nothing suspicious appeared... :(

The system log says something like:

Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 309735103
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 309735191
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 309735279
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 309735359
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 309735447
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 309735535
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 309735103
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 309735103
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 307662855
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 315316647
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 315316655
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 315316663
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 315316671
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 315316735
Mar  5 10:57:16 testsrv last message repeated 38 times
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 307662855
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 262717023
Mar  5 10:57:16 testsrv last message repeated 2 times
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 258482255
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 141041743
Mar  5 10:57:16 testsrv kernel: EXT3-fs error (device xvdb1): ext3_find_entry: reading directory #17629185 offset 0
Mar  5 10:57:16 testsrv kernel: Aborting journal on device xvdb1.
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 16519
Mar  5 10:57:16 testsrv kernel: Buffer I/O error on device xvdb1, logical block 2057
Mar  5 10:57:16 testsrv kernel: lost page write due to I/O error on xvdb1
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 63
Mar  5 10:57:16 testsrv kernel: Buffer I/O error on device xvdb1, logical block 0
Mar  5 10:57:16 testsrv kernel: lost page write due to I/O error on xvdb1
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 59692223
Mar  5 10:57:16 testsrv last message repeated 33 times
Mar  5 10:57:16 testsrv kernel: ext3_abort called.
Mar  5 10:57:16 testsrv kernel: EXT3-fs error (device xvdb1): ext3_journal_start_sb: Detected aborted journal
Mar  5 10:57:16 testsrv kernel: Remounting filesystem read-only
DejanLekic
  • 304
  • 3
  • 16
  • Nothing suspicious? Looks like lots of I/O errors to me - and the system is remounting the disk R/O after detecting corruption in the journal. – symcbean Mar 05 '12 at 12:15
  • What I meant was - nothing suspicious in the output of disk utilities like tune2fs, etc... – DejanLekic Mar 05 '12 at 12:17

1 Answers1

5

@DejanLekic: tune2fs etc do only scan the file system for errors. The kernel error messages you observed are about hardware failures, which is one layer beneath the file system. Modern file systems and operating systems are taking hardware failures very seriously; ext* remounts itself as read-only to prevent any write access causing further damage, XFS shuts itself down completely and so on.

So you have some kind of hardware problem. Bad RAID controller, dead hard drive, unreliable SAN, something else.

Janne Pikkarainen
  • 31,454
  • 4
  • 56
  • 78
  • +1 for what you said, Janne. I am not a XEN expert - that is why I asked this question because someone may have seen this in the past, and are willing to share the experience, and explain what was done to solve the problem. My first guess is that we have iSCSII problems. Either that, or, as you said, RAID or SAN problems... – DejanLekic Mar 05 '12 at 12:23
  • Where is the file system for your XEN server located? If your problematic virtual host is served from iSCSI, well, then you have a problem with iSCSI. Then it could be the iSCSI box itself, network, or your XEN host settings. At one of my backup servers (running Debian 6) iSCSI related problems went completely away after I compiled a much newer kernel to it by myself, as Debian 6 only ships 2.6.32. Before the kernel upgrade the iSCSI box was sh*tting itself couple of times a week, after upgrading to kernel 3.0.x it has been running smoothly for months. – Janne Pikkarainen Mar 05 '12 at 12:27
  • All VMs are served from iSCSI, that is why I suspected the problem is in iSCSI at first place. Thanks for the info! – DejanLekic Mar 05 '12 at 12:46