62

On login to EC2 (Ubuntu) instance, I see 

*** /dev/xvda1 should be checked for errors ***

I can't fsck /dev/xvda1 because it is mounted, and sudo umount /dev/xvda1 fails because it is in use. lsof shows 

jbd2/xvda  172               root  cwd       DIR              202,1     4096          2 /
jbd2/xvda  172               root  rtd       DIR              202,1     4096          2 /
jbd2/xvda  172               root  txt   unknown                                        /proc/172/exe

and kill -SIGKILL 172 is ineffective.

What to do?

user1836587
  • 723
  • 1
  • 6
  • 4

4 Answers4

103

For our ec2 ubuntu instance, the above answers did not work completely.

On Ubuntu, by default, the check is not enabled in the rcS file. So

  1. Edit rcS file sudo vi /etc/default/rcS

below the line

#FSCKFIX=no

Add 

FSCKFIX=yes
  1. Edit fstab file. sudo vi /etc/fstab

Look for the the record for / and if the last digit is '0' change it to '1'. The last field is fsckorder and is used by fsck to decide the order of checking disks. If the value is '0' that disk is skipped.

For e.g.,

LABEL=cloudimg-rootfs   /        ext4   defaults,discard        0 0

to

LABEL=cloudimg-rootfs   /        ext4   defaults,discard        0 1

  1. Create /forcefsck file

    sudo touch /forcefsck

Reboot from ec2 console.

Revert the fsckorder value from '1' to '0' in /etc/fstab Delete the line FSCKFIX=yes to get back rcS file to original state.

Hope it helps.

Community
  • 1
krishnakumarp
  • 1,141
  • 2
  • 7
  • 6
  • 5
    This worked for me. Simply doing "touch /forcefsck" doesn't solve it, I had to enable FSCKFIX also. – Florin Andrei Aug 26 '15 at 23:10
  • 1
    Same here; just doing `sudo touch /forcefsck` before rebooting did not work on an Ubuntu 14.04 system, but this answer did. – Teemu Leisti Dec 22 '15 at 08:10
  • I had to do one other step - a modification to /etc/fstab as described here: https://nathanpfry.com/fix-dev-xvda1-should-be-checked-for-errors-on-amazon-ec2-ubuntu-instances/ That's odd because the last time I had this happen, I used your answer and it worked as is. – childofsoong Sep 19 '16 at 17:11
  • 2
    Thanks, this resolved it for me on an ec2 instance from AWS. I didn't have to reboot from the console, a simple `sudo reboot` with your changes resolved this. – Ralph Dec 07 '16 at 08:21
  • This also fixed the issue on my local computer. I believe this should be marked as best. – Arda Mar 16 '17 at 09:00
  • @childofsoong The article you linked to doesn't suggest anything different from this answer. Again, the steps above worked for me on an EC2 instance, however rebooting from the console didn't do it, I had to `sudo reboot`. – Ralph May 04 '17 at 14:29
  • @n3rve that was added after my comment (see the revision history). As for rebooting from ec2 console vs on instance, I can't comment - I probably just did it on instance myself, because it's faster when I'm already logged in. – childofsoong May 09 '17 at 20:24
  • FYI: This worked - but had to do it for both `/` and `/boot` - and you can do both at the same time (i.e. set the `/etc/fstab` for both `/` and `/boot` to `1`) - to save yourself a reboot. – colm.anseo May 05 '20 at 00:14
51

Most Linuxes these days should perform a forced fsck at boot time when the file /forcefsck is present on the system. If you are at liberty to reboot the VM, run

touch /forcefsck

Then reboot at your convenience

Sgaduuw
  • 1,823
  • 12
  • 16
7

Since you cannot boot into live disc, this is may sounds a bit weird, but i guess it'll do the job)

high level:

  • boot into new (another) instance.
  • attach storage from old instance to new (current).
  • run fsck.
  • detach storage from new (current) and reattach it to old instance.
alexus
  • 12,342
  • 27
  • 115
  • 173
1

I just do 3 steps below and issue resolved.

  1. Add "FSCKFIX=yes" to /etc/default/rcS
  2. sudo touch /forcefsck
  3. sudo reboot

Thanks all.

thach.nv92
  • 21
  • 3