System went read-only suddenly

My web server (Linux, Debian Wheezy) has suddenly gone read-only, MySQL have crashed (but Apache didn't) - giving errors when pressing tab to complete file name as:

bash: cannot create temp file for here-document: Read-only file system

Also, when I restart Apache it won't turn on anymore.

What I tried to do : I've replaced the hard disk , moved web files again ; error repeated after 1 week.

Logs shows nothing, one notice that I've seen many failed attempts to access the server through ssh.

df output:

Filesystem                                             1K-blocks    Used Available Use% Mounted on
rootfs                                                 249773956 2328024 234758164   1% /
udev                                                       10240       0     10240   0% /dev
tmpfs                                                     406336     200    406136   1% /run
/dev/disk/by-uuid/e45e30eb-efa4-4cd9-aaf9-c6cbe46aa41c 249773956 2328024 234758164   1% /
tmpfs                                                       5120       0      5120   0% /run/lock
tmpfs                                                    2489760       0   2489760   0% /run/shm
/dev/sdb1                                              249773956 2303784 234782404   1% /mnt/sdb1

mount -n output:

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=506431,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=406336k,mode=755)
/dev/disk/by-uuid/e45e30eb-efa4-4cd9-aaf9-c6cbe46aa41c on / type ext4 (ro,relatime,errors=remount-ro,user_xattr,barrier=1,data=ordered)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=2489760k)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
/dev/sdb1 on /mnt/sdb1 type ext4 (rw,relatime,user_xattr,barrier=1,data=ordered)

Ahmed Yusuf

Posted 2015-05-03T11:15:01.633

Reputation: 71

2these many ssh attempts that you report, are they by you (or some other expected person)? If not, is it right that a third party can get ssh access to this machine? Any evidence that they have succeeded in logging in?

If it isn't mindless (or worse mindful) vandalism by a third party, it seems there must be an error occurring on your / mount, then it remounts read-only. I would boot the system from CD/USB (e.g. systemrescuecd) and then do e2fsck -fp /dev/disk/by-uuid/e45e30eb-efa4-4cd9-aaf9-c6cbe46aa41c – gogoud – 2015-05-03T11:26:24.723

@gogoud No there are no successful attempts except for mine, and fsck won't be helpful since it's a new harddisk even – Ahmed Yusuf – 2015-05-03T15:25:02.570

System went read-only suddenly

Answers