I just had a problem on a client's server, it wasn't responding to anything so I forced a reboot. The client called shortly after to say the website reverted to a version from May. Checked the backups we do every night, and indeed the backup from last night are different from what's on the disk. The logs are empty on the server, and in the backups I can't really find anything useful (I do have logs from May for /var/log/kernel and /var/log/messages but not for /var/log/syslog where there is really too many writes, got rotated quicker).
Someone on IRC suggested that the kernel maybe had a problem and stopped syncing to the disk. The partitions are mdraid RAID 1 with ext4 partitions.
Do you see any way of recovering the data ? I restored the backup from last night but the data from this morning seems lost, including all the orders.
Do you see any way of detecting that ? Can a call to the command sync fail ? I'd like to add something in our monitoring to detect that next time. I'm a little afraid other servers might be in that situation, just waiting to reboot and lose everything, since our install are pretty standardized.
Thanks !
