Question marks showing in ls of directory. IO errors too

Question

Has anyone seen this before? I've got a raid 5 mounted on my server and for whatever reason it started showing this:

jason@box2:/mnt/raid1/cra$ ls -alh
ls: cannot access e6eacc985fea729b2d5bc74078632738: Input/output error
ls: cannot access 257ad35ee0b12a714530c30dccf9210f: Input/output error
total 0
drwxr-xr-x 5 root root 123 2009-08-19 16:33 .
drwxr-xr-x 3 root root  16 2009-08-14 17:15 ..
?????????? ? ?    ?      ?                ? 257ad35ee0b12a714530c30dccf9210f
drwxr-xr-x 3 root root  57 2009-08-19 16:58 9c89a78e93ae6738e01136db9153361b
?????????? ? ?    ?      ?                ? e6eacc985fea729b2d5bc74078632738

The md5 strings are actual directory names and not part of the error. The question marks are odd, and any directory with a question mark throws an io error when you attempt to use/delete/etc it.

I was unable to umount the drive due to "busy". Rebooting the server "fixed" it but it was throwing some raid errors on shutdown. I have configured two raid 5 arrays and both started doing this on random files. Both are using the following config:

mkfs.xfs -l size=128m -d agcount=32
mount -t xfs -o noatime,logbufs=8

Nothing too fancy, but part of an optimized config for this box. We're not partitioning the drives and that was suggested as a possible issue. Could this be the culprit?

I have seen these question marks when autofs had issues mounting a directory. Shutting down autofs allowed me to delete the directory and try mounting manually which pointed to a permissions error on the remote server. — Pace, Apr 25 '13 at 01:34

score 60 · Answer 1 · edited May 23 '13 at 20:07

60

I had a similar problem because my directory had read (r) but not execute (x) rights. My directory listing showed:

myname@srv:/home$ ls -l service/mail/
ls: cannot access service/mail/001_SERVICE INBOX: Permission denied
total 0
-????????? ? ? ? ?                ? 001_SERVICE INBOX
d????????? ? ? ? ?                ? 01_CURRENT SERVICE

The mail directory had the r bit set, but not the x that you need for listing or search and access. Doing sudo chmod -R g+x mail solved this problem.

edited May 23 '13 at 20:07

nickb

153
1
9

answered Dec 31 '10 at 13:35

user65174

701
5
2

5

'sudo chmod -R g+x mail' and then `chmod +x mail` solved my problem, thanks! – glebm Oct 10 '11 at 14:41
7

what if your `ls` returns `no such file or directory` with a bunch of `?????` in the privileges/rights area? – Kevin Meredith Apr 15 '14 at 16:39

score 13 · Answer 2 · answered Mar 30 '16 at 01:38

The answers mentioning the read, but not execute or stat() are correct. But there is a common cause of this (other than corruption) that's bitten me a few times and would match your question with the IO errors nicely. If you improperly mount a filesystem, the mount point of that filesystem may show up with question marks. If you're seeing these where you've just tried mounting a new filesystem, try the following before worrying about corruption and fsck.

$ sudo umount /mnt/raid1/cra/257ad35ee0b12a714530c30dccf9210f
$ ls -alh /mnt/raid1/cra

You should see the 257ad35ee0b12a714530c30dccf9210f folder with permissions and attributes rather than question marks. If so, search for other options for your mount command or /etc/fstab file. If not, perhaps it is time to read the other answers, backup what you can, and run a fsck.

See https://unix.stackexchange.com/a/675373/132913 for comfort that the question marks are not necessarily a symptom of filesystem corruption. — XavierStuvw, Oct 30 '21 at 07:47

score 12 · Answer 3 · answered Sep 15 '09 at 18:50

The question marks in the ls output just indicate that it could not stat() the directory entry. You can also see those if you ls a directory for which you have r(ead) but not x (search) permission. However in that case it would not report I/O error.

In your case it looks like there is a disk error or possibly filesystem corruption. /var/log/messages or dmesg is likely to reveal further details.

score 6 · Answer 4 · answered Sep 15 '09 at 17:42

6

Take a backup as soon as humanly possible, if only so that if you mess it up further while trying to repair any potential damage you can go back to the original less-broken state. After backing up, you might run fsck to see if it thinks there's any problems.

answered Sep 15 '09 at 17:42

Steven Schlansker

170
3

in my case this already is *the* backup. – Paolo Dec 12 '14 at 18:09

score 1 · Answer 5 · answered Nov 19 '10 at 20:11

We had a server with a corrupted filesystem (reiserfs) and it generated directory entries with question marks for all the attributes except the filename. In our case, the filenames were unaffected.

Also, the free space was being reported incorrectly. Using du -sh /* we could only account for about 30G, but the drive was being reported as over 200G in use.

Rebooting the server with shutdown -rF now to force a file-system check did not work. I had to reboot into single-user mode and run:

fsck.reiserfs --rebuild-tree /dev/sda3

This almost worked. It got through a few passes, then locked up. Had to reinstall the OS.

Maintain your backups!

Pace · Answer 6 · 2013-04-25T01:30:39.220

1

I've also seen this when running autofs but autofs can't mount the directory. So then to figure out why it couldn't mount the directory I disabled autofs and tried to mount the directory manually (this also allowed me to delete the directory). I tried mounting the directory manually and found that there was a permissions error. After fixing that, the directory returned to normal again.

edited Apr 25 '13 at 01:30

answered Apr 24 '13 at 19:56

Pace

235
2
11

score 1 · Answer 7 · answered Sep 16 '09 at 08:57

1

May be filenames just contain undisplayable characters. Try to check file names with emacs DirEd:

http://www.cs.utah.edu/dept/old/texinfo/emacs19/emacs_32.html

answered Sep 16 '09 at 08:57

Vladislav Rastrusny

2,581
12
39
56

score 1 · Answer 8 · answered Jan 11 '17 at 09:47

Watch out for other running processes on the server for example rsync

[root@server upload]# ls -la
ls: cannot access .3bfb3dc5-cb55-435f-8e23-2afcab2c6873_image4993891600240007749.jpg.bV6VTV: No such file or directory
total 194496
drwxr-x--- 2 gx apache    1382 Jan 11 10:36 .
drwxr-x--- 3 gx apache       3 Jan 11 10:29 ..
-rw-r--r-- 1 gx apache   94850 Dec 10  2015 37d355b9-210d-45df-8061-968ea5cb9f31_mob.jpg
...
-rw-r--r-- 1 gx apache   10864 Jul 24  2015 3bfb23bf-8ff5-4603-aa57-9b23ca498e2c_internet.png
-rw-r--r-- 1 gx apache   10864 Jul 24  2015 .3bfb23bf-8ff5-4603-aa57-9b23ca498e2c_internet.png.nHmIPk
-????????? ? ?  ?            ?            ? .3bfb3dc5-cb65-435f-8e23-2agcab2c6873_image4993891600240007749.jpg.bV6VTV

It generates temporary files that get created and dropped fast which will cause errors if you try to call other simple file management commands like rm, mv etc.

score 0 · Answer 9 · answered Nov 09 '15 at 14:42

Just to give a different perspective - I had this when I was programatically generating directories from a list directories in a file (in ruby).

Of course the line from the file came through as a string with a \n on the end - which looked good and seemed to work. However as I started creating directories rather than being chomped it ended up creating two of each directory: /whatiwanted and /whatiwanted?.

score 0 · Answer 10 · answered Sep 14 '16 at 18:51

0

I sometimes see this as a transient error when an NFS server is heavily overloaded.

The OP asked about RAID but several answers mention NFS, and in fact that was the search that brought me here.

answered Sep 14 '16 at 18:51

arp

101
2

Question marks showing in ls of directory. IO errors too

10 Answers10

Linked

Related