When doing an ls inside an Amazon EFS mount point, it just hangs.
The EFS troubleshooting section on AWS EFS troubleshooting
Mentions the following:
Mount Does Not Respond
An Amazon EFS mount appears unresponsive. For example, commands like ls hang.
Action to Take
This error can occur if another application is writing large amounts of data to the file system. Access to the files that are being written might be blocked until the operation is complete. In general, any commands or applications that attempt to access files that are being written to might appear to hang. For example, the ls command might hang when it gets to the file that is being written. This is because some Linux distributions alias the ls command so that it retrieves file attributes in addition to listing the directory contents.
To resolve this issue, verify that another application is writing files to the Amazon EFS mount, and that it is in the Uninterruptible sleep (D) state, as in the following example:
$ ps aux | grep large_io.py
root 33253 0.5 0.0 126652 5020 pts/3 D+ 18:22 0:00 python large_io.py /efs/large_file
After you've verified that this is the case, you can address the issue by waiting for the other write operation to complete, or by implementing a workaround. In the example of ls, you can use the /bin/ls command directly, instead of an alias, which will allow the command to proceed without hanging on the file being written. In general, if the application writing the data can force a data flush periodically, perhaps by using fsync(2), this might help improve the responsiveness of your file system for other applications. However, this improvement might be at the expense of performance when the application writes data.
So I verified to see if anything was writing to it but the only thing that showed up was
root 43556 0.0 0.0 124356 756 pts/6 D+ 19:15 0:00 ls --color=auto /efs/
root 43558 0.0 0.0 112664 972 pts/3 S+ 19:16 0:00 grep --color=auto efs
So nothing is being written to EFS as far as I know. Are there any other things I can look into as causes of this?
I also tried mounting the EFS on a separate machine just to verify, I also tested another machine in a different AZ to the other mount point in that AZ and saw the same behavior.
update:
lsof shows:
nfsv4.1-s 113422 root cwd DIR 202,1 4096 128 /
nfsv4.1-s 113422 root rtd DIR 202,1 4096 128 /
nfsv4.1-s 113422 txt cwd unknown /proc/113422/exe
This disappears when unmounted, and reappears after mounting.