Ec2 instance down with error: cannot create temp file for here-document

Question

Iam running my website built on ruby on rails in development mode on ec2 instance, when i ssh to that instance it says..

    Welcome to Ubuntu 12.10 (GNU/Linux 3.5.0-17-generic x86_64)

     * Documentation:  https://help.ubuntu.com/

      System information as of Thu Mar 28 05:10:40 UTC 2013

      System load:  0.48              Processes:           84
      Usage of /:   94.9% of 7.87GB   Users logged in:     0
      Memory usage: 49%               IP address for eth0: 10.130.18.205
      Swap usage:   0%

      => / is using 94.9% of 7.87GB

      Graph this data and manage this system at https://landscape.canonical.com/

    156 packages can be updated.
    86 updates are security updates.

    Get cloud support with Ubuntu Advantage Cloud Guest

Now, When i try to cd into one of the folders present at /home/ubuntu, it gives me error message

 set-bash: cannot create temp file for here-document: No space left on device

and the cd command fails, what could be the issue as the website is down and i'm not able to figure out the issue.What are the other o/p's i need to post for this issue.

EDIT $ sudo du -sch /tmp*

52K /tmp

52K total

$ df -hi

    Filesystem     Inodes IUsed IFree IUse% Mounted on
    udev              72K   389   72K    1% /dev
    tmpfs             74K   270   74K    1% /run
    none              74K     4   74K    1% /run/lock
    none              74K     1   74K    1% /run/shm
    none              74K     1   74K    1% /run/user
    /dev/xvda1       512K  222K  291K   44% /

$sudo df -h

     Filesystem      Size  Used Avail Use% Mounted on
    udev            287M  8.0K  287M   1% /dev
    tmpfs           118M  188K  118M   1% /run
    none            5.0M     0  5.0M   0% /run/lock
    none            295M     0  295M   0% /run/shm
    none            100M     0  100M   0% /run/user
    /dev/xvda1      7.9G  7.5G   13M 100% /

temp files are written in /tmp dir. Run a `du -sch /tmp*` and see the used space. Also `df -hi` to see the iNodes. The /tmp dir will be cleared after you reaboot the machine. See if that helps. I guess some script is writing a lot of output to the /tmp dir. — Valentin Bajrami, Mar 28 '13 at 06:23
i checked the /tmp dir, could not trace any unusual behaviour ..thanks — Bijendra, Mar 28 '13 at 06:58
You did not update your post by adding additional information that could help us to see what's happening there. `unusual behavior` is very relative. — Valentin Bajrami, Mar 28 '13 at 07:08
Please check the update and let me know if any o/p is required. — Bijendra, Mar 28 '13 at 07:19
That explains. You have 13M left over for the `/` root partition. You need to look for large files that can be deleted to free up space as suggested by @lain. — Valentin Bajrami, Mar 28 '13 at 07:29

user9517 · Answer 1 · 2013-03-28T07:26:50.190

Usage of /: 94.9% of 7.87GB

and from the updated question

/dev/xvda1 7.9G 7.5G 13M 100% /

5% of disk space is reserved by default for system use. Your / shows 94.9% full so effectively your filesystem is full. Take a look at the output of du -sh / and df -h, if they both show broadly the same usage then it is likely that your filesystem is being filled by a large file that you will need to track down and deal with (even more likely a log file). Use

du -h --max-depth=1 /

Check the output carefully for the high usage files/directories. Deal with files appropriately and directories by rinse and repeat (if /var shows high usage)

cd /var    
du -h --max-depth=1 .

and so on. Eventually you will find a file (or series of files) that you will want to delete. The safest way to go this is to shutdown the service that is using the file, delete it (after archiving it if you care), and then start the service again.

If the output of the df and du commands above differs considerably then it's likely that there is a deleted file that is being held open by a process. Use the command

lsof +L1

to track down the file and then restart the process that is holding it open.

If neither of these work then you may need to fsck the filesystem.

Ec2 instance down with error: cannot create temp file for here-document

1 Answers1