2

I am currently in charge of a server running Red Hat and a bioinformatics webapp that deals with huge files, some of which are over 100GB when uncompressed. The act of decompressing these files is done by several different programs, all of which use the system temporary directory, /tmp. When a huge file is being decompressed, /tmp fills up and halts the operation in progress, causing downstream errors in the webapp. I have to go in and delete the problem file from /tmp.

The server's filesystem is setup like this:

/*output of df-h follows*/
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_root-LogVol01 83G   34G   45G  43% /
tmpfs                  95G     0   95G   0% /dev/shm
/dev/sda1             485M   81M  379M  18% /boot
/dev/sdb1             8.0T  6.2T  1.5T  82% /data

/*output of blkid follows*/
/dev/sda1: UUID="5f489589-0678-46c1-9f0f-e4e66c6a9e04" TYPE="ext4" 

/dev/sda2: UUID="k2rP0k-1YhK-72D9-fRrQ-BxUW-3gaC-6KF0nh" TYPE="LVM2_member" 

/dev/sdb1: UUID="4e54bee3-6450-446f-af80-70ca6268e12f" TYPE="ext4" 

/dev/mapper/vg_root-LogVol01: UUID="b6f228dc-fa6c-43c1-b88e-3b43a75e980b" TYPE="ext4" 

/dev/mapper/vg_root-LogVol00: UUID="80c6863d-c9f8-4abe-a353-a6a6818dc82d" TYPE="swap"

/*output of fstab*/
/dev/mapper/vg_root-LogVol01 /   ext4    defaults   1 1
UUID=5f489589-0678-46c1-9f0f-e4e66c6a9e04 /boot   ext4   defaults   1 2
/dev/mapper/vg_root-LogVol00 swap    swap    defaults    0 0
/dev/sdb1   /data   ext4    defaults    1 1
tmpfs        /dev/shm   tmpfs   defaults        0 0
devpts      /dev/pts    devpts  gid=5,mode=620  0 0
sysfs       /sys        sysfs   defaults        0 0
proc        /proc       proc    defaults        0 0

I know the basics of Linux, but I don't know much about filesystems. What I would like to do is enact a permanent solution to allocate more space to the temporary directory. I would be open to setting the TMPDIR environment variable to point to another filesystem with more space. Or, if it's possible, I'd be equally satisfied allocating more of the existing space to LogVol-01.

My question is how, given my current filesystem details, to permanently allocate more space to the temporary directory. As you can see by the df output, I have disk space to spare. I didn't set the server up, but I am now in charge of it and have root access, for better or worse!

DeeDee
  • 333
  • 2
  • 6
  • 16
  • 1
    The `df -v` is missing the right-most column. Can you show /etc/fstab? – cjc Mar 13 '12 at 22:11
  • Thanks for the answers, folks! They have been very helpful. I will post the output of fstab as soon as I get into work tomorrow. – DeeDee Mar 13 '12 at 22:38
  • To everyone having trouble with the mount points, the "Mounted on" field in the `df` output is being shown on the next line (probably copied from a terminal that was too small). The first line is the `/` mount point. The second line is `tmpfs` not `/tmpfs` (the `/` that should have been on the previous line also had the space after it removed) and is mounted at `/dev/shm`, etc. Unfortunately I cant edit as serverfault doesnt let you fix whitespace issues. – phemmer Mar 14 '12 at 00:03
  • @Matt @swasheck @cjc @Patrick OK, here are the results of /etc/fstab: `code` /dev/mapper/vg_root-LogVol01 / ext4 defaults 1 1 UUID=5f489589-0678-46c1-9f0f-e4e66c6a9e04 /boot ext4 defaults 1 2 /dev/mapper/vg_root-LogVol00 swap swap defaults 0 0 /dev/sdb1 /data ext4 defaults 1 1 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0 – DeeDee Mar 14 '12 at 13:32
  • Yes, @Patrick, good catch on the `df -v` output. I edited my post to fix that issue. Thanks for that! – DeeDee Mar 14 '12 at 13:48
  • Use `df -h` instead. – ewwhite Mar 14 '12 at 14:54
  • See my updated answer on how to fix it. – hookenz Mar 14 '12 at 17:36

2 Answers2

4

Update:

Since you've now confirmed that /tmp is not a symbolic link and that there are no mounts at /tmp. It's basically saying that / is filling up. How do you fix that? 2 options.

  1. I'd insert another hard disk and mount it at /tmp by inserting an appropriate entry in /etc/fstab.
  2. Or as a temporary measure move /tmp to another drive like /dev/sdb1 which currently has hundreds of GB free. It's simply a matter of creating a tmp directory on /dev/sdb1

i.e.

sudo mkdir /data/tmp
sudo chmod 1777 /data/tmp
sudo rm /tmp
sudo ln -s /data/tmp /tmp

If you'd prefer to add more storage (which is a better long term solution), there are lots of tutorials on the net on how to go about adding extra drives to Linux.

How to add an extra hard drive to Linux

Original answer...

tmpfs could be the culprit. It's ram based and it'll use swap when it needs to as well. That means it'll keep growing and growing until no more ram can be used for tmpfs. You're ending up with an incomplete decompress.

What you need to do then is edit /etc/fstab and comment out (with a leading '#') the line beginning with '/tmp' and reboot or unmount /tmp (with 'sudo umount /tmp')

Looks like you've got the whole drive using LVM so it's safe to not mount a replacement for /tmp.

Edit:

Wish I could see a default fstab file from redhat, or yours. Anyhow... It's possible that /tmp doesn't exist in fstab because it could be a symbolic link.
To find out run "ls -ld /tmp". If it's a symbolic link you'll see it listed with /tmp -> /dev/shm or something.

If it is actually a link to /dev/shm or to a directory that is mounted as tmpfs, Fix that with:

sudo rm /tmp
sudo mkdir /tmp
sudo chmod 1777 /tmp

EDIT2: On second thought, it could other reasons.

  1. Either /tmp is filling up, or
  2. The decompression tools aren't working too well on your large archives. 100GB uncompressed data is probably a large archive! What tools are you using to extract the archive. What's it's format? I know some versions of tar have issues, or
  3. The filesystem type does not allow for some files to to be extracted because they exceed the filesystem limit. What type is the target filesystem? is it ext3 or 4 or something else?
  4. Or the filesystem is really just full.
hookenz
  • 14,132
  • 22
  • 86
  • 142
  • I think tmpfs is mounted as /dev/shm and not as /tmp. It's hard to tell unless the OP shows us /etc/fstab. – cjc Mar 13 '12 at 22:26
  • 1
    `/dev/shm` is still shared memory though. – swasheck Mar 13 '12 at 22:27
  • @swasheck whats that got to do with anything? He's not extracting to `/dev/shm` – phemmer Mar 13 '12 at 23:51
  • and tmpfs is just the filesystem type. it's possible /tmp is a symbolic link to /dev/shm which is mounted as tmpfs though. – hookenz Mar 13 '12 at 23:54
  • @Patrick if `/tmp` is mounted on/linked to `/dev/shm` then it is extremely relevant. Sorry that my comment was so vague. End of a busy day. – swasheck Mar 14 '12 at 14:25
  • @Matt The webapp uses gzip to decompress files that are compressed at ~30GB. The logs from the webapp give a "disk full" error. When I check the output df -v at this point, I see that the usage of /dev/mapper/vg_root-LogVol01 is at 100%, and in the /tmp folder there is a huge file. If I remove that file, then the usage on LogVol1 goes back down to the 40% range, like what you see in my OP. I verified that /tmp is not a symbolic link by following your instructions. Thanks! – DeeDee Mar 14 '12 at 16:47
  • @Matt That answer is exactly what I needed. Thanks a bunch for your guidance! – DeeDee Mar 14 '12 at 18:33
2

I'm pretty sure that /tmp and tmpfs are RAM-based, so decompressing them and storing them in RAM may be the reason that things grind to a halt. The OS starts swapping like crazy so you begin to really notice the performance hit.

Must this operation occur in /tmp?

Adding an actual answer:

http://wiki.centos.org/TipsAndTricks/ExpandLV

swasheck
  • 713
  • 9
  • 20
  • I think tmpfs is mounted in /dev/shm, actually. It's hard to tell from the OP's copy and paste of the `df` output. It's not clear how /dev/sdb is mounted, also. – cjc Mar 13 '12 at 22:13
  • 1
    you're absolutely correct – swasheck Mar 13 '12 at 22:15
  • And I believe that tmpfs can also use swap under low memory. – hookenz Mar 13 '12 at 22:17
  • But swapping would still cause a performance hit, right? – swasheck Mar 13 '12 at 22:20
  • Yes, it would. Though, I think the OP is saying that the compute processes are actually crashing due to a lack of space rather than just performing slowly. – cjc Mar 13 '12 at 22:22
  • True. But that's some serious swapping going on if at least ONE of the files in the process is 100GB when uncompressed. "Crashing" is probably "timeout." Full `fstab` would be extremely helpful. – swasheck Mar 13 '12 at 22:42