User account automatically filling up with dead.letter file

Question

I have one user account on a server with about 400 accounts that is filling up automatically. The dead.letter file in the users home directory automatically grows until the account is full (about 10 - 40 Mb per day). The user is using Microsoft Outlook to send and receive mail.

What can be causing this and how can I avoid it from happening?

Right now I have an emergency cron-job to delete the file but I would like "real" solution.

Edit: The server version is Red Hat Enterprise Linux ES release 4 (Nahant Update 4)

Edit 2: It seems mainly spam and I see different mailer headings (from php to Outlook Express) and a frequent appearing header is USER_NAME@vsap.no.loop

Update: I have asked the hosting provider where I use that dedicated server to look into the problem as well, as it's their Control Panel that could be a cause of the problem.

What's the RHEL box actually being used for in this scenario? Home directories via Samba? Mail server (IMAP? SMTP?)? — cjc, Feb 27 '12 at 23:26
@cjc It is being used as a web- / mail-server (smpt, pop, imap), users have no shell nor ftp access. — jeroen, Feb 28 '12 at 00:47
@Ladadadada Any hints? Today's version starts with what looks like spam, a lot of it seems spam, and a lot of the messages have a header like: `X-Loop: USERNAME@vsap.no.loop` — jeroen, Feb 28 '12 at 00:53
Looks like spam, have you been able to catch the process creating the dead.letter file with the `lsof` command or you haven't tried? — Yanick Girouard, Feb 28 '12 at 01:19
@Yanick Girouard No, I haven't been able to do get any results with that. Do you know of a way to keep `lsof` "open" like `tail -f`? — jeroen, Feb 28 '12 at 01:57
Let me write a script to do it and I'll post it as an answer. — Yanick Girouard, Feb 28 '12 at 02:13

score 3 · Answer 1 · answered Feb 28 '12 at 14:39

3

Does that user have a web content tree being served by a web server of this system?

Check their content tree for a CGI or something that handles GET/POST submissions. My guess is they have some standard web software installed -- a page layout tool, or something like WordPress. Some 3rd party/ies is using some security hole in that web software to try to send mail out from this system. Their exploit isn't working correctly, or at least not always, and so some/all of the outgoing mail is failing; the local mail transport agent is putting the mail in the user's dead.letter.

I'm out on limb here... but that's where I would look first.

answered Feb 28 '12 at 14:39

Craig Constantine

759
8
22

Thanks for your help. Unfortunately the user has mail-only privileges, no ftp, shell, or web-services apart from a system-wide control panel to change stuff like passwords and system-wide web-mail. This is the only account affected of about 400. – jeroen Feb 28 '12 at 15:09
2

ok. dead.letter holds outgoing attempted messages that fail. So your deviant dead.letter file is mail originating on this machine that isn't making it off of the box. Have you check the process table for anything running as this user? ...and does your cron-or-similiar service log anywhere? ...any logs about cron jobs for this user? – Craig Constantine Feb 28 '12 at 15:37
...and check that "system wide control panel" carefully (it's logs). That would a web-exposed front end to processes running on the system. It could have a vulnerability that leads to processes running as that user, which nefarious persons could try to manipulate to send mail. – Craig Constantine Feb 28 '12 at 15:38

Yanick Girouard · Answer 2 · 2012-02-29T00:38:34.027

Here's a script you could run against the dead.letter file and maybe catch the process creating it.

#! /bin/bash

if [ $# -eq 0 ]; then
        echo "Syntax: $(basename $0) <file_to_watch>"
        exit 1
fi

FILE_TO_WATCH=$1
LOGFILE=/var/tmp/$(basename $0).$(date +"%Y-%m-%d").log
SLEEP_DELAY=5

if [ ! -e $LOGFILE ]; then
        echo -e "DATE                   COMMAND     PID      USER   FD      TYPE     DEVICE SIZE/OFF       NODE NAME" > $LOGFILE
fi

echo "Starting lsof tail job with PID $$"
echo "lsof output will be appended to $LOGFILE"

while true; do
        if [ -e $FILE_TO_WATCH ]; then
                lsof $FILE_TO_WATCH | sed 1d | sed -e "s/^/$(date +"%Y-%d-%m %H:%M:%S    ")/" >> $LOGFILE 2>/dev/null
                sleep $SLEEP_DELAY
        fi
done

Feel free to change the variables to make the delay more aggressive for example. If you want to launch it as a background job, just call it like this:

nohup script.sh /path/to/dead.letter &

The script will echo the PID it uses for your convenience so you can kill it.

EDIT: As per your comment, it looks like the file is not held opened by a process long enough for you to be able to catch it. Another thing you could try is to set the immutable flag on the dead.letter file in hope that it will generate errors in /var/log/messages or another log. Immutables files can't be changed even by root.

Follow these steps:

# rm -f dead.letter
# touch dead.letter
# chattr +i dead.letter
# lsattr dead.letter
----i---------- dead.letter

Then try to touch it again with root. You'll get this:

touch: cannot touch `dead.letter': Permission denied

This confirms you did it right.

Wow, thanks, I'm going to try this and I'll get back to you. — jeroen, Feb 28 '12 at 13:34
My pleasure! Let me know how it turns out. Hope you find the rogue process causing this. — Yanick Girouard, Feb 28 '12 at 14:44
Thanks for the script, but I've left it running for a day without any results. I'll up the frequency tomorrow to see if that helps. — jeroen, Feb 28 '12 at 23:11
If the file is not "held" opened for at least a second, I doubt you will be able to catch it that way actually. You could try this though. You could set the immutable flag on the file, so that even root can't write to it. This will most likely generate errors in the logs which you'll be able to see. To do this, just run this command: `chattr +i dead.letter`, and then to confirm it's been set: `lsattr dead.letter`. You should see a `i` in the list of attributes. This means even root can't write to it! (try it) — Yanick Girouard, Feb 29 '12 at 00:34

score 1 · Answer 3 · answered Feb 27 '12 at 22:51

1

If I remember correctly, I had a case where this was caused by a broken cron script many years ago, so check the users crontab.

answered Feb 27 '12 at 22:51

Sven

97,248
13
177
225

Hmmmm, will do although the user has no shell access or anything... – jeroen Feb 27 '12 at 23:04
No crontab for the user in question. Thanks for the response though. – jeroen Feb 27 '12 at 23:07

User account automatically filling up with dead.letter file

3 Answers3