0

UPDATE (7th Dec 2010) & WARNING: I copied the files discussed below (from the /var/tmp/vi.recover directory) to my Windows PC and sacnned with ESET NOD 32. It identifed two of the file (sshd: and juno) as the Linux/RST.B virus. Not sure if it is accurate in saying that but since the machine is a development server I'm going to re-build to be safe.

Hello,

I am using a Red Hat server and am getting the following errors every minute in the messages file:

Dec  1 16:50:01 ocalhost kernel: sshd:[4981]: segfault at 0000000000000079 rip 000000000807100f rsp 00000000ffb901ec error 6
Dec  1 16:51:01 ocalhost kernel: sshd:[4990]: segfault at 0000000000000079 rip 000000000807100f rsp 00000000ffce4b3c error 6
Dec  1 16:52:01 ocalhost kernel: sshd:[5000]: segfault at 0000000000000079 rip 000000000807100f rsp 00000000ffb7f9dc error 6
Dec  1 16:53:01 ocalhost kernel: sshd:[5010]: segfault at 0000000000000079 rip 000000000807100f rsp 00000000fff08d5c error 6
Dec  1 16:54:01 ocalhost kernel: sshd:[5022]: segfault at 0000000000000079 rip 000000000807100f rsp 00000000ffaf194c error 6

I've had a look in the SSH log file and it doesn't show any errors (not even those above). Can anyone tell me what an 'error 6' is or shed any light on why this may be happening. If anyone can suggest anywhere else I can look for more clues that would be great, I seem to have reached a dead end on this one.

Further to the comments below, here is the start function from the /etc/init.d/sshd file:

start()
{
        # Create keys if necessary
        if [ "x${AUTOCREATE_SERVER_KEYS}" != xNO ]; then
                do_rsa1_keygen
                do_rsa_keygen
                do_dsa_keygen
        fi

        cp -af /etc/localtime /var/empty/sshd/etc

        echo -n $"Starting $prog: "
        $SSHD $OPTIONS && success || failure
        RETVAL=$?
        [ "$RETVAL" = 0 ] && touch /var/lock/subsys/sshd
        echo
}

Thanks C

columbo
  • 219
  • 1
  • 12

1 Answers1

0

Can you tell me if hpsmh is installed and running?

rpm -q hpsmh; [ -f "/opt/hp/hpsmh/logs/httpd.pid" ] && echo "running" || echo "not running"

Also, if hpsmh is installed, what are the permissions on its install root?

Tok
  • 391
  • 1
  • 3
  • Thanks. hpsmh is not running. I've done a whereis and can't find hpsmh. – columbo Dec 01 '10 at 17:29
  • @columbo - can you please execute a `find / -type f -name 'libssl.so.*'? – Tok Dec 01 '10 at 17:36
  • Thanks for your reply....the result of the find is /lib/libssl.so.0.9.8b & /lib64/libssl.so.0.9.8b – columbo Dec 02 '10 at 09:02
  • Some more info... I installed atop and from atop's data I can see that the number of sshd processes running is increasing by 1 a minute. So, it would seem, each of these errors is causing a new sshd instance to start. This in turn is gradually sapping the RAM until it crashes the server. – columbo Dec 02 '10 at 09:52
  • Found this occuring every minute in the cron log /var/tmp/vi.recover/update >/dev/null 2>&1. Happens at 1 second past every minute the same as the SSHD errors... – columbo Dec 02 '10 at 11:06
  • @columbo - Well, based on this reference [http://www.eldos.com/documentation/sbb/documentation/ref_err_ssherrorcodes.html], an error 6 from SSH references an incorrectly composed packet. That this is happening every minute suggests to me a library problem, hence my questions regarding SSL. I would recommend stack tracing one or more of these processes to see if there might be a common library or routine where they are failing. `strace -vff -osshd-strace sshd` would watch one stack, alternatively you can add this to your init script and restart the service. – Tok Dec 02 '10 at 14:13
  • I tried strace -vff -osshd-strace sshd and it returned nothing but on closer inspection there is the process sshd and sshd:. it is sshd: that is causing me problesm (a new one spawning every minute).The sshd: file is located in the /var/tmp/vi.recover directory (which seems weird to me). when I try out your command on this one like so 'strace -vff -osshd-strace /var/tmp/vi.recover/sshd:' I get back: init: Couldn't open the file ./kswap.set. – columbo Dec 02 '10 at 14:42
  • @columbo - I apologize, I should have explained. The strace process will create a file named sshd-strace.pid for each process that is spawned, just sshd in this case. That file will contain the stack trace for the process(es) spawned. With reference to the sshd file, is that the executable sshd or the sshd_config file? – Tok Dec 02 '10 at 14:48
  • Thanks Tok, excuse my ignorance I'm a bit out of my depth on this one. I'm looking for the produced file now.It's the executable, I have two versions on my server. /usr/sbin/sshd (428k) and /var/tmp/vi.recover/sshd: (504k). It is sshd: that keeps getting spawned. When I look at atop I can see all the sshd: instances. – columbo Dec 02 '10 at 15:13
  • Lot's of jargon in that file including this:-------------- bind(7, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EADDRINUSE (Address already in use) ----------but that is just an excert. – columbo Dec 02 '10 at 15:27
  • @columbo - Were you ever trying to edit or view the /usr/sbin/sshd executable? Typically the /var/tmp/vi.recover directory is for files that were open in vi/vim when it, vi, exited uncleanly. What is the command in the init file for ssh that starts the daemon? – Tok Dec 02 '10 at 15:47
  • Thanks. Can't recall ever editing the sshd file. The vi.recover also contains kswap files, an executable called stealth and others none of which I recall editing. Not sure what you mean by the init file but I have traced the sshd: startup to /var/tmp/vi.recover/run which is activated by /var/tmp/vi.recover/update which is activated every minute according to my cron log, but I can't see where it is getting activated. If I could I would have a look at removing it. – columbo Dec 02 '10 at 16:03
  • @columbo - You should have a file, probably /etc/init.d/ssh, that is called to start up the service. If you view the file and look for the start logic, there will likely be multiple lines, it should help by telling us from where it is executing sshd. – Tok Dec 02 '10 at 16:07
  • Thanks again. I found the init file (where you suggested but it was called sshd) and it contains the line SSHD=/usr/sbin/sshd. – columbo Dec 02 '10 at 16:14
  • @columbo - Can you find the `case` statement in the file and past the `start` expression in your question above? – Tok Dec 02 '10 at 16:20
  • @columbo - I apologize for not replying earlier, I missed your update above. The init script appears correct, based on your earlier post that SSHD is set to /usr/sbin/sshd in the file. Have you tried stopping SSH entirely and deleting the /var/tmp/vi.recovery/sshd file? – Tok Dec 03 '10 at 15:36
  • No problem at all, thanks again for your help. Yes, this does indeed the SSHD errors listed above. But the cron job that fires it still goes off and I think causes a gradual memory drain. But someone has just pointed out to me where the cron job is that runs vi.recover/update its in /var/spool/cron. I'm going to stop it and see what happens. – columbo Dec 06 '10 at 11:54
  • @columbo - That is true that cron runs, typically, from files in /var/spool/cron, however, I would recommend instead of editing or removing these files manually that you leverage the `crontab` command. `grep -ilr "vi.recover" /var/spool/cron/*` will print the name of all files in /var/spool/cron which contain the string "vi.recover" which should correspond, roughly, to the username of the owner of the cron file. Becoming that user and executing `crontab -l` to view the cron and `crontab -e` to edit should allow you to comment out or remove the entry entirely. – Tok Dec 06 '10 at 13:12
  • I can see the owner is apache from the cron log. But when I do a sudo -u apache crontab -l it says "no crontab for user". – columbo Dec 06 '10 at 14:03
  • I'm being a fool, I'd renamed the apache file in /var/spool so of course it could not find it. Removed it. Thanks for your help. – columbo Dec 06 '10 at 14:10
  • @columbo - What did the grep command above return? Is there a file in /var/spool/cron/crontabs/ named apache? – Tok Dec 06 '10 at 14:12
  • Yes there is. Well it was in /var/spool/cron and it had the line in it that was starting the vi.recover script. – columbo Dec 06 '10 at 14:24
  • @columbo - I would recommend saving the file in case there are unintended consequences, but I think that you can safely remove/relocate the file out of /var/spool/cron. You may wish to restart the cron daemon after this as well. – Tok Dec 06 '10 at 14:40
  • Cheers Tok, I've saved a copy of the file so I'll see what happens. I appreciate all your help. If I had 15 reputation I'd click the voting up button. Ta. – columbo Dec 06 '10 at 16:58
  • UPDATE copied the files from the vi.recover over to my Windows PC and scanned them. Nod AV identified two of the files as the linux/RST.B virus! – columbo Dec 07 '10 at 09:30
  • @columbo - Maybe this should be the subject of a different post, considering the pages of comments already, but I would glad to walk you through some forensic and security steps to analyze what may have be, or remain, compromised and ways to protect yourself in the future. If you do spawn a new thread, please link me to the post if I may be of assistance. – Tok Dec 07 '10 at 13:39
  • Thanks Tok, we've reinstalled with CentOS and I'm learning a bit more about security now. – columbo Dec 14 '10 at 08:37