Apache2 status when server overloaded

Question

How can I check the status of apache2 when it is overloaded? That is, when it does not respond to the HTTP requests?

apache2ctl status is basically a wget on the status page. I need something that works in the command line without requesting that page.

Answer 1

I'm aware of two methods you may find useful:

1) mod_backdoor specifies a special thread and listening socket that allows you to hit /server-status when all the normal threads are tied up (should find it easily in a web search)

http://people.apache.org/~trawick/mod_backdoor.txt http://people.apache.org/~trawick/mod_backdoor.c

2) perl can parse the apache scoreboard if you let it use an on-disk scoreboard (ScoreBoardFile)

http://search.cpan.org/~opi/Apache2-ScoreBoardFile-0.01/lib/Apache2/ScoreBoardFile.pm

Answer 2

Interesting question, but I'm quite sure you can't. A workaround may be killing of some apache-threads and make sure you are the first one to connect to the newly spawned processes to show the apache status.

Edit: yes, let's play with bash

the following works by pulling the scoreboard of your Apache server-status page and letting you define when to react.

Use the _f_custom() function for that. I've built in two examples:

1. If there are more than 150 processes in Sending Reply mode, you will receive an email
2. If there are less then 20 open slots, also send an email.

`

#!/bin/bash
_sleep="5"

_f_getfullstatus() {
    curl $_uri 2> /dev/null
}

_f_mailto() {
    ( echo -e "To:$1\nFrom:$1\nSubject:${2}:\n\n${3} ${4}" ; _f_getfullstatus ) | sendmail -t
}

_f_custom() {
    case "$1" in
            _)
            ;;
            S)
            ;;
            W)
            if [[ $2 -gt 150 ]] ; then _f_mailto $_email "Houson, we've got a problem" "Currently there are $2 processes in "sending reply" mode" ; fi
            ;;
            K)
            ;;
            D)
            ;;
            L)
            ;;
            G)
            ;;
            I)
            ;;
            .)
            if [[ $2 -lt 20 ]] ; then _f_mailto $_email "Running out of open slots" "there are only $2 available atm" ; fi
            ;;
    esac
}

_f_count() {
    for _status in _ S R W K D C L G I \.
            do
            _counter=$(echo $_auto_output | sed "s/[^$_status]//g" | wc -m)
            if [[ $_verbose == "yes" ]] ; then
                    echo -n "Status of key ${_status}:" 
                    echo $_counter
            fi
            _f_custom $_status $_counter
    done
}

while [[ $# > 0 ]] ; do
    _opt="$1"
    case $_opt in
            -u)
            shift
            _uri="$1"
            ;;
            -m)
            shift
            _email="$1"
            ;;
            -s)
            shift
            _sleep="$1"
            ;;
            -D)
            _daemon="yes"
            ;;
            -v)
            _verbose="yes"
            ;;
    esac
    shift
done

_auto_output=$(curl ${_uri}?auto 2> /dev/null | tail -1 | sed 's/Scoreboard: //g')

if [[ $_daemon == "yes" ]] ; then
    while true ; do
            _f_count
            sleep $_sleep
    done
else
    _f_count
fi

This needs sendmail and curl in $PATH

as --help is not implemented:

-u: URI to server-status page -m: sets a emailaddress to send your errors to -v: let it run verbosly -D: let it run in w while true loop (daemonize) -s: interval in seconds between status requests

Answer 3

This is one of the cases where having proper monitoring system pays back. It would have gathered you all the stats over time and would show you the last known stats even when the service goes down. Not to mention the possibility to get alerted when your resource pools start getting empty so you can act before the service is disrupted.

When the milk has already been spilled on the floor, I guess what r_3 suggested is rather good solution.

Answer 4

install proper monitoring system, or play with bash:

#!/bin/bash
# WTF APACHE???
# cpu count
CPU=$(cat /proc/cpuinfo | grep "^processor" | wc -l)
# load average count
LOAD_5=$(cat /proc/loadavg | awk '{print $2}')
LOAD_AVERAGE_5=$(($(echo ${LOAD_5} | awk '{print 100 * $1}') / ${CPU}))
# Red Alert : 85% high load 5 min
# send some mail with status
if [ ${LOAD_AVERAGE_5} -ge 85 ] ; then
        httpd status > /tmp/apache_status.log
        mail -s "APACHE STATUS" you@mail.com < /tmp/apache_status.log
        service httpd stop
    else [ ${LOAD_AVERAGE_5} -le 50 ] ; then
        if ps aux | grep [h]ttpd; then echo 'OK'; else service httpd start; fi
fi