take a look here, these are sendmail based directive which can prevent flooding and bad behaviours, maybe this helps you:
http://www.acme.com/mail_filtering/sendmail_config.html
I for myself use these configs:
FEATURE(`greet_pause',2)
define(`confTO_ICONNECT', `15s')dnl
define(`confTO_CONNECT', `3m')dnl
define(`confTO_HELO', `2m')dnl
define(`confTO_MAIL', `1m')dnl
define(`confTO_RCPT', `1m')dnl
define(`confTO_DATAINIT', `1m')dnl
define(`confTO_DATABLOCK', `1m')dnl
define(`confTO_DATAFINAL', `1m')dnl
define(`confTO_RSET', `1m')dnl
define(`confTO_QUIT', `1m')dnl
define(`confTO_MISC', `1m')dnl
define(`confTO_COMMAND', `1m')dnl
define(`confTO_STARTTLS', `2m')dnl
define(`confTO_IDENT', `0s')dnl
define(`confTO_RESOLVER_RETRANS', `7s')dnl
define(`confTO_RESOLVER_RETRY', `4')dnl
define(`confMAX_RCPTS_PER_MESSAGE', `15')dnl
define(`confMAX_DAEMON_CHILDREN',`256')dnl
define(`confCONNECTION_RATE_THROTTLE',`8')dnl
define(`confBAD_RCPT_THROTTLE', `1')dnl Sendmail v8.12+
define(`confQUEUE_LA', `10')dnl
define(`confREFUSE_LA', `30')dnl
Further you can search for an Implementation called greypit. I'm not really up to date on that topic, but greypit should have ip base connection limits, maybe theres a solaris version out there.
Another way is as follow. Check your logs for massive dos activities or false logins and use the greetpause in access.
If you identity malicious behaviour insert a line as follows in your access and regenerate your access.db
GreetPause:bad.ip.dos.attacker.com 100
From now each request from the ip or hostname hast to wait 100 seconds before getting a helo.
I used this feature the other way around, but it can also be used for blocking unwanted connections.
The script which did these entrys was just a cron script, but care it is just the other way around getting good traffic and you have to manually recreate your access.db:
#!/bin/sh
declare -a a
let count=0
accessmap="/tmp/access.test"
logfiles="/var/log/mail.log"
mailfile="/tmp/tmpmail.mail"
email="myemail@test.com"
## hole alle IP Eintraege aus sendmail access und packe sie in ein array mit prefix und postfix
for x in $(echo $(grep -e "^GreetP" $accessmap | cut -f 2 -d ":" | cut -f 1 -d " ")); do
a[$count]=$(echo "^"$x"|");
((count++));
done
echo Number of elements: ${#a[@]} > $mailfile
#entferne whitespaces
#entferne | am ende der Zeile
b=$(echo ${a[@]} | sed "s/ //g"| sed "s/|$//")
#nun steht in der Variable den string den wir zum filtern wollen!
#echo $b
buffer=0
buffer_changed=0
datum=$(date +%Y.%m.%d__%H:%M:%S)
for x in $(grep authid $logfiles |grep "AUTH=server"|cut -f 3 -d "[" | cut -f 1 -d "-" | sort | uniq |egrep -v -e "$b" | sed "s/ (may be forged)//"|sed "s/]//"|sed "s/, authid=/#/"
if [ $buffer -eq 0 ]; then
buffer=1
echo >> $accessmap
echo "#Eintraege vom $datum" >> $accessmap
echo >> $accessmap
buffer_changed=1
fi
echo "GreetPause:$x"| sed "s/#/ \t\t0\t#/" >> $accessmap
done
if [ $buffer -eq 1 ]; then
echo "Command: zgrep with filter $b" >> $mailfile
echo >> $mailfile
echo >> $mailfile
echo "accessmap GreetingPause:">> $mailfile
cat $accessmap | grep -B 2 "GreetPause" >> $mailfile
echo >> $mailfile
mail -s "Acessmap changed" $email < $mailfile
fi