1

I only really use the server a few hours a day a few days a week.

It is a backup server, it requests the backup data from the clients.

That part is taken care of, it wakes via a scheduled magic packet and does its thing. That is all good. I can wake it up to use it off schedule, that is also fine.

How do I just have it know that the network hasn't been used in a while and to put itself to sleep? The network traffic I would want to have record of are SSH, SFTP, rsync, and updates from Canonical. All other traffic is just chatter that I don't care about.

I'd like to maybe put the following pseudo code in as a cron script... that checks every 15 minutes or so. I am not worried about adding the cron functionality, I feel confident there.

if [ lastSignificantNetworkActivity > 3h ] { hibernate }

I may have an X->Y problem. I just want to put my server in a low power save to disk state for the usual 18 hours it would otherwise be doing nothing. I think network activity was a good metric to test against. I am open to more developed & robust solutions or inherent server properties to check against that exist.

(I am not sure if the daily power cycling would be worse than the constant wear and tear from ZFS running the data integrity checks all day long... just not sure.)

BradChesney79
  • 69
  • 5
  • 11
  • 2
    If it requests the backups from the client, and that is the only thing it does, then you could just add a shutdown after the backup script finishes running. – HubertNNN Feb 11 '20 at 22:51
  • Sort of, every once in a while I log in a few times to fiddle with old files to bring old work into new projects-- I'll copy an old nginx config with the SSL particulars already figured out, for instance. I want a delay on it. – BradChesney79 Feb 12 '20 at 02:12
  • I have had success with a cron script checking for who -u every 12 minutes for user activity. It is normal for my SSH sessions to last 12 minutes or more-- so there's that. /etc/crontab: 0,12,24,36,48 * * * * root bash /usr/local/sbin/hibernation – BradChesney79 Feb 17 '20 at 14:18
  • Please, could you inform which version of Ubuntu server is being used and, also, whether `rsync` traffic is flowing via direct connections (873/tcp port) or SSH tunnels? I have a suggestion, however I want to test it before writing an answer. – Anderson Medeiros Gomes Feb 24 '20 at 03:21
  • I am also assuming that manual hibernation of the server is already working. Am I right? – Anderson Medeiros Gomes Feb 24 '20 at 03:24
  • I think rsync is using SSH... "rsync user@remote.host.tld:/path/to/goodies /local/path/for/archival" hibernate is working fine as is the WoL that wakes up the server. – BradChesney79 Feb 24 '20 at 15:47

2 Answers2

1

A possible implementation to find out signs of network traffic generated by SSH sessions is via firewall rules:

  • Initially, rules matching traffic coming to and coming from port 22/tcp are loaded at boot time;
  • Then, a script periodically retrieves statistics information from firewall and stores hit counts of those rules into a temporary file;
  • The script will be able to detect network traffic by detecting changes in any hit count.

That approach would also detect traffic generated by SFTP and RSYNC, as both applications run over SSH protocol.

Detection of update activities would be hard to perform via firewall because package downloads use FTP, HTTP and HTTPS protocol and firewall rules would need to be tuned to differentiate software updates from recorded HTTP traffic. Because of that, my suggestion is to detect software updates by checking modification time of /var/lib/apt/lists, /var/cache/apt/archives and /var/lib/dpkg/lock.

My implementation proposal is as follows:


These command lines set up iptables and ip6tables rules.

# apt-get install iptables-persistent

# iptables -w -N fwstats
# iptables -w -A fwstats
# iptables -w -A INPUT -p tcp --dport 22 -m state --state ESTABLISHED -j fwstats
# iptables -w -A OUTPUT -p tcp --dport 22 -m state --state ESTABLISHED -j fwstats
# iptables-save > /etc/iptables/rules.v4

# ip6tables -w -N fwstats
# ip6tables -w -A fwstats
# ip6tables -w -A INPUT -p tcp --dport 22 -m state --state ESTABLISHED -j fwstats
# ip6tables -w -A OUTPUT -p tcp --dport 22 -m state --state ESTABLISHED -j fwstats
# ip6tables-save > /etc/iptables/rules.v6

This is the equivalent setup for nftables:

# apt-get install nftables
# nft add chain inet filter fwstats
# nft add rule inet filter fwstats counter
# nft add rule inet filter input tcp dport ssh ct state established jump fwstats
# nft add rule inet filter output tcp dport ssh ct state established jump fwstats
# echo -e \#\!`which nft` -f\\nflush ruleset > /etc/nftables.conf
# nft list ruleset >> /etc/nftables.conf

And this is a template of a monitoring script. The getStats function should be adjusted according to the firewall in use.

#!/bin/bash

getStats () {
    if using_nftables; then
        nft list chain inet filter fwstats | grep counter
    elif using_xtables; then
        for xtable in iptables ip6tables; do
            "${xtable}" -w -xnvL fwstats | egrep '^([[:space:]]+[0-9]+){2,2}'
        done
    fi
}

stateFile="/run/hibernation_state"

currentStats="`getStats`"
if [ "x${currentStats}" != "x" ]; then
    previousStats="`cat \"${stateFile}\"`"
    if [ "x${currentStats}" == "x${previousStats}" ]; then
        # No network traffic has been detected. Check files related do DPKG and APT
        clearToHibernate='true'
        now="`date '+%s'`"
        for path in "${stateFile}" \
            '/var/lib/apt/lists' \
            '/var/cache/apt/archives' \
            '/var/lib/dpkg/lock' ; do
            pathModTime="`stat -c '%Y' "${path}"`"
            # 10800 seconds = 3 hours
            if [ "$((now-10800))" -lt "${pathModTime}" ]; then
                clearToHibernate='false'
            fi
        done
        if "${clearToHibernate}"; then
            # OK to hibernate.
            systemctl hibernate
        fi
    else
        # Network traffic has been detected. Refresh stats.
        echo "${currentStats}" > "${stateFile}"
    fi
fi
0

Terrible idea time! Let's roll our own!

Expire neglected SSH sessions

cat > /etc/ssh/sshd_config <<EOF
AcceptEnv LANG LC_*
Banner none
ChallengeResponseAuthentication no
ClientAliveCountMax 0
ClientAliveInterval 1800
PasswordAuthentication yes
PrintMotd no
Subsystem sftp /usr/lib/openssh/sftp-server
UsePAM yes
X11Forwarding yes
EOF

Install hibernate

apt install hibernate

Set a system wide log format, for funsies

echo "HISTTIMEFORMAT=\"%Y-%m-%d %T \"" > /etc/environment

Start up a cron job!

echo "0,12,24,36,48  *    * * *   root    bash /usr/local/sbin/hibernation" >> /etc/crontab

And now, the thrown together awful:

cat <<-EOF > /usr/local/sbin/hibernation

#!/bin/bash -i

# The hibernation delay script

echo "----------------------" >> /tmp/justchecking.txt

EXTEND=0
SCRIPTTIME=$(date +%s)

echo $(date) >> /tmp/justchecking.txt

echo "script timetamp" $SCRIPTTIME >> /tmp/justchecking.txt

# Make sure there is a hibernation time file

if [ ! -f /var/run/hibernation.time.txt ]
    then
        echo $SCRIPTTIME > /var/run/hibernation.time.txt
        echo "created hibernation.time.txt sbin script" >> /tmp/justchecking.txt
fi

SLEEPTIME=$(</var/run/hibernation.time.txt)

if [ $SLEEPTIME -eq 999999999999 ]
    then
        echo "just came out of suspend, set extend to 360" >> /tmp/justchecking.txt
        EXTEND=360
fi

# Check the logs for recent activity

HISTTIMEFORMAT="%Y-%m-%d %T "
HISTFILE=/root/.bash_history
set -o history
history 75 > /tmp/recenthistory.txt
set +o history
unset HISTFILE
ece
echo $(tail -n5 /var/log/apt/history.log | head -n1) "apt" >> /tmp/recenthistory.txt
LASTACTIVITY=$(grep -e apt -e dpkg -e chebackup /tmp/recenthistory.txt | tail -n1 | awk '{print $2" "$3}')
echo "last activity" $LASTACTIVITY >> /tmp/justchecking.txt

#LASTACTIVITYDATE="$(</tmp/recenthistory.txt)"

#echo "last activity date"$LASTACTIVITYDATE"." >> /tmp/justchecking.txt

# Store last activity date time as a timestamp

ACTIVITYTIMESTAMP=$(date -d "$LASTACTIVITY" +%s)
echo "activity time stamp" $ACTIVITYTIMESTAMP >> /tmp/justchecking.txt

# Calculate 

ACTIVITYCOOLDOWNTIME=$(( 3960 - ( $SCRIPTTIME - $ACTIVITYTIMESTAMP ) ))

echo "activity cool down time" $ACTIVITYCOOLDOWNTIME >> /tmp/justchecking.txt

(($ACTIVITYCOOLDOWNTIME > 0)) && EXTEND=$ACTIVITYCOOLDOWNTIME
echo "extend variable after activity cool down command" $EXTEND >> /tmp/justchecking.txt

#if [ $ACTIVITYCOOLDOWNTIME > 0 ]
#    then
#        echo "activity was recent, adding cooldowntime to extend" >> /tmp/justchecking.txt
#        EXTEND = $ACTIVITYCOOLDOWNTIME
#fi

# Check for updates or backup currently running

if [ $(ps -aux | grep -e apt -e dpkg -e chebackup | wc -l) -gt 1 ]
    then
        EXTEND=3960
        SLEEPTIME=$(($SCRIPTTIME + $EXTEND))
        echo $(ps -aux | grep -e apt -e dpkg -e chebackup | wc -l) > /tmp/justchecking.txt
        echo "activity actively running, extended 3960" >> /tmp/justchecking.txt
fi

# Check for any users logged in

if [ $(who -u | wc -l) -gt 0 ]
    then
        EXTEND=10800
        #EXTEND=180
        echo $(who -u) >> /tmp/justchecking.txt
        echo "extended 10800" >> /tmp/justchecking.txt
        #echo "extend 180" >> /tmp/justchecking.txt
fi

echo "sleeptime" $SLEEPTIME >> /tmp/justchecking.txt

if [ $EXTEND -gt 0 ]
    then
        SLEEPTIME=$(($SCRIPTTIME + $EXTEND))
fi

if [ $SLEEPTIME -lt $SCRIPTTIME ]
    then
        echo "hibernate" >> /tmp/justchecking.txt
        echo $(date +%s) >> /tmp/justchecking.txt
        echo "hibernate" >> /tmp/justchecking.txt
        echo "999999999999" > /var/run/hibernation.time.txt
        hibernate &
        disown
    else
        echo $SLEEPTIME > /var/run/hibernation.time.txt
        echo "update sleeptime" $SLEEPTIME >> /tmp/justchecking.txt
fi

tail -n100 /tmp/justchecking.txt > /tmp/justchecking.txt
wc -l /tmp/justchecking.txt >> /tmp/justchicking.txt

exit 0

EOF

Don't forget to change the permissions

chmod 500 /usr/local/sbin/hibernation
BradChesney79
  • 69
  • 5
  • 11