Bash Command to Download Specific Line ONLY of Github Script

0

Searching for this answer has failed me, perhaps because I'm not asking for it in the right way.

I'm basically looking to use something (wget or curl for example) to retrieve the 2nd line of a text file hosted on github. The whole purpose of this is to determine if the script being used is the most up to date version and automatically update it if not.

Right now, this is how I achieve that...

sudo wget -q https://raw.githubusercontent.com/Wade821/PiDrawpile/master/drawpile-installer-official-build-from-sources
sudo chmod 777 "$dpinstallloc/Programs" -R
echo
dpcheckversion=$(grep -m 1 "drawpilescriptversion=" "$dpinstallloc/Programs/$scriptname" | cut -c 23-26)
currentscriptname="$(basename "$(test -L "$0" && readlink "$0" || echo "$0")")"

if [[ "$drawpilescriptversion" < "$dpcheckversion" ]]; then
    echo "Newest Script is using version $dpcheckversion."
    echo "Newer version of script available, the old script has been copied over"
    echo "the old script. Please re-run this script to use newest version."
    cd "$dpinstallloc"
    sudo rm -rf "$SCRIPTDIR/$currentscriptname"
    sudo cp "$dpinstallloc/Programs/$scriptname" "$SCRIPTDIR/$currentscriptname"
    sudo chmod 777 "$SCRIPTDIR/$currentscriptname"
    sudo rm -rf "$dpinstallloc/Programs/$scriptname"
    echo "Terminating script in 30 seconds."
    echo
    sleep 30s
    exit
elif [[ "$drawpilescriptversion" = "$dpcheckversion" ]]; then
    echo "Script is newest version available."
    sudo rm -rf "$dpinstallloc/Programs/$scriptname"
elif [[ "$drawpilescriptversion" > "$dpcheckversion" ]]; then
    echo "It appears the creator of the script has failed to update github with the newest version."
    echo "Please contact him to update the script there by emailing him at (redacted, minimizing number of posted locations for now)"
    echo
    echo "Script in use: $drawpilescriptversion"
    echo "Github Script: $dpcheckversion"
    echo
    echo "Pausing script for 30 seconds before continuing."
    sudo rm -rf "$dpinstallloc/Programs/$scriptname"
    sleep 30s
else
    echo "Unable to determine script version, proceeding with script."
fi

And while this isn't a particularly bad method, it's not the most efficient since the entire file needs to be downloaded every time. If I could somehow limit it to downloading the specified file line, that would make things faster overall. If no such method exists, I suppose I'll stick with this unless someone has a better idea on how to manage this.

Edit: I should also clarify the script in question is to compile and run a server of the program called Drawpile for use on a Raspberry Pi. Due to differences in versions of available packages, changes sometimes must be made to source code in order to compile successfully. Other times, as new features are added, new packages need to be installed. The idea of a self updating script is to prevent unnecessary reports of "this doesn't work" when in fact an update is available as the user reuses a script or updates the server. Thus, being that this is for end users, using the developer API authentication through github or the token is not a desirable result as that would create a security issue. As such, the solution should not require authentication.

Wade S

Posted 2019-04-30T07:12:46.000

Reputation: 46

1If it is a newer version you will need to download it regardless. In addition the script itself contains a check on whenever it's up to date or not. The easiest option otherwise would be just have a version file on the repo that just contains that singular information, assuming it's your repository. – Seth – 2019-04-30T07:23:00.717

Yeah, but often times the check would determine no update available as changes generally don't get made that often. As such, minimizing the download needed would be useful. I suppose a file placeholder with that version information is the only option, which didn't occur to me for some reason. Still, I was hoping for an actual command that could download a specified line of a text document. Oh well... – Wade S – 2019-04-30T07:26:38.763

IMHO sudo wget is a bad idea, do a plain wget, then sudo cp if you need to update. – xenoid – 2019-04-30T07:48:46.600

2Downloading a single line is not really possible. The HTTP protocol doesn't know anything about lines. The best you could do, would be to send a range request for a specific range of bytes. But that is very fragile and will break often. The if-modified-since header was designed for this exact purpose – darnir – 2019-04-30T11:47:18.910

Answers

3

Completely different solution, use an if-modified-since header in wget, using the date of your current local version:

The If-Modified-Since request HTTP header makes the request conditional: the server will send back the requested resource, with a 200 status, only if it has been last modified after the given date. If the request has not been modified since, the response will be a 304 without any body; the Last-Modified response header of a previous request will contain the date of last modification.

To get the file date in GMT (no local timezone taken in account), you can use date -u: date -u -d @$(stat -c %X targetfile). You can define the output format of date but date -R could be an simple answer.

The expected format is defined in this RFC.

This works for me:

lastMod=$(date -ud @$(stat -c %X $localFile) +'%a, %d %b %Y %X GMT')
wget --header "If-Modified-Since: $lastMod" http://server/path/to/the/file

My Apache is quite picky, it really wants the day of the week and the GMT indicator, but a simple variant seems to also work although not coherent with the RFC:

lastMod=$(date -R -ud @$(stat -c %X $localFile))

date -R formats date according to RFC5322. But this only works with TZ +0000 so you still have to use date -u.

PS: while looking for examples, I stumbled on a page expounding that supporting this was very important because GoogleBot uses it, and every single site needs to be very nice with GoogleBot :)

xenoid

Posted 2019-04-30T07:12:46.000

Reputation: 7 552

It should work, since you are supposed to use GMT time

– xenoid – 2019-04-30T08:39:06.250

1Edited answer to show how to come up with adequate header. – xenoid – 2019-04-30T12:44:57.160

I seem to have run into a bit of a snag. I tried your methodology, but it doesn't seem to work. It seems the "if-modified-since" header isn't available, which I checked using wget --timestamping https://raw.githubusercontent.com/Wade821/PiDrawpile/master/drawpile-installer-official-build-from-sources (error: Last-modified header missing -- time-stamps turned off.). I may just have to go with the suggestion from @Seth to simply use a file in GitHub that contains only the version number. – Wade S – 2019-05-02T17:59:45.143

@WadeS This says that if-modified-since is supported on GitHub, but maybe you should use the "official" API instead of doing "raw" GETs?

– xenoid – 2019-05-02T21:00:14.303

I see, but the official API seems to want authentication, which would mean my credentials. The project in question is more to ensure end users ALWAYS have the up to date version. I can't provide my credentials or the OAuth application client secret thing to end users for security reasons, because I presume in turn that would allow someone to potentially change my script (maliciously even). – Wade S – 2019-05-02T21:37:00.387