scheduled bash script : how to transfer a file from HTTPS web server to FTP server

Question

I need to automate a task that downloads an XML file on a https:// web server (with authentification) and upload it on an FTP server (with another authentification).

So I think the best approach is to use CURL (or WGET) to download the file and lftp to upload !?

But I don't know how to script it in .sh file to program it sith cron (every thursday at 8 AM).

Have you got an example ?

EDIT

Example of file to download : https://www.domain.ltd/.../export.aspx?dt=07.08.2017 (dd.mm.YYYY format)

Example of filename in FTP server while uploaded : export_07.08.2017.xml

Answer 1

Your question composes actually of three questions: 1) How to download a file (with auth) 2) How to upload a file (with auth) 3) How to schedule a CRON job

My first question is how do you want to store the id/passwd

ad 1) You can use both curl and wget depending on what is better for your use case. I recommend reading excellent post by Daniel Stenberg curl vs Wget to understand the differences (a quick summary - curl is more developer friendly and is also a library a wget is a command).

I'm recommending using certificates without user/passwd as such would be visible to everyone having your id or group.

In my example, I'll use curl and certificate:
curl --cert certificate_file.pem https://example.com/example.xml

ad 2) To upload a file curl -T example.xml --cert certificate_file.pem ftps://ftp.server.com/remotedir/

ad 3) Cron format:

# Minute   Hour   Day of Month       Month          Day of Week        Command    
# (0-59)  (0-23)     (1-31)    (1-12 or Jan-Dec)  (0-6 or Sun-Sat)                
    0        2          12             *                *            /usr/bin/find

You only crontab -e to edit your crontab file. Other ways can lead to file corruption.

If you want to run your file every Thursday at 8AM do it the following way:
0 8 1-31 1-12 4 /path/your_script.sh or you can do it with *: 0 8 * * 4 /path/your_script.sh .

If you want to read more do it at: Cron and Crontab usage and examples

Now put it together:

#!/bin/bash

# $1 is your command line input (e.g. example.xml)
file_download=$1
file_upload=$2

actual_download="curl --cert certificate_file.pem https://example.com/$file_download"

eval $actual_download

if [ -e "$file_upload" ] then
  actual_upload="curl -T $file_upload --cert certificate_file.pem ftps://ftp.server.com/remotedir"
  eval $actual_upload
else
  echo "The $file_upload does not exist!"
fi

You would then execute the file: your_script.sh /path/example_download.xml /path/example_upload.xml

Answer 2

With help of @djsmiley2k on chat and @tukan here... here is my working approach :

#!/bin/bash

#source 
host_from="https://some_web_server/.../export"
file_download=export.aspx?dt=`date +%d.%m.%Y`
user_from="www_user"
pass_from="www_password"

#download file form web server
xml_file="curl -u '$user_from:$pass_from' $host_from/$file_download"

# destination
host_to="some_ftp_server/httpdocs/subfolder/.../Export"
file_upload=`date +%d.%m.%Y`.xml
user_to="ftp_user"
pass_to="ftp_password"

#upload file to FTP server
xml_upload="curl -T $file_upload -u '$user_to:$pass_to' $host_to/$file_upload"

Main issue was to escape user and password vars with single quotes.