How can I get the latest files from a Unix server to another Unix server?

How can I get the only the latest files from a Unix server to another Unix server?

I have connected via SFTP to server 2 and trying to apply a for loop to check the time stamp, which is not working:

sftp $SERVER2  << !EOF
cd $Server2_FILE_LOCATION
echo Pwd File location: $pwd

LastUpdatedTS=$(grep "value of TimeStamp is" /root/airtelSnD/BoTreeScript/BotreeLastFileTS.txt | cut -d'=' -f2)

echo 1

for file in mad_*.*
do
CurrentFileTS=$(stat -c %Y $file |awk '{print  strftime( "%Y%m%d%H%M%S", $1 )}')

echo 2

echo TS of last updated file is : $LastUpdatedTS
echo value CurrentFileTS is $CurrentFileTS

echo 3

if [[ $CurrentFileTS -gt $LastUpdatedTS ]]
 then
    echo if......
    mget  $file $DESTINATION_SERVER
    echo value of TimeStamp is=$CurrentFileTS > $LASTFILE_TS    
else
    echo else...       
 fi
done

quit
!EOF

Maddy

Posted 2012-12-13T13:58:07.153

Reputation: 11

bolD's answer to the following question might interest you: scp-without-replacing-existing-files-in-the-destination

– PetaspeedBeaver – 2016-04-23T10:47:41.303

I agree with the posted answers that you should use a tool that is designed to do this task. But if you want to know how to get scripts like this to work, try putting a backslash before all dollar sign symbols except for the ones that are using a value from the sftp environment. (It looks like $Server2_FILE_LOCATION and $DESTINATION_SERVER are the only ones to which that applies.) – Scott – 2012-12-13T15:57:23.317

Answers

Have you tried rsync?

rsync is an open source utility that provides fast incremental file transfer.

It will only transfer changed/new files, and only deltas for changed files. It also has various other nifty features.

This doesn't really address any issues with your code, but I think rsync will be easier than trying to reinvent the wheel with SFTP.

dsolimano

Posted 2012-12-13T13:58:07.153

Reputation: 2 778

Nope,as per my requirement I dont have to use any tool. I need to do it through scripts only. – Maddy – 2012-12-28T09:29:45.660

Rsync (or Unison) is best

Another solution would be to use find

 find . -name 'mad_*' -newer last_update -exec sftp ... {} \;
 touch last_update

RedGrittyBrick

Posted 2012-12-13T13:58:07.153

Reputation: 70 632