move all files in local directory to remote directory

Question

I want to move all files in local directory to remote directory.

And I am using this code to achieve it

 SOURCE_FILE=/var/www/oneserver/*
 TARGET_DIR=/var/www/anotherServer
 ARCHIEVEFILE=/var/www/archieveServer

/usr/bin/expect<<EOD
spawn /usr/bin/sftp -o Port=$PORT $USER@$HOST
expect "password:"
send "$PASSWORD\r"
expect "sftp>"
send "put $SOURCE_FILE $TARGET_DIR\r"
expect "sftp>"
send "bye\r"
EOD 

It works fine, but sometime it stops and only send some files.

I also want to move the already sent file to ARCHIEVEFILE by using mv command. But dont get idea how to move it. I can't use scp because the remote server not allowed using basic port, and the only way is using sftp.

Can anybody help, please?

[EDIT]

the $ARCHIEVEFILE is still the local server, just to backup/move so those files are not sent anymore with next cronjob

the TARGET_DIR is the remote server.

Answer 1

There's no command in OpenSSH sftp to move files to remote directory.

What you can do is to:

• use sftp put to upload the files (as you are doing already), and then

• use shell rm command to delete the files after sftp is done (i.e. after EOD):

  rm $SOURCE_FILE
  

  Or use ! to escape to shell from sftp script, after you sent put:

  send "!rm $SOURCE_FILE\r"
  expect "sftp>"        
  

---

Of course, this is not an atomic solution. If a file is added between put and rm, it will be lost. For an atomic solution, you have to iterate files in a local directory and upload and delete them one by one. Also for a robust solution, you need to check if an upload was successful.

Answer 2

The following approach may simplify things a lot:

1. Use sshfs to mount the remote share(s) as local path(s).
2. Work with cp and/or mv as if all operations were local.

Compare this answer of mine.

You still need some logic to detect if cp to the remote location succeeded, only then mv to the local archive; otherwise retry or something. But now all SFTP-related work should be transparently handled by sshfs.

Additionally shell globbing and quotes you use makes your code prone to errors related to spaces in filenames etc.

After you mount the remote share e.g. in /mnt/a b/remote, it will be so much easier to handle this.

(Note: it is a good practice to use lowercase variable names).

• To mount:

  sshfs -p $port $user@$host:"/path/on/the/remote/host/" "/mnt/a b/remote/"
  

  Use key based authentication or read Username and password in command line with sshfs. Read about security concerns.

• A code stub to copy files:

  # I deliberately use paths with spaces to show how to handle them
  for filename in "/source/location with spaces"/*; do
     cp "$filename" "/mnt/a b/remote/" &&
     mv "$filename" "/archive/location with spaces/"
  done
  

  This && ensures mv will run only if cp succeeds.

• To unmount:

  fusermount -u "/mnt/a b/remote/"
  

Answer 3

You could zip the source directory you wish to transfer

zip -r -9 /path/to/your/zip_file.zip var/www/oneserver/*   

(-r will recurse into subdirs , -9 will apply maximum compression for quicker transfer)

then scp the file to your remote server

scp /path/to/your/zip_file.zip user@host:/var/www/anotherServer

and then unzip the file on the remote server

ssh user@host 'unzip /var/www/anotherServer/zip_file.zip'

(there are many flags you can use with unzip you can use to overwrite or never overwrite exsiting files on your remote server, check man unzip for them , especially -n -o and -u )

hope it helps you