0
I am looking for a strategy suggestion. I am very new to Linux shell scripting. Just learning tcsh not more than a month. I need a script to automatically detects when is the result files are done copied back from a remote server to a folder in a remote machine, then start scp the files back to my workstation. I do not know in advance when the job will finish run, so the folder could have no result files for a long while. I also do not know when will the last result file done copied back from remote server to the folder (and thus can start the scp).
I had tried crontab. Work fine when I guess correctly, most of the time just disappointing. So I tried to write a script myself and I have it now. I intend to produce a script that serves me and my colleagues too. To use the script, the user first need to login to the remote machine manually. Then only execute the script at remote machine. The script first asks for user to input their local machine name and directory where they wish to save the result files. Then the script will just looping to test when is the total number of files change. When it detected that, which means the first result file is starting to be copied back from the remote server, then it loops again to detect when is the total files size in the folder stop changing, which means last result file is finished copied to the folder. After that it executes scp to send all the result files to the user workstation, at the initially specified directory.
Script works fine but I wish to make the script able to run in background and still running by itself even if the user logout from the remote machine and close the terminal. And also I wish to let the user just type in a simple command in terminal to start the script, something like a simple
./script.tcsh
I tried to run the script by command
./script.tcsh &
but fails, because background process unable to accept user input.
Google and found something called disown, but the command is not found. Apparently the remote machine and my machine does not support this command.
Tried to modify the script to first accept the user input, then attempt to use
cat > temp_script.tcsh << EOF
{rest of my script}
EOF
and then a line of
./temp_script.tcsh &
to try to create another script file and use the first script to initiate the second script in background. Also fail, because cat does not treat $variable as a literal text, it replaces it with values. I have a foreach i(1 2) loop, and the cat command just keep reporting error (missing value of variable i, which is just a counter in foreach loop syntax).
I am out of idea at the moment. Can anyone enlighten me with some strategy that I can try myself? The goal is to use only 1 script file, and prompt user for 2 inputs (machine name and directory to save), then no more interaction with user or waiting, and able to run even closing the terminal.
Note: I do not need password to login to remote machine and back.
Scout
Posted 2017-11-04T03:01:35.623
Reputation: 1
1
Is it okay to use rsync? If so, you could have a crontab job that runs it every n minutes. It will copy incomplete files, but eventually the end result will be clean. Here's a script that ensures only one rsync is running: https://superuser.com/questions/847850/behavior-of-rsync-with-file-thats-still-being-written?s=1%7C54.5125
– Louis – 2017-11-04T04:12:31.040Thanks for the reply. I can use rsync in the remote machine. However due to internet connection speed limitation, I am incline to compress all the requires file first (using tar command) before starting scp. From experience it makes a difference because a lot of the files are large ASCII files, so they are very compressible. – Scout – 2017-11-04T08:16:48.600
Another concern is, some of my colleagues are not exactly fluent in Linux. So I try to make it as little effort needed as possible. I think I can make a script to create a crontab that uses rsync, but if I understand the idea correctly, the user still needs to delete the crontab manually once it is finished, otherwise it will just keep running in the remote machine (albeit copy nothing). This step of needing the user to manually delete the crontab makes me less incline in this methodology.
I wish to have the user executes the script, then stop worrying about it forever. – Scout – 2017-11-04T08:17:02.517
Having said that, I like rsync a lot. I wish I discover this command earlier. – Scout – 2017-11-04T08:17:06.080
Yes, crontab will have to be stopped manually. Maybe you could have the script run intelligently, and not actually do anything after some point. But for crontab I believe it's as simple as the user entering
crontab -eand then putting a#(pound sign) before rsync job to comment it, or remove the#to reinsert in the scheulde. Enable compression in rsync is done with thezflag. It's probably slower, but uses less bandwidth. Also, in linux, the good/scary thing is if you can sudo, then you can sudo into your colleague's crontab. So you could always fix things for them if they got lost. – Louis – 2017-11-04T08:23:44.107I will need to test whether I can use the
sudocommand. First time I heard of it. On the other hand, I just discovered thatcatcan be made to print literally without replacing the variable, all I need is to add double quote. what I mean iscat > temp_script.tcsh << "EOF"If this works as I imagine I might solve my issue. I think my possible issue might be how to let the second script able to use user input variable value that is stored from first script. I will have to try it in office again. – Scout – 2017-11-04T08:36:52.667I suggest you don't prompt a user for input like this through cron. I think I'd find it to be really annoying and possibly dangerous, if I gave input that did something unexpected. I suggest doing something different, or only asking them once. – Louis – 2017-11-07T04:12:43.093
The
cat > temp_script.tcsh << "EOF"works as I imagined and I finally got my script working as intended. Thanks Louis for your feedback so far. You are the only one willing to entertain me. – Scout – 2017-11-08T02:30:58.580