5

I have a script, written in bash, which runs from cron. One of the first things it does is SSH to a remote host and retrieve a list of files in a directory. Everything works fine when run from the command line, but not from cron.

The section of the script originally looked like this:

FILES=$($SSH_BINARY -i $SSH_KEY $SSH_USER@$REMOTE_HOST "ls $REMOTE_DIRECTORY")
echo "Got files = $FILES"

I added an echo statement above that line (shown below) to prove it wasn't a path or variable issue:

echo "$SSH_BINARY -i $SSH_KEY $SSH_USER@$REMOTE_HOST \"ls $REMOTE_DIRECTORY\""

If I take the resultant output line and run it as the same user cron will (root), it works without issue.

Thinking it may have something to do with assignment to a variable, I modified the FILES= line to read (thus, putting the output directly into my last_run_output file):

$SSH_BINARY -vv -i $SSH_KEY $SSH_USER@$REMOTE_HOST "ls $REMOTE_DIRECTORY"

The cron entry looks like this:

34 * * * * /root/path/to/my/script/get_logs.sh > /root/path/to/last_run_output 2>&1

So, PATH, variable assignment, and permissions issues aside, I started using debug flags in ssh. I ran once from the command line, then from cron, and compared the outputs. Here are some highlights from the diff:

The - side is the unsuccessful attempt, the + side is the successful attempt, run outside of cron.

@@ -87,9 +77,7 @@
 debug1: Remote: X11 forwarding disabled.
 debug1: Remote: Forced command: /home/sshacs/acssshsink netstorageuser
 debug1: Authentication succeeded (publickey).
-debug2: fd 4 setting O_NONBLOCK
 debug2: fd 5 setting O_NONBLOCK
-debug2: fd 6 setting O_NONBLOCK
 debug1: channel 0: new [client-session]
 debug2: channel 0: send open
 debug1: Entering interactive session.

I can't explain why these extra file descriptors are mentioned in debug2 when run from cron, but it appears to be related (notice read<=0 rfd 4 len 0 line below):

@@ -100,20 +88,672 @@
 debug2: callback done
 debug2: channel 0: open confirm rwindow 0 rmax 32768
 debug2: channel 0: rcvd adjust 131072
-debug2: channel 0: read<=0 rfd 4 len 0
-debug2: channel 0: read failed
-debug2: channel 0: close_read
-debug2: channel 0: input open -> drain
-debug2: channel 0: ibuf empty
-debug2: channel 0: send eof
-debug2: channel 0: input drain -> closed
+  [[ Very large output of the ls command ]]
+debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
 debug2: channel 0: rcvd eof
 debug2: channel 0: output open -> drain
 debug2: channel 0: obuf empty
 debug2: channel 0: close_write
 debug2: channel 0: output drain -> closed
-debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
 debug2: channel 0: rcvd close
+debug2: channel 0: close_read
+debug2: channel 0: input open -> closed
 debug2: channel 0: almost dead
 debug2: channel 0: gc: notify user
 debug2: channel 0: gc: user detached

Any ideas are greatly appreciated.

Kyle Smith
  • 9,563
  • 1
  • 30
  • 32
  • Is the remote directory the same when run from each? SSH command should be: $SSH_BINARY -vv -i $SSH_KEY $SSH_USER@$REMOTE_HOST ls "$REMOTE_DIRECTORY" – ewindisch Jan 19 '11 at 17:54
  • ewindisch, the command is exactly the same when run from each, as I use the output of the 'echo' line to determine what to run outside of the script. Also, what does '-w' do? My ssh binary doesn't appear to support it. – Kyle Smith Jan 19 '11 at 18:01
  • One suggestion: Can you set your shebang to `#!/bin/bash -ex` (assuming you use bash), this will echo all executed commands to stdout (`-x`) before running them and exit if an undefined $VAR is referenced in the script (`-e`). Useful for debugging this stuff. – SmallClanger Jan 19 '11 at 18:09
  • SmallClanger: Very good idea, unfortunately in this case I found nothing unexpected. Seems to do what I expect.. :( I will keep -x in mind in the future though. – Kyle Smith Jan 19 '11 at 18:15
  • Related: [SSH session never closes when executing “apt-get install”](http://serverfault.com/q/489939/130437) – kenorb Aug 07 '15 at 16:17

2 Answers2

8

Try adding -t -t to your SSH connection options. This will; force a pseudo-terminal to be allocated.

Multiple -t options force tty allocation, even if ssh has no local tty.

kenorb
  • 5,943
  • 1
  • 44
  • 53
meanasspenguin
  • 96
  • 1
  • THANK YOU! We've been looking at this for far too long, we had actually tried the -t option but hadn't read to the part where 'multiple -t's will force a psuedo-terminal even if a local psuedo-terminal is absent'. Thanks again! – Kyle Smith Jan 19 '11 at 19:50
1

You should debug your script further. Two things to try:

Rwrite it to do file output directly instead of through output redirection. That is, in the script do this:

$SSH_BINARY -i $SSH_KEY $SSH_USER@$REMOTE_HOST "ls $REMOTE_DIRECTORY" > savefile

Run your script with bash -x inside cron and save result. Change first line of script to #!/bin/bash -x and try something like

34 * * * * /root/path/to/my/script/get_logs.sh > script_debug 2>&1

that should help you narrow down exactly what your script is doing. I suspect redirection of output in cron is not working correctly in this case.

Also, dumb question: have you verified that root can write to /root/path/to/last_run_output? Have you verified that you don't have noclobber set somehow when the script runs in cron?

EDIT: further troubleshooting ideas based on comments from OP.

So none of the above ideas seem to be working. What about saving the fielist in a file on the remote machine and scp'ing it back? That would eliminate any weird quoting or input redirection problems.

Phil Hollenback
  • 14,647
  • 4
  • 34
  • 51
  • Phil, thanks for the ideas. I had actually already tried eliminated the output redirection in the cron entry (thinking it may be related to fd4, or something) and it made no difference. I did also try redirecting at that command instead of in cron, same problem. At SmallClanger's suggestion (see OP comments), I tried a -x as well. Nothing unexpected. Also I have write access to that file, it gets replaced with useful debug at every run. – Kyle Smith Jan 19 '11 at 18:19