Jenkins Pipeline Syntax: Quoting Pains When Using Multiple Quote Blocks

Question

I have kind of a weird setup that I need to run a command as a docker container in a sh block in a Jenkinsfile.

Issue that I'm facing is specifically around the awk command used to trim the output.

Here is the command that WORKS FINE when ran directly on a bash shell:

OPFILENAME=$(docker run -t \
-e AWS_SECRET_ACCESS_KEY='<omitted>' \
-e AWS_ACCESS_KEY_ID='<omitted>' \
-e AWS_DEFAULT_REGION='us-east-1' \
mydockerimage:0.1 \
bash -c "aws s3 ls my-bucket-name/dir/ | sort | tail -n 1 | awk '{print \$4}' ") && \
echo $OPFILENAME

So I need to run this exact same thing on a remote host through a Jenkins pipeline, here is the general syntax:

pipeline {              
 agent any

 environment {
  BUILDHOST = 'buildhost.example.com' 
  SSHCMD = "ssh -o StrictHostKeyChecking=no jenkins@${env.BUILDHOST}"
 }


stages {
  stage('Get filename from s3') {
    steps { 
      sshagent ( ['ssh_config']) {
        sh """${SSHCMD} '''
            OPFILENAME=\$(sudo docker run -t \
            -e AWS_SECRET_ACCESS_KEY='<omitted>' \
            -e AWS_ACCESS_KEY_ID='<omitted>' \
            -e AWS_DEFAULT_REGION='us-east-1' \
            mydockerimage:0.1 \
            bash -c "aws s3 ls my-bucket-name/dir/ | sort | tail -n 1 | awk '{print \$4}') && \
            echo \$OPFILENAME
            '''
          """
       }
      }
     }
    }
}

Here is the error that Jenkins throws:

...bash -c "aws s3 ls my-bucket-name/dir/ | sort | tail -n 1 | awk {print' '}") && echo $OPFILENAME "
                        '
bash: -c: line 1: unexpected EOF while looking for matching `"'
bash: -c: line 3: syntax error: unexpected end of file

Notice how it transformed the awk command as such: awk {print' '}

Trying with various changes: ...| sort | tail -n 1 | awk \'{print \$4}\'") results in the exact same error.

Tried like this: awk "'{print \$4}'" and it doesn't throw an error but in the logs it shows like this: awk "{print' '}" thus the awk trimming desired doesn't take place. Grr!

So I know there is something wrong with the quoting, and since I am using triple-quotes for multi-line commands it's dirtying it up even more!

I have referenced this gist, trying to make sense of how to do this: https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4 but still running into issue after issue.

Caveats: - Command must run in an sshagent block and on a remote host. Commands cannot be ran in the local Jenkins workspace.

Answer 1

The error is because of a missing double quote between '{print \$4}' and ). You have it in your original script but not in the Jenkinsfile.

That said, this is a very hard problem since several processes will strip/process quotation. First, there is Groovy itself. Then there is Jenkin's sh command (plus the shell process the command will start), then ssh, then the remote shell which ssh will start, then the bash which you start. One of them is removing the \ before $4 which breaks your AWK script.

You can try to fix this specific problem by using awk -f script or by using sed to collapse multiple spaces into one and then using cut: ... | sed -e 's/ +/ /g' | cut -d " " -f 4 | ...

But in the end, it will always be brittle.

I strongly suggest to create a script file, copy that to the remote host and run ssh bash ./script.sh. That will remove so much headache. On top of that, you'll be able to test and debug the script from the command line without having to go through Jenkins after every change.

Alternatively, put most of the script into a custom docker image and select that as default command for the image.

Using the SSH Pipeline Steps plugin might help since it has an option "sshScript" which copies a script to the remote server and then executes it there: https://github.com/jenkinsci/ssh-steps-plugin#sshscript