Per-job environments in Jenkins with virtualenv

Question

I'm trying to use virtualenv to programmatically manage Python environments for each job on a Jenkins server, implemented via a Shared Library extension to activate environments on a per job basis. E.g.:

/vars/activateEnvironment.groovy:

def call(String env = "/usr/local/etc/environments/jenkins-$JOB_NAME") {

    sh """
    mkdir ${env}
    virtualenv ${env}
    source ${env}/bin/activate
    """
}

Pipeline script, in which the virtualenv-scripts repository contains the above file:

@Library('virtualenv-scripts') _

pipeline {
    agent any
    stages {
        stage("Test") {
            steps {
                activateEnvironment()
                sh 'which pip'
                sh 'echo \$PATH'
            }
        }
    }
}

Running this pipeline script, I get the following output:

[Pipeline] sh
[example-pipeline] Running shell script
+ echo /sbin:/usr/sbin:/bin:/usr/bin
/sbin:/usr/sbin:/bin:/usr/bin
[Pipeline] sh
[example-pipeline] Running shell script
+ which pip
/bin/pip

I've tried using this answer to make Jenkins use a login shell, but that still reloads the environment with every sh call.

I also saw this answer which would require pasting extra text every time a sh step is used in a Pipeline -- not ideal.

Is there a good way to have environment persist between sh commands? Alternately, is there a better way to achieve per-job environments with virtualenv? Thanks for all help/suggestions!

score 5 · Accepted Answer · answered Feb 19 '18 at 18:35

I had the same issue. After talking with some veteran Jenkins admins, this is the solution I arrived at:

def runCommandInMyEnvironment(cmd) {
  sh "setup_environment_command; source ./some/file; ${cmd}"
}

pipeline {
  agent any
  stages {
    stage("My Stage") {
      steps {
        runCommandInMyEnvironment('first_command')
        runCommandInMyEnvironment('second_command')
        // and so on
      }
    }
  }
}

It's not pretty and it can muddy up the console output quite a bit, but it's also the most reliable way to do this.

Another approach would be to parse the output of some command and chop it up into a bunch of environment variables and then pass those to a withEnv block, but this can be a very tricky and unreliable approach.

In any case, as you alluded to, Jenkins doesn't support persisting environments without withEnv, so ultimately there's no real good or clean way to do it.

There may be a better way to use virtualenvs with Jenkins, but I've never written a Jenkins job that runs tasks in a virtualenv, so I can't say. There is this plugin, but another stackoverflow answer suggests that the approach I gave in this answer is the preferred method for working with virtualenvs in Jenkins.

This seems to work relatively well with `virtualenv`. If setting up the environment were a heavier-weight task, I wouldn't want to call it a bunch of times, but in my case it is not. Thanks, @jayhendren — cwh, Feb 22 '18 at 17:02

Per-job environments in Jenkins with virtualenv

1 Answers1