Feed environment variables through bsub (Platform LSF)

1

I seem unable to pass environment variables to scripts submitted to a cluster scheduler through bsub. It works fine when I submits commands that use these environment variables directly, but these environment variables cannot be used in subsequent scripts. Why is that, and how can I fix it?

Here is a simple example where I tried to pass x=1 to a script:

me@cluster:~[407]$ bsub -I -tty -env 'x=1' 'echo .$x.'
Info: No jobname given, set to: Job_with_no_name
Info: No output file given, set to: output_%J_%I.txt
Info: No memory limit given, set to: 512MB
Info: No runtime limit given, set to: 15min
Job <35590315> is submitted to queue <normal>.
<<Waiting for dispatch ...>>
<<Starting on linuxbmc0243>>
.1.

me@cluster:~[408]$ echo 'echo .$x.' > myechox

me@cluster:~[409]$ more myechox
echo .$x.

me@cluster:~[410]$ chmod +x myechox

me@cluster:~[411]$ bsub -I -tty -env 'x=1' myechox
Info: No jobname given, set to: Job_with_no_name
Info: No output file given, set to: output_%J_%I.txt
Info: No memory limit given, set to: 512MB
Info: No runtime limit given, set to: 15min
Job <35590318> is submitted to queue <normal>.
<<Waiting for dispatch ...>>
<<Starting on linuxbmc1279>>
..

me@cluster:~[412]$ bsub -I -tty -env 'x=1' 'export x;myechox'
Info: No jobname given, set to: Job_with_no_name
Info: No output file given, set to: output_%J_%I.txt
Info: No memory limit given, set to: 512MB
Info: No runtime limit given, set to: 15min
Job <35590340> is submitted to queue <normal>.
<<Waiting for dispatch ...>>
<<Starting on linuxbmc1045>>
..

Update: This works:

me@cluster:~[456]$ bsub -I -tty -env 'x=1' 'export x=$x;myechox'
Info: No jobname given, set to: Job_with_no_name
Info: No output file given, set to: output_%J_%I.txt
Info: No memory limit given, set to: 512MB
Info: No runtime limit given, set to: 15min
Job <35609253> is submitted to queue <normal>.
<<Waiting for dispatch ...>>
<<Starting on linuxbmc0975>>
.1.

In the meantime, I have also found out that bsub prefers to be used with bsub < scriptfile rather than bsub scriptfile; this seems to be required to enable parsing of embedded options (using #BSUB), and it also explains why variables are only set locally (because no sub-script has to be called).

However, now I have also tried this, which, surprisingly, does not work:

me@cluster:~[457]$ bsub -I -tty -env 'x=1' < myechox
Info: No jobname given, set to: Job_with_no_name
Info: No output file given, set to: output_%J_%I.txt
Info: No memory limit given, set to: 512MB
Info: No runtime limit given, set to: 15min
Job <35609459> is submitted to queue <normal>.
<<Waiting for dispatch ...>>
<<Starting on linuxbmc1140>>
..

Update 2: I have tried exporting x first, which has the same effect as using -env 'x=1':

me@cluster:~[458]$ export x=1

me@cluster:~[460]$ bsub -I -tty 'echo .$x.'
Info: No jobname given, set to: Job_with_no_name
Info: No output file given, set to: output_%J_%I.txt
Info: No memory limit given, set to: 512MB
Info: No runtime limit given, set to: 15min
Job <35610151> is submitted to queue <normal>.
<<Waiting for dispatch ...>>
<<Starting on linuxbmc1140>>
.1.

me@cluster:~[459]$ bsub -I -tty < myechox
Info: No jobname given, set to: Job_with_no_name
Info: No output file given, set to: output_%J_%I.txt
Info: No memory limit given, set to: 512MB
Info: No runtime limit given, set to: 15min
Job <35609990> is submitted to queue <normal>.
<<Waiting for dispatch ...>>
<<Starting on linuxbmc1140>>
..

Update 3: This, again, works, which I consider an answer to my question:

me@cluster:~[571]$ (echo export x=$x && cat myechox) | bsub -I -tty
Info: No jobname given, set to: Job_with_no_name
Info: No output file given, set to: output_%J_%I.txt
Info: No memory limit given, set to: 512MB
Info: No runtime limit given, set to: 15min
Job <35641024> is submitted to queue <normal>.
<<Waiting for dispatch ...>>
<<Starting on linuxbmc1045>>
.1.

bers

Posted 2017-03-27T09:31:45.867

Reputation: 557

Have you try to set: export x=1 and then exec the script? – Romeo Ninov – 2017-03-27T10:16:36.483

@RomeoNinov: even better: I have tried export x=$x;myechox, and this works, too. I wonder if this is expected to be done like that... – bers – 2017-03-27T11:34:37.440

Bers, yes, if you export variable in shell it is populated to all subshell instances you start after the export (which is IMHO what you need :) ) – Romeo Ninov – 2017-03-27T11:36:45.857

@RomeoNinov Okay, I see. In the meantime, I have found out that bsub prefers to be used with bsub < scriptfile rather than bsub scriptfile; this seems to be required to enable parsing of embedded options (using #BSUB). So while I need to my question, the original question still stands unfortunately ;) – bers – 2017-03-27T12:37:29.407

No answers