1
I seem unable to pass environment variables to scripts submitted to a cluster scheduler through bsub
. It works fine when I submits commands that use these environment variables directly, but these environment variables cannot be used in subsequent scripts. Why is that, and how can I fix it?
Here is a simple example where I tried to pass x=1
to a script:
me@cluster:~[407]$ bsub -I -tty -env 'x=1' 'echo .$x.'
Info: No jobname given, set to: Job_with_no_name
Info: No output file given, set to: output_%J_%I.txt
Info: No memory limit given, set to: 512MB
Info: No runtime limit given, set to: 15min
Job <35590315> is submitted to queue <normal>.
<<Waiting for dispatch ...>>
<<Starting on linuxbmc0243>>
.1.
me@cluster:~[408]$ echo 'echo .$x.' > myechox
me@cluster:~[409]$ more myechox
echo .$x.
me@cluster:~[410]$ chmod +x myechox
me@cluster:~[411]$ bsub -I -tty -env 'x=1' myechox
Info: No jobname given, set to: Job_with_no_name
Info: No output file given, set to: output_%J_%I.txt
Info: No memory limit given, set to: 512MB
Info: No runtime limit given, set to: 15min
Job <35590318> is submitted to queue <normal>.
<<Waiting for dispatch ...>>
<<Starting on linuxbmc1279>>
..
me@cluster:~[412]$ bsub -I -tty -env 'x=1' 'export x;myechox'
Info: No jobname given, set to: Job_with_no_name
Info: No output file given, set to: output_%J_%I.txt
Info: No memory limit given, set to: 512MB
Info: No runtime limit given, set to: 15min
Job <35590340> is submitted to queue <normal>.
<<Waiting for dispatch ...>>
<<Starting on linuxbmc1045>>
..
Update: This works:
me@cluster:~[456]$ bsub -I -tty -env 'x=1' 'export x=$x;myechox'
Info: No jobname given, set to: Job_with_no_name
Info: No output file given, set to: output_%J_%I.txt
Info: No memory limit given, set to: 512MB
Info: No runtime limit given, set to: 15min
Job <35609253> is submitted to queue <normal>.
<<Waiting for dispatch ...>>
<<Starting on linuxbmc0975>>
.1.
In the meantime, I have also found out that bsub
prefers to be used with bsub < scriptfile
rather than bsub scriptfile
; this seems to be required to enable parsing of embedded options (using #BSUB
), and it also explains why variables are only set locally (because no sub-script has to be called).
However, now I have also tried this, which, surprisingly, does not work:
me@cluster:~[457]$ bsub -I -tty -env 'x=1' < myechox
Info: No jobname given, set to: Job_with_no_name
Info: No output file given, set to: output_%J_%I.txt
Info: No memory limit given, set to: 512MB
Info: No runtime limit given, set to: 15min
Job <35609459> is submitted to queue <normal>.
<<Waiting for dispatch ...>>
<<Starting on linuxbmc1140>>
..
Update 2: I have tried export
ing x
first, which has the same effect as using -env 'x=1'
:
me@cluster:~[458]$ export x=1
me@cluster:~[460]$ bsub -I -tty 'echo .$x.'
Info: No jobname given, set to: Job_with_no_name
Info: No output file given, set to: output_%J_%I.txt
Info: No memory limit given, set to: 512MB
Info: No runtime limit given, set to: 15min
Job <35610151> is submitted to queue <normal>.
<<Waiting for dispatch ...>>
<<Starting on linuxbmc1140>>
.1.
me@cluster:~[459]$ bsub -I -tty < myechox
Info: No jobname given, set to: Job_with_no_name
Info: No output file given, set to: output_%J_%I.txt
Info: No memory limit given, set to: 512MB
Info: No runtime limit given, set to: 15min
Job <35609990> is submitted to queue <normal>.
<<Waiting for dispatch ...>>
<<Starting on linuxbmc1140>>
..
Update 3: This, again, works, which I consider an answer to my question:
me@cluster:~[571]$ (echo export x=$x && cat myechox) | bsub -I -tty
Info: No jobname given, set to: Job_with_no_name
Info: No output file given, set to: output_%J_%I.txt
Info: No memory limit given, set to: 512MB
Info: No runtime limit given, set to: 15min
Job <35641024> is submitted to queue <normal>.
<<Waiting for dispatch ...>>
<<Starting on linuxbmc1045>>
.1.
Have you try to set: export x=1 and then exec the script? – Romeo Ninov – 2017-03-27T10:16:36.483
@RomeoNinov: even better: I have tried
export x=$x;myechox
, and this works, too. I wonder if this is expected to be done like that... – bers – 2017-03-27T11:34:37.440Bers, yes, if you export variable in shell it is populated to all subshell instances you start after the export (which is IMHO what you need :) ) – Romeo Ninov – 2017-03-27T11:36:45.857
@RomeoNinov Okay, I see. In the meantime, I have found out that
bsub
prefers to be used withbsub < scriptfile
rather thanbsub scriptfile
; this seems to be required to enable parsing of embedded options (using#BSUB
). So while I need to my question, the original question still stands unfortunately ;) – bers – 2017-03-27T12:37:29.407