Can't get a bash to work with python on a SGE

Question

I'm currently working on a SGE and I'm extremely unfamiliar with the linux environment. I have to execute python scripts but the setup is kinda confusing for me and I can't get it to work.

The setup is the following : The default python installed is the 2.4 and I need to use the 2.7 with some libraries.

I then linked all I needed with these lines :

export LD_LIBRARY_PATH=/home/volatile/xxx/local/lib:$LD_LIBRARY_PATH
export LD_RUN_PATH=/home/volatile/xxx/local/lib:$LD_RUN_PATH
export PATH=/home/volatile/xxx/local/bin:$PATH
export PYTHONPATH=/home/volatile/xxx/src/scikit-learn:$PYTHONPATH

Then if I type these lines and call python test.py it executes my code and it links everything great.

Then if I try to make a bash script (eligible for to send to the SGE) it won't work

': [Errno 2] No such file or directory

Here is the script

#!/bin/bash

#$ -N JOB_TKO
#$ -l h_vmem=1000M
#$ -l h_rt=864000
#$ -S /bin/bash
#$ -cwd

unset SGE_ROOT

export LD_LIBRARY_PATH=/home/volatile/xxx/local/lib:$LD_LIBRARY_PATH
export LD_RUN_PATH=/home/volatile/xxx/local/lib:$LD_RUN_PATH
export PATH=/home/volatile/xxx/local/bin:$PATH
export PYTHONPATH=/home/volatile/xxx/src/scikit-learn:$PYTHONPATH

python test.py

It won't even work if I removed the lines related to SGE and I do $ bash job.sh

#!/bin/bash

export LD_LIBRARY_PATH=/home/volatile/xxx/local/lib:$LD_LIBRARY_PATH
export LD_RUN_PATH=/home/volatile/xxx/local/lib:$LD_RUN_PATH
export PATH=/home/volatile/xxx/local/bin:$PATH
export PYTHONPATH=/home/volatile/xxx/src/scikit-learn:$PYTHONPATH

python test.py

If someone could make me understand why it doesn't work that would be really great, thanks!

Answer 1

Your bash script has DOS line endings, but bash expects Unix-style line endings (just a line-feed, not a carriage-return/line-feed pair. You'll need to remove them; dos2unix is a good tool to use, as it tr -d '\r'.

Specifically, it appears the error message comes from

python test.py

since bash takes the carriage return following the y in test.py as part of the file name. The "real" error message consists of the bytes

python: can't open file 'test.py\r': [Errno 2] No such file or directory

but the \r, when displayed at the terminal, causes the cursor to return to the beginning of the line, so that the rest of the error message starting at ': [Errno 2]... overwrites the preceding part, producing as you saw

': [Errno 2] No such file or directory