55

I have a problem in one of my shell scripts. Asked a few colleagues, but they all just shake their heads (after some scratching), so I've come here for an answer.

According to my understanding the following shell script should print "Count is 5" as the last line. Except it doesn't. It prints "Count is 0". If the "while read" is replaced with any other kind of loop, it works just fine. Here's the script:

echo "1">input.data
echo "2">>input.data
echo "3">>input.data
echo "4">>input.data
echo "5">>input.data

CNT=0 

cat input.data | while read ;
do
  let CNT++;
  echo "Counting to $CNT"
done 
echo "Count is $CNT"

Why does this happen and how can I prevent it? I've tried this in Debian Lenny and Squeeze, same result (i.e. bash 3.2.39 and bash 4.1.5. I fully admit to not being a shell script wizard, so any pointers would be appreciated.

wolfgangsz
  • 8,767
  • 3
  • 29
  • 34

6 Answers6

44

This is kind of a 'common' mistake. Pipes create SubShells, so the while read is running on a different shell than your script, that makes your CNT variable never changes (only the one inside the pipe subshell).

Group the last echo with the subshell while to fix it (there are many other way to fix it, this is one. Iain and Ignacio's answers have others.)

CNT=0

 cat input.data | ( while read 
do
  let CNT++;
  echo "Counting to $CNT"
done 
echo "Count is $CNT" )

Long explanation:

  1. You declare CNT on your script to be value 0;
  2. A SubShell is started on the | to while read;
  3. Your $CNT variable is exported to the SubShell with value 0;
  4. The SubShell counts and increase the CNT value to 5;
  5. SubShell ends, variables and values are destroyed (they don't get back to the calling process/script).
  6. You echo your original CNT value of 0.
coredump
  • 12,573
  • 2
  • 34
  • 53
  • 2
    First shell script I ever wrote gave me the same issues, banged my head against the wall for awhile before finding out that that pipes spawn additional shells. Any variable you mess with in a pipe will go out of scope as soon as the pipe ends--meaning that if you really, really want to do something with a variable outside of the pipe in which it was used, you'll have to hold state through something funky like a temporary file. – photoionized Apr 13 '11 at 17:17
  • Excellent answer, unfortunately I can only give one acceptance bonus. Sorry. – wolfgangsz Apr 13 '11 at 17:23
32

See argument @ Bash FAQ entry #24: "I set variables in a loop. Why do they suddenly disappear after the loop terminates? Or, why can't I pipe data to read?" (most recently archived here).

Summary: This is only supported from bash 4.2 and up. You need to use different ways like command substitutions instead of a pipe if you are using bash.

Florian Heigl
  • 1,440
  • 12
  • 19
Ignacio Vazquez-Abrams
  • 45,019
  • 5
  • 78
  • 84
  • You get the bonus, since your answer provided me with the widest range of options. – wolfgangsz Apr 13 '11 at 17:22
  • 7
    The link is dead. This is why link-only answers are bad. At least summarise the answer here. – rudolfbyker Apr 29 '17 at 19:38
  • god, yet another time where ksh is simply so much better... why, just why did everyone flock around bash. – Florian Heigl Dec 12 '17 at 00:16
  • @FlorianHeigl: Are you claiming that ksh is the One True Shell? – Ignacio Vazquez-Abrams Dec 12 '17 at 00:20
  • @IgnacioVazquez-Abrams no, but i claim that the while loop handling in bash is a horribly PITA. The loop handling was the long-runner keeping it from catching to 1993's functionality. The other things are getopt handling where the (also 1993) builtin handler was simple and capable, something you still can't get unless using i.e. docopt. I'm claiming that bash has put itself behind the curve for over 20 years, insistently and the amount of time spent on THIS THING HERE or millions of bad getopts uses is beyond measure - only accepted because most people will never know. – Florian Heigl Dec 12 '17 at 00:28
  • 1
    @IgnacioVazquez-Abrams if you look at the above FAQ it lists 7 workarounds, even for other shells where you don't need a workaround. You can just use them. And then, at last, it has workaround 8 which should be not workaround 8 but simply the reference solution. – Florian Heigl Dec 12 '17 at 00:31
  • What if the variable is being assigned a string? – Ken Ingram Jan 24 '20 at 03:25
  • 1
    I used ksh in environments with properietary *nix. Since ksh is proprietary, I had to use bash outside of commercial *nix environments. The proprietary nature of some tools forces another path. – Ken Ingram Apr 12 '20 at 02:37
13

This works

CNT=0 

while read ;
do
  let CNT++;
  echo "Counting to $CNT"
done <input.data
echo "Count is $CNT"
user9517
  • 114,104
  • 20
  • 206
  • 289
  • I like that, the smart way because you know where the needed data is and only need to get it back. If you dont know high skill solutions, you can always "read a file" hahahha. +1 for you. – m3nda Jan 21 '15 at 08:11
  • 1
    Anyone reading this, be aware that the solution provided by Iain only works when you have your script explicitly invoking bash, by having the first line: #!/bin/bash and that: #!/bin/sh will not work. – Roadowl May 09 '15 at 00:08
  • 1
    Interesting, first example I ever saw where a [Useless Use of Cat](http://unix.stackexchange.com/q/249804/135943) actually *prevented code from working*. By the way @Roadowl, the only bashism here is the line `let CNT++` which should instead be `CNT="$((CNT+1))"` to make use of [POSIX-compliant arithmetic expansion](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_04). The rest of it is portable already. – Wildcard Jan 12 '16 at 16:31
  • 1
    What about when the loop is producing a string? – Ken Ingram Jan 24 '20 at 03:22
8

Try passing the data in a sub-shell instead, like it's a file before the while loop. This is similar to lain's solution, but assumes you don't want some intermittent file:

total=0
while read var
do
  echo "variable: $var"
  ((total+=var))
done < <(echo 45) #output from a command, script, or function
echo "total: $total"
Steve
  • 81
  • 1
  • 2
4

Another solution is just simply add shopt -s lastpipe before the while loop.

I say, if the trouble comes cause the while is in the last segment of the pipeline, and in Bash all the commands in a pipeline executes in a subshell in a separated process isolated one from the other, then, using the lastpipe will execute the last command in the pipeline in the foreground.

For example:

CNT=0
shopt -s lastpipe
cat input.data | while read ;
...

And almost everything stay the same.

Cuauhtli
  • 141
  • 4
0

I found a way using the stderr file to store the value of var i.

# reading lines of content from 2 files concatenated
# inside loop: write value of var i to stderr (before iteration)
# outside: read var i from stderr, has last iterative value
f=/tmp/file1
g=/tmp/file2

i=1
cat $f $g | \
while read -r s;
do
  echo $s > /dev/null;  # some work
  echo $i > 2
  let i++
done;
read -r i < 2
echo $i
bugdot
  • 1