101
29
Does any one have a template shell script for doing something with ls
for a list of directory names and looping through each one and doing something?
I'm planning to do ls -1d */
to get the list of directory names.
101
29
Does any one have a template shell script for doing something with ls
for a list of directory names and looping through each one and doing something?
I'm planning to do ls -1d */
to get the list of directory names.
120
Edited not to use ls where a glob would do, as @shawn-j-goff and others suggested.
Just use a for..do..done
loop:
for f in *; do
echo "File -> $f"
done
You can replace the *
with *.txt
or any other glob that returns a list (of files, directories, or anything for that matter), a command that generates a list, e.g., $(cat filelist.txt)
, or actually replace it with a list.
Within the do
loop, you just refer to the loop variable with the dollar sign prefix (so $f
in the above example). You can echo
it or do anything else to it you want.
For example, to rename all the .xml
files in the current directory to .txt
:
for x in *.xml; do
t=$(echo $x | sed 's/\.xml$/.txt/');
mv $x $t && echo "moved $x -> $t"
done
Or even better, if you are using Bash you can use Bash parameter expansions rather than spawning a subshell:
for x in *.xml; do
t=${x%.xml}.txt
mv $x $t && echo "moved $x -> $t"
done
69
Using the output of ls
to get filenames is a bad idea. It can lead to malfunctioning and even dangerous scripts. This is because a filename can contain any character except /
and the null
character, and ls
does not use either of those characters as delimiters, so if a filename has a space or a newline, you will get unexpected results.
There are two very good ways of iterating over files. Here, I've used simply echo
to demonstrate doing something with the filename; you can use anything, though.
The first is to use the shell's native globbing features.
for dir in */; do
echo "$dir"
done
The shell expands */
into separate arguments that the for
loop reads; even if there is a space, newline, or any other strange character in the filename, for
will see each complete name as an atomic unit; it's not parsing the list in any way.
If you want to go recursively into subdirectories, then this won't do unless your shell has some extended globbing features (such as bash
's globstar
. If your shell doesn't have these features, or if you want to ensure that your script will work on a variety of systems, then the next option is to use find
.
find . -type d -exec echo '{}' \;
Here, the find
command will call echo
and pass it an argument of the filename. It does this once for each file it finds. As with the previous example, there is no parsing of a list of filenames; instead, a fileneame is sent completely as an argument.
The syntax of the -exec
argument looks a little funny. find
takes the first argument after -exec
and treats that as the program to run, and every subsequent argument, it takes as an argument to pass to that program. There are two special arguments that -exec
needs to see. The first one is {}
; this argument gets replaced with a filename that the previous parts of find
generates. The second one is ;
, which lets find
know this is the end of the list of arguments to pass to the program; find
needs this because you can continue with more arguments that are intended for find
and not intended for the exec'd program. The reason for the \
is that the shell also treats ;
specially - it represents the end of a command, so we need to escape it so that the shell gives it as an argument to find
rather than consuming it for itself; another way of getting the shell to not treat it specially is to put it in in quotes: ';'
works just as well as \;
for this purpose.
+1. I wish more people would use find
. There is magic that can be done, and even the perceived limitations of -exec
can be worked around. The -print0
option is also valuable for use with xargs
. – ghoti – 2014-10-17T22:24:30.460
The loop option won't show hidden directories. The find option won´t show symlinks. – jinawee – 2019-05-21T14:41:03.023
2+1 this is definitely the way to go when you need to generate a file list and use it in a command. find -exec is limited by only being able to run a single commands. With the loop, you can pipe to your heart's content. – MaQleod – 2012-05-10T22:45:21.687
17
For files with spaces in you will have to make sure to quote the variable like:
for i in $(ls); do echo "$i"; done;
or, you can change the input field separator (IFS) environment variable:
IFS=$'\n';for file in $(ls); do echo $i; done
Finally, depending on what you're doing, you may not even need the ls:
for i in *; do echo "$i"; done;
3Filenames can also contain newlines. Breaking on \n
is insufficient. It's never a good idea to recommend parsing the output of ls
. – ghoti – 2014-10-17T22:18:25.880
nice use of a subshell in the first example – Jeremy L – 2009-08-28T18:46:53.130
Why does IFS need a $ after the assignment and before the new character? – Andy Ibanez – 2012-11-29T18:57:50.623
5
If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this:
ls | parallel echo {} is in this dir
To rename all .txt to .xml:
ls *.txt | parallel mv {} {.}.xml
Watch the intro video for GNU Parallel to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ
4
Just to add to CoverosGene's answer, here is a way to list just the directory names:
for f in */; do
echo "Directory -> $f"
done
1
Why not set IFS to a newline, then capture the output of ls
in an array? Setting IFS to newline should resolve issues with funny characters in file names; using ls
can be nice because it has a built-in sort facility.
(In testing I was having trouble setting IFS to \n
but setting it to newline backspace works, as suggested elsewhere here):
E.g. (assuming desired ls
search pattern passed in $1
):
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
FILES=($(/bin/ls "$1"))
for AFILE in ${FILES[@]}
do
... do something with a file ...
done
IFS=$SAVEIFS
This is especially handy on OS X, e.g., to capture a list of files sorted by creation date (oldest to newest), the ls
command is ls -t -U -r
.
2Filenames can also contain newlines, and they often do when users are permitted to name their own files. Breaking on \n
is insufficient. The only valid solutions use a for loop with pathname expansion, or find
. – ghoti – 2014-10-17T22:21:44.713
The only reliable way to transfer a list of file names is to separate them with a NUL character, as this is the only one definitely not contained in a file path. – glglgl – 2015-01-02T09:23:50.590
-3
This is how I do it, but there are probably more efficient ways.
ls > filelist.txt
while read filename; do echo filename: "$filename"; done < filelist.txt
Your solution doesn't at all address the problem with file names containing newlines and other fancy stuff. – glglgl – 2015-01-02T09:24:50.713
6Stick to pipes in place of the file:
– Jeremy L – 2009-08-28T18:46:11.650
ls | while read i; do echo filename: $i; done
Cool. I should say that you can also use $EDITOR filelist.txt in between the two commands. Lots of stuff you can do in an editor that is easier than on the command line. Not relevant to this question, though. – TREE – 2009-08-31T18:45:55.930
1A better alternative if you really want to use
ls
isls -1 | while read line; do stuff; done
. At least that one won't break for whitespaces. – ejoubaud – 2014-08-01T08:47:38.9131with
mv -v
you don't needecho "moved $x -> $t"
– DmitrySandalov – 2016-01-15T12:59:47.063What if we want to use some of the options of ls? – Steven2163712 – 2018-12-15T08:14:16.297
23What if the filename has a space in it? – Daniel A. White – 2009-08-28T17:17:32.373
I'm iterating all folders. – Daniel A. White – 2009-08-28T17:18:43.357
5Unfortunately, as Daniel said, the code in this answer will break if any of the files or folders contains a space or newline in their name. It shows a very common misuse of
for
loops, and a typical pitfall of trying to parse the output ofls
. @DanielA.White, you might consider unaccepting this answer if it wasn't helpful (or is potentially misleading), since like you said, you're acting on directories. Shawn J. Goff's answer should provide a more robust and working solution to your issue. – slhck – 2013-01-23T17:48:34.6374
-∞ You shouldn't parse
– l0b0 – 2014-04-17T19:31:15.993ls
output, you shouldn't read output in afor
loop, you should use$()
instead of `` and Use More Quotes™. I'm sure @CoverosGene meant well, but this is just terrible.