Linux shellscript combine all files without for loop

I have hundreds of files in one directory, is there a simple command or pipes of command I can use to append them together? I don't want to use any loops.

user775187

Posted 2011-06-12T21:08:47.727

Reputation:

whats wrong with for loop? you anyway will have a complexity of O(n) to get all the files in the folder. – zengr – 2011-06-12T21:10:43.273

@zengr: The difference is n−ceil(sum(length(name)+1 for name in names)/(ARG_MAX−4)) additional cat processes. – user1686 – 2011-06-13T13:58:34.473

...Except that any sane shell has cat implemented as a builtin which is run when someone asks to run "/bin/cat", so the new process overhead isn't relevant within a shell loop. – dannysauer – 2011-06-21T19:56:06.473

Answers

cat * >/path/to/somewhere

don't do

cat * > toall.txt

because "toall.txt" is created before cat is started and you will get strange result, "cat"ing toall.txt into toall.txt.

if want cat in the current directory, you should use

cat [some_globbing] > file #or
cat * > .dotted_file

.dotted_file is not expanded by * globbing.

or for example

(ls *.txt | xargs cat ) > /some/file

jm666

Posted 2011-06-12T21:08:47.727

Reputation: 807

If there aren't too many files:

cat * > /some/new/file

Otherwise:

find . -exec cat {} + > /some/new/file
find . -exec cat {} \; > /some/new/file

Ignacio Vazquez-Abrams

Posted 2011-06-12T21:08:47.727

Reputation: 100 516

It'd probably be worth noting that the first one is GNU find-specific, and works similarly to "find|xargs". :) – dannysauer – 2011-06-21T19:59:21.073

1BSD find understands -exec ... + as well. – Ignacio Vazquez-Abrams – 2011-06-21T20:07:00.043

Is the BSD you're using running a GNU find? :) "find --version" If not, then I learned something today. – dannysauer – 2011-06-21T20:11:04.283

FreeBSD OS X – Ignacio Vazquez-Abrams – 2011-06-21T20:12:48.183

I'm not shocked that OSX's command line is similar to that of FreeBSD. ;) But the man page looks suffieciently different. I'm glad to see some portability; I know it does not work on my HP-UX or AIX boxes using their POSIX-compliant find. :) – dannysauer – 2011-06-21T20:21:25.883

Similar to Ignatio's suggestion:

rm /the/output && find . -print0 | xargs -0 cat >> /the/output

Without using xargs, you're apt to blow your maximum command line length. Without using -print0, you'll potentially have problems with files that have weird chars (like spaces) in the names. But that's also GNU-specific.

dannysauer

Posted 2011-06-12T21:08:47.727

Reputation: 837

-1

This will append all files to outfile:

for file in !(outfile); do
        cat "$file" >> outfile
done

You need to delete outfile first if it exists. This will however not catch the dot-files. If you want to catch those as well you can use the two patterns

.??* !(outfile)

to create your file list. It requires a dot-file to have at least length 3, hence excludes . and ..!

barbaz

Posted 2011-06-12T21:08:47.727

Reputation: 2 696