Convert CRLF's to line feeds on Linux

Question

What's the best way to convert CRLF's to line feeds in files on Linux?

I've seen sed commands, but is there anything simpler?

Answer 1

Use this command:

fromdos yourtextfile

The other way around:

todos yourtextfile

These commands are found in the tofrodos package (on most recent distributions), which also provides the two wrappers unix2dos and dos2unix that mimic the old unix tools of the same name.

Answer 2

Use dos2unix.

dos2unix - DOS/MAC to UNIX text file format converter

dos2unix  [options] [-c convmode] [-o file ...] [-n infile outfile ...]

Options:
          [-hkqV] [--help] [--keepdate] [--quiet] [--version]

Answer 3

I prefer perl:

perl -lne 's/\r//g; print' winfile.txt > unixfile.txt

But that's well-suited to my uses, and it's very easy for me to remember. Not all systems have a dos2unix command, but most that I work on have a perl interpreter.

Another is recode, a powerful replacement for dos2unix and iconv; it's available in the "recode" package in Debian repositories:

recode ibmpc..lat1 winfile.txt   # dos2unix
recode lat1..ibmpc unixfile.txt  # unix2dos

For awk fans:

awk '{ sub("\r$", ""); print }' winfile.txt > unixfile.txt

...and sed:

sed 's/\r$//' winfile.txt > unixfile.txt

---

And now, only slightly-less-convoluted than deleting the CR's by hand in a hex editor, straight from one of our stackoverflow.com friends, useable with the beef interpreter (located on your friendly neighborhood Debian repository),

dos2unix in brainfuck!

,[[->+>+<<]>>>,[<-------------[+++++++++++++.>>>]<[>>----------[>+++++++++++++.-------------]<++++++++++>]<<<<[-]>>>[-<<<+>>>]]<[-]<[-]<]++++++++++.

big thanks to jk for wasting an hour of his life to write this!

Answer 4

I do this on Bash:

cat cr_stuffed.file | tr -d \r > no_more_crs.file

Answer 5

I think you can use tr, as well (though I have no funny format files on which to try):

tr -d '\r' < file1 > file2

Answer 6

I found a very easy way… Open file with nano: ## nano file.txt

press Ctrl+O to save, but before pressing Enter press: Alt+D to toggle betwen DOS and Unix/Linux line-endings, or: Alt+M to toggle betwen Mac and Unix/Linux line-endings then press Enter to save and Ctrl+X to quit.

Answer 7

In vi or Vim:

:%s/^V^M//g

Answer 8

I prefer Vim and :set fileformat=unix. While not the fastest, it does give me a preview. It is especially useful in the case of a file with mixed endings.

Answer 9

CR LF to LF using awk:

awk -v RS='\r?\n' 1
command | awk -v RS='\r?\n' 1
awk -v RS='\r?\n' 1 filename

Usage example:

echo -e 'foo\nbar\r\nbaz' | awk -v RS='\r?\n' 1 | hexdump -C

Explanation:

-v RS='\r?\n' sets variable RS (input record separator) to \r?\n, meaning input is read line by line separated by LF (\n) which may (?) be preceded by CR (\r).

1 is the script awk executes. A script consists of condition { action }. In this case, 1 is the condition which evaluates to true. The action is omitted, so the default action is executed, which means print the current line (which could also be written as {print $0} or simply {print}).

---

LF to CR LF: You can set the variable ORS (output record separator) to modify the line ends of the output. Example:

echo -e 'foo\nbar\r\nbaz' | awk -v RS='\r?\n' -v ORS='\r\n' 1 | hexdump -C

Answer 10

Use Perl's generic \R in a regex. That way, you can convert files with any of CR, CRLF or already LF or a mix of them (yes, there are files which mix 2 different newline conventions!).

perl -i.bak -pe 's/\R/\n/g' $yourfile

(-i.bak tells perl to convert the file in-place, saving the original as ${yourfile}.bak)

More info on \R in this answer

Answer 11

If you want a GUI method, try the Kate text editor (other advanced text editors may be able to handle this too). Open the find / Replace dialog (Ctrl+R), and replace \r\n with \n. (NB: you'll need to choose "Regular expression" from the drop down and deselect "Selection only" from the options.)

EDIT: Or, if you simply want to convert to Unix format, then use the menu option Tools > End of Line > Unix.

Answer 12

Paste this into dos2unix.py Python script.

#!/usr/bin/env python
"""\
convert dos linefeeds (crlf) to unix (lf)
usage: dos2unix.py <input> <output>
"""
import sys

if len(sys.argv[1:]) != 2:
  sys.exit(__doc__)

content = ''
outsize = 0
with open(sys.argv[1], 'rb') as infile:
  content = infile.read()
with open(sys.argv[2], 'wb') as output:
  for line in content.splitlines():
    outsize += len(line) + 1
    output.write(line + '\n')

print("Done. Saved %s bytes." % (len(content)-outsize))

Should work on any platform with Python installed. Public domain.

Answer 13

I used this script for files I needed to emergency transfer files from a windows system to a unix system.

 find . -type f | xargs file | grep CRLF | cut -d: -f1 | xargs dos2unix

find . -type f

Finds all the files, recursively in the directory you're running the command from

xargs file

Pass it to the file program to get an analysis of the file.

grep CRLF

We only want the output of file that shows CRLF.

cut -d: -f1

Get the output up to to the color. discard the rest. We should only have a filename now

xargs dos2unix

Pass the filename to the program dos2unix using xargs.