How to use sed to remove null bytes?

Question

What is the sed incantation to remove null bytes from a file? I'm trying:

s/\000//g

but that is stripping out strings of zeroes.

s/\x00//g

seems to have no effect. I'm trying to do this in a sed script, so I'm not sure the echo trick will work.

Answer 1

I don't know how you can exactly achieve this with sed, but this is a solution that works with tr:

tr < file-with-nulls -d '\000' > file-without-nulls

This is a solution for sed that works in some occasions but not all:

sed 's/\x0//g' file1 > file2

This is a solution that involves replacing into space characters which should work in all occasions:

sed 's/\x0/ /g' file1 > file2

Answer 2

tr tripped over some other bytes in my file and sed didn't replace anything. I ended up doing it not in sed but in Python:

f = open('file-with-0bytes.dump')
for l in f.readlines():
  print l.replace("\0", '')

Here's a pipeable one-liner:

python -c 'import sys; sys.stdout.write(sys.stdin.read().replace("\0", ""))'

I also noticed some commands actually leave the null bytes there but they're not visible anymore, at least not in an OSX terminal. I used hexdump to debug this.

Answer 3

It's quite easy to use Perl to perform a regex. Just replace sed with perl -np -e:

$ printf 'one\0two\0' | perl -np -e 's/\0/\n/g'
one
two

With the -n option, regexes are run line by line, just like sed.

If you want to use zero bytes as record separators, use Perl's -0 option.

$ printf 'one\0two\0' | perl -np0 -e 's/^/prefix /; s/\0/\n/g'
prefix one
prefix two
$ printf 'one\0two\0' | perl -np -e 's/^/prefix /; s/\0/\n/g'
prefix one
two

You can look up the command-line options of Perl by running perldoc perlrun.

Answer 4

To match a null byte, I use this regex with Cygwin's SED:

[^\x01-\x7F]