How can I easily convert HTML special entities from a standard input stream in Linux?

Question

CentOS

Is there an easy way to convert HTML special entities from a data stream? I'm passing data to a bash script and sometimes that data includes special entities. For example:

"test" & test $test ! test @ # $ % ^ & *

I'm not sure why some characters show up fine and other don't but unfortunately, I don't have control over the data coming in.

I'm thinking I might be able to use SED here but that seems like it would be cumbersome and possibly prone to false positives. Is there a Linux command I could pipe to that specializes in decoding this type of data?

score 14 · Answer 1 · answered Oct 22 '12 at 05:51

14

Perl is (as always) your friend. I think this will do it:

perl -n -mHTML::Entities -e ' ; print HTML::Entities::decode_entities($_) ;'

E.g.:

echo '"test" &amp; test $test ! test @ # $ % ^ &amp; *' |perl -n -mHTML::Entities -e ' ; print HTML::Entities::decode_entities($_) ;'

With output:

someguy@somehost ~]$ echo '"test" &amp; test $test ! test @ # $ % ^ &amp; *' |perl -n -mHTML::Entities -e ' ; print HTML::Entities::decode_entities($_) ;'
"test" & test $test ! test @ # $ % ^ & *

answered Oct 22 '12 at 05:51

Jason Tan

2,742
2
17
24

This works on my OSX10.8 laptop and a RHEL5.something host. – Jason Tan Oct 22 '12 at 05:52
1

In order to output the file in UTF-8, use binmode: echo "«" | perl -n -mHTML::Entities -mutf8 -e 'binmode(STDOUT, ":utf8"); print HTML::Entities::decode_entities($_) ;' – falstaff Nov 23 '13 at 22:46

Michael Hampton · Accepted Answer · 2012-10-22T13:18:10.890

10

PHP is well suited to this. This example requires PHP 5:

cat file.html | php -R 'echo html_entity_decode($argn);'

edited Oct 22 '12 at 13:18

answered Oct 22 '12 at 02:33

Michael Hampton

237,123
42
477
940

1

This command strips the new lines from my output, which is not wanted in some cases. – Pierre François May 28 '20 at 18:11

score 10 · Answer 3 · answered Apr 03 '17 at 09:57

10

recode seems available on default packages repositories of main GNU/Linux distributions. E.g. to decode HTML entities into UTF-8 :

…|recode html..utf8

answered Apr 03 '17 at 09:57

Skippy le Grand Gourou

1,145
14
23

score 5 · Answer 4 · edited Mar 27 '18 at 14:14

5

With Python 3:

python3 -c 'import html,sys; print(html.unescape(sys.stdin.read()), end="")' < file.html

edited Mar 27 '18 at 14:14

Andrew Schulman

8,561
21
31
47

answered Mar 27 '18 at 12:59

ariddell

151
1
2

score 0 · Answer 5 · answered Aug 01 '20 at 22:25

I use this script. Save it as html2utf.py, and use it ala echo $some_html | html2utf.py.

#!/usr/bin/env python3
"""
An alternative for `perl -Mopen=locale -MHTML::Entities -pe '$_ = decode_entities($_)'` (which you can use by `cpanm HTML::Entities`) and `recode html..`.
"""

import fileinput
import html

for line in fileinput.input():
    print(html.unescape(line.rstrip('\n')))

score 0 · Answer 6 · edited Jun 20 '14 at 07:35

0

Takes text file from stdin:

#!/bin/bash
#
while read lin; do
  newl=${lin//&gt;/>}
  newl=${newl//&lt;/<}
  newl=${newl//&amp;/<}
  # ...other entites
  echo "$newl"
done

It probably needs bash >= version 4

edited Jun 20 '14 at 07:35

Eric Leschinski

4,031
4
20
27

answered Oct 04 '13 at 18:42

bobom

1

How can I easily convert HTML special entities from a standard input stream in Linux?

6 Answers6

Linked