search PDFs with non-standard character encodings

Question

Some PDF files produce garbage ("mojibake") when you copy text (even though they render OK). This makes it impossible to search them (whatever you search for will not match the garbage).

Does anyone have an easy workaround?

Examples:

1. TEAC TV manual EU2816STF (yields above problems in Adobe Reader on both Windows and a Mac, but works fine in Preview on a Mac)
2. Leadtek Winfast PVR2 manual (FTP link; also has problems in Preview on a Mac)
3. Swann TV tuner card manual (FTP link; also has problems in Preview on a Mac)
4. Phonedisc license agreement (from the now-defunct DTMS)
5. Macquarie IFP quarterly fund review
6. BAN-TACS Small Business Booklet (archived version)
7. Easterfest 2004 flyer (also from the archive)

I am using Adobe Reader (latest version) for Windows - perhaps an alternative viewer might help? I'm looking for a free solution for Windows. Open-source would be even better.

Edit: The docs for the Multivalent Extract Text tool have a good summary of why things can go wrong, including: (quoted document last modified Jan 2006)

• Text may not have a Unicode mapping. PDF Type 3 fonts often do not, and TeX DVI has characters that do not have Unicode equivalents.
• The Unicode encoding may be buggy. Open Office maps some characters into the same Unicode, resulting in apparant letter dropping and doubling.

I guess the ultimate solution in these cases would be to OCR each glyph in a font to figure out what character it really is. Note that this would be easier than OCRing a noisy scanned document because the exact shape of the glyph is available (at infinite resolution since it's a "vector" image).

Answer 1

Foxit Reader, perhaps?

For what it's worth, I just checked the PDF you linked to with Safari 4.0.4 on Mac OS X 10.6.2 and while there is some Engrish, the PDF it renders flawlessly without any onscreen "garbage". Perhaps you're having Unicode issues (more common on Windows than Mac OS)?

Answer 2

Simplest way to get around this is to open the file in a recent version of Google Chrome with built-in PDF reading plugin. Then you can use Chrome's search feature to find text, and copy-paste works correctly.

Answer 3

For the TV Manual example: same issue in Adobe Reader 8.1.2 on a Mac, but no problems using Mac's Preview to copy or search text. Also, sending it to a Gmail account and then choosing "View" and then "Plain HTML" reveals the text. But Adobe Reader doesn't like it.

Its document properties shows "Encoding: Custom" for the fonts. Another document shows things like "Encoding: Ansi" or "Roman", and has no issues in neither Preview nor Adobe Reader on a Mac:

However, both the Leadtek and the Swann examples give problems in Preview on a Mac as well, and in Gmail, and both show "Encoding: Identity-H". The Phonedisc test fails too, with "Encoding: Custom".

Confusing, and not consistent, but on some Adobe forum I found the following explanation for yet another example that shows "Encoding: Custom" (emphasis mine):

After looking inside the PDF it turns out that no usable encoding information is present (neither in the PDF nor in the embedded font data) to derive the meaning of the characters/glyphs that are displayed on the pages in the document.

The fonts actualy are all embedded, but in a way that all encoding information has been removed. This is a typical example of a PDF that is syntactically fully compliant with the PDF spec but where important information about the meaning of the text in it has been thrown away during the process of making the PDF. As far as I can tell it would be very difficult to recover the encoding info.

This does not explain why Mac's Preview (and apparently Infix as well) can handle some of the examples when Adobe Reader fails, even with "Encoding: Custom". Maybe Preview has no issues when the exact font happens to be present on the computer itself? Or maybe it's just guessing an encoding, which happens to work for some but not all of the documents?

Whatever causes this: if passing through Google Docs or Gmail doesn't work, then maybe the easiest (but far from easy) workaround is indeed to save as TIFF and then do OCR. Services like Evernote might do it on the fly (it does OCR on images; I doubt it will do OCR on a PDF).

Answer 4

The download of file 1 failed for me, file 2 I could open with xpdf, a fast and open-source pdf-viewer. I guess it can't handle forms, but for pure text and grafic I prefer it for its fast startup time.

Answer 5

Unfortunately it cannot be helped. PDF documents do not actually contain any letters, but they contain shapes of letters. In other words instead of reading a letter and drawing it on the screen Adobe Reader as any other PDF reading application would simply draw the vector graphics encoded in the file.

However, some PDF readers come with software that allows to analyze the shape and recover the text by using text recognition. It works same as if you scanned a paper of printed text and used software like ABBYY FineReader to convert it back to text, but due to infinitely high quality of vector drawings results are typically much better than for scanned documents.

Some documents can be protected from being converted to text by fooling the Adobe Reader. For example letters can be drawn in several overlapping shapes in such way that visually they would still look the same, while text recognition software would fail to recognize text. Your document is an example of such protection.

One way would be to print the document into an image and let text recognition software recognize it. Higher resolution for the image will improve the quality. This method however is not really handy.