The following anecdote does not have a general value:
In 1999, I took the source code for PGP (not GnuPG, still in its infancy), version 5.5, and compiled it on my machine (which was an Alpha running NetBSD, i.e. a rather "normal" Unix system). PGP's source code had been available for quite some years, and it was often touted as necessarily secure since it was open source.
Of course, I almost immediately encountered bugs. I looked at the source, and it turned out that:
- The code was using the
char
type as if it was unsigned, and, as such, was completely incapable of correctly processing any character beyond the 7-bit ASCII set. It was so throughout the code.
- The source code was several hundreds of source files, in more than 30 sub-directories (and sub-directories thereof).
So I came to the conclusion that despite being a high-profile target, with source code open for inspection for several years (the source code had even been printed as a book to work around the US export regulations of that time), nobody actually bothered to look at it (or those who did, did not possess enough knowledge in C to do it correctly).
The code did not make it easy to be audited; auditing code is already a quite boring endeavour that unpaid amateurs are unlikely to tackle on their own accord. It is much more interesting, for the amateur, to rewrite the whole thing from scratch, which is exactly what occurred (with GnuPG).
While I do not believe, generally speaking, in hordes of unnamed amateur reviewers who audit code simply because it is there, there are projects which gather a bit of review, mostly because of people who want to modify the project to suit their needs. Therefore, malicious alterations of security tools are a risky thing. What you will find in all projects, open source or not, security-related or not, are bugs, some of them being exploitable into behaving to the advantage of third parties (i.e. vulnerabilities).
The plausible way to inject a backdoor in open-source security tools is to make an honest-looking implementation or design error, in particular when dealing with PRNG (there have been some allegations, apparently unfounded, of such a backdoor in OpenBSD; see this answer; the backdoor was not there, but everybody took the thing seriously, because it is plausible).
On a pure theoretical basis, external audit can find out all backdoors exactly as much as it can find out all bugs in the audited code -- i.e. it cannot.