While it is theoretically possible to thoroughly examine the source code of Open Source Software to check for backdoors (neglecting a Ken Thompson hack), and given sufficiently adequate knowledge in Electrical Engineering one can probably figure out what a given observable circuit can do, how can one ever be sure an Integrated Circuit does what it is supposed to do and nothing else (at least on purpose)?
As an example, how can one be sure a TPM chip doesn't actually call your local security agency when it feels like it, e.g. via an integrated GSM modem or a bridged Ethernet port?
And even if the schematics where Open Hardware, how could one be sure the manufacturer (who probably won't let you supervise their super-secret production of your individual chip) doesn't add their own "optimizations"?