I can only guess, but I think it may derive from a sense that asymmetric encryption is just more mathematically shaky than symmetric.
For instance, secure symmetric cryptography is known to be theoretically possible: Consider for instance the one time pad, which can even sometimes be used in practice. Standard symmetric ciphers are in a way a scheme for emulating this but with shorter keys (especially if you consider e.g. CTR mode).
Then consider the case of any asymmetric system. Here you need it to be mathematically possible to have a mathematical function F(secret-to-protect, pub key) that produces an encrypted result that cannot be used to infer secret-to-protect even if pub key is known to the attacker. I.e. you don't have any secret argument the function can use for protecting the secret (as is the case for the symmetric scheme) - the only thing that is secret is what you want to protect!
Still the function must be bijective since otherwise the intended recipient can't decrypt it. So all the information must be there. Indeed, it must be computationally easy to reverse the operation knowing some extra information related to the public key, namely the private key. It is much less clear that such functions exist - it is not known that they do. RSA and ECC are essentially proposals for such a scheme - but only proposals since they are not proven secure. Establishing the security of any asymmetric scheme, would by implication establish P != NP, and hence seems to be a very difficult problem.
So given that no known instance of a secure asymmetric scheme is known (and indeed an asymmetric scheme must satisfy some much more difficult mathematical properties than the symmetric scheme), but given that secure examples are known of symmetric schemes, then all else being equal, it is reasonable to be more suspicious of asymmetric schemes.
From a practical point of view, it seems that the complexity of factorization and the discrete log problem (as employed in asymmetric ciphers) is reduced year by year (even though the best algorithms are still exponential in time) whereas AES and DES (almost) are holding their ground. The biggest problem with DES was the small key size, not that the algorithm was cracked. 3DES is still secure, but quite slow compared to AES.