WRT to smart card being used as an authentication factor for computer access, the private key on the smart card can be protected by a PIN/password. So the smart card auth can also provide the additional factor of "what you know" in addition to "what you have."
For most users, the risks are acceptable when compared to the cost of managing provisioning/distribution and operation of multiple sets of physical authentication tokens. I've seen people in different organizations and consultants carry around a chain of smart cards and usb tokens for the various organizations they are consulting for. Granted, offering a service provider model for identity management is still a while away, there's a trend to offer simpler auth models (i.e. outsourced second factor auth, openid, hoth, etc).
For higher risk users or locations, I often see additional factors for authentication. For example, three factor is fairly common (biometric, card, pin, weight). In addition, I've worked with organizations where I only need to present a card + biometric in a mantrap during business hours but after business hours, I also need to use an additional factor. In addition, high risk users usually are subject to additional segregation of auth keys versus signing keys. So, there's a second device used to contain a non-repudiated signing key to authorize certain transactions.
Organizations and users with very sensitive information leverage additional controls that sacrifice simplicity and speed for security. Even a simple mantrap takes longer to auth a user than turnstyle.
With that said, organizations that consolidate all functions are usually larger in size with multiple physical locations. Such organizations can realize cost savings by consolidating into one vendor, one card, etc. Though I'e seen more small-medium organizations go the route of consolidated access control, they are few and far between (password's are still common and by far, cheaper than implementing hardware auth).
If anything, security 101 includes a long blurb and discussion on defense-in-depth. Consolidating functions don't necessarily open up more risk as long as risk is identified and managed.