Hardware Security Modules's (HSM's) are used in systems where a company has determined that the risk involved requires a higher level of control that a personal key.
The Use of HSM's for Certificate Authorities
HSM's are common for CA applications, typically when a company is running there own internal CA and they need to protect the root CA Private Key, and when RAs need to generate, store, and handle asymmetric key pairs.
Uses outside of a CA
HSM's are suggested for a companies Privileged Access Security System. These system's use the concept of a "vault" which is unlocked via a key or multiple keys. These keys are usually only required on startup of the vault.
With vaulting systems, typically options for storage are:
- Store them on the server. This is not very secure even when you limit access to the server.
- Store them on Disk/USB. This requires that disk be mounted every time the system is restarted (trouble if you do regular restarts).
- Store them on a HSM.
In secure systems, this allows key to be generated without a human needing access to it, stored in a system that is FIPS Level 2+ compliant, and only accessed when a system starts.
Using an USB Key vs a HSM
Primarily, end user USB's are designed for the end-users access. They are storing keys which might authenticate them, have encryption keys for data they are responsible for, or are part of a process that identifies the user is who they say they are.
Examples of this are the use of a YubiKey to hold the private encryption key for sensitive company data. You might have access to a trade secret, or are allowed to perform analysis on compliance critical data such as PII and Healthcare or Credit Card Data.
HSM's are not designed to be a personal authentication unit. They are installed on servers to provide a secure generation, storage, auditing, and distribution of keys for high risk applications and systems.
Examples include the storage of server certificates for authentication between systems, encryption keys which are used by processes but don't need to be seen by humans, and shared keys between servers used to access encrypted data or establish encrypted connections.
Pseudo-Summary
You could think of the use cases in terms of people verses systems.
If a company has a need to provide increased security of systems which a person uses require access to as part of there job, something like a YubiKey makes sense.
If a company has a need to provide increased security to systems where processes and the systems themselves need access but a user wouldn't normally be required to know, then a HSM is the better option.
Anecdote
Imagine walking around with they key to your companies entire certificate authority in your pocket that provides the authenticity for 1200 servers and that Key slipped out a hole in your pants. Now imagine the nightmare of having to re-issue a new CA certificate, all the new signing certificates, and all the server certificates.
edits:
At the suggestion of @MikeOunsworth, I updated and expanded this answer.