Amazon now support the use of SafeNet HSMs with CloudHSM.
Microsoft Azure supports the use of Thales HSMs: Thales, Microsoft serve secure crypto in the cloud.
I doubt it's yet possible to describe "typical" usage, key management, or storage, the details will vary with both cloud provider and HSM vendor.
The Azure "Bring Your Own Key" concept does allow you to use keys generated on your own HSM, but you almost certainly still have to trust your cloud provider to manage the HSMs correctly.
(HSMs are designed to be extremely difficult for an unauthorized person to extract keys from, but allowing HSMs to share keys for failover and scalability means sharing secrets between them, and that has to be done in a trusted fashion. In the case of the nCipher (now Thales) HSMs that @thomas-pornin mentions, if an attacker gets uncontrolled access to a quorum of the Administrator smartcards used to add an HSM to a Security World, they can effectively extract keys from that World. In the SafeNet case, similar considerations apply to a Luna PED. Unless the cloud provider allows you direct access to "your" HSM in the datacentre, you have to trust their initial setup. Once secrets are established with the HSMs, then secure authenticated communication with them is possible.)
Edit, some years later: Amazon's CloudHSM now uses their own hardware, the SafeNet (now Gemalto) HSMs are now "CloudHSM Classic", which is going to be phased out because the Luna 5 HSMs are End Of Life AWS CloudHSM Classic FAQs.
Thales and Gemalto are merging, but the former nCipher HSM business is being sold to Entrust Datacard. Thales to sell GP HSM business.
Microsoft Azure now offers dedicated HSMs What is Azure Dedicated HSM?.