If someone steals a CA's certificate signing key, the already signed certificates remain valid. But that doesn't mean they are not harmed. The whole point of certificates is that only a trusted party can produce them, so that when you see a certificate, you can be confident that it is valid and that you're talking to the entity you intended to talk to. If an untrusted party can produce a mathematically valid certificate, then the certificate is not worth the electrons it's written on. This means that all websites (not just the ones who used the CA's legitimate service!) are vulnerable to a man-in-the-middle attack carried out by the attacker (barring partial countermeasures such as certificate pinning). See What are the risks of a Certificate Authority hack for 'the average user'? for more explanations of the consequences of a stolen CA certificate signing key. Basically, not just the certificates emitted by this CA but all certificates are harmed.
To counter this harm, it is necessary to revoke the trust in the compromised key. This means that verifiers — browsers and other TLS clients (and servers, for client certificates) — must stop trusting the compromised key; until this is done no certificate from that CA can be trusted.
Normally, a CA has another key which they can use to revoke certificates. The revocation key should be kept with stringent security measures: unlike the signing key, which is used daily to sign certificates and thus has to be relatively accessible, the revocation key is hardly ever used (only to certify a new signing key or to revoke an existing one), so it can be made hard to access (typically requiring multiple officers of the CA to enter key fragments manually in an offline, physically secure environment). Once the compromised key is revoked and, a one is deployed, and the old certificates are re-signed, the harm is undone. As you can imagine, doing all of this, especially propagating the information to all browsers out there, would take a while, and given that it's an uncommon event, there would undoubtedly be platforms that keep thinking the old CA signing key is valid.
Another more drastic way is to update verifiers to remove the compromised CA from the list of trusted CAs. This solves the problem for certificates that aren't emitted by the compromised CA, but leaves the customers of the compromised CA in a lurch: they need to get another CA to sign a new certificate.
By the way, it has happened before.