When you use S3 SSE anyone with the right IAM credentials can read and/or write your S3 objects, just like if you weren't using SSE. At first glance it looks like the only added benefit is that the data is protected from situations where someone gets access to S3 in an offline manner, like disk drives or backups (which I doubt that AWS makes, it's much more likely that they rely on replication only). However, I think you need to compare it to the alternative to get the real benefits:
There are two components to needed for client side encryption with S3: an encryption key and IAM credentials for authentication and authorization. When using server side encryption you only need IAM credentials.
When using client side encryption you need to distribute the encryption key to all machines that have read and/or write access to the encrypted data on S3. In both cases you also need to distribute the IAM credentials.
If your machines are compromised your encryption key is compromised. You can invalidate the IAM credentials as soon as you know of the break in, and if you use IAM roles or temporary IAM credentials the attacker only has access to your data as long as they have control of the machine (which is bad enough, but might not be the end of the world, you also need to think of what happens next). With client side encryption the attacker will have your encryption key, and all of the data encrypted with the compromised encryption key needs to be reencrypted. With server side encryption you will not have to reencrypt your data, because neither you nor the attacker will have the encryption key.
Even if you don't have a break in there are situations where your encryption key can become compromised, if a laptop is lost or stolen, if someone who doesn't know better e-mails it to someone, or if someone quits and you can't be entirely sure that they didn't take things with them. At this point your encryption key is compromised and you should probably reencrypt all of your data. That can be a lot of work. With server side encryption all you have to do is invalidate the IAM credentials and issue new ones.
There are probably ways to mitigate the issues with client side encryption that I've mentioned above, but to me it feels like using SSE with S3 has fewer downsides than managing it yourself.
Finally, there's the issue of how secure it is to let AWS manage your encryption keys:
According to AWS the system managing the encryption keys is separate from S3, with the intention that if someone breaks in to S3 from the outside they will not get your data because they will not have the encryption keys. If they break into the key management system only (which probably isn't directly accessible from the outside anyway), they will not have your data because they don't have access to it on S3. They need to break into both S3 and the key management system to get to your data.
If they instead break into the physical data center they might get access to both the key management system and S3 at the same time, but the question is if this makes things easier or not. I think that first of all we need to trust AWS to have appropriate security measures in place to stop people from entering their data centers, and secondly that to actually get hold of keys from the key management system you'd need to do something more than simply yank some disk drives. As far as I've seen AWS does not publish exactly how the key management system is protected, more than saying that it is protected with multiple layers of security. It is speculation, but disk encryption is probably one of these.