Java applets attempted to lock down a privileged standard library from within the process.
APIs like serialization and URLConnection were intended to work in both a full access mode (Java out of the sandbox) and untrusted mode (applets, web start). Privileges were determined by checking the caller’s class loader. This created an attack vector for gadgets: stringing together a sequence of API calls to trick a privileged Java class (say JEditorPane) into performing a privileged action on behalf of an unprivileged class.
It didn't work because the boundaries and responsibilities were unclear. Why was JEditorPane responsible for enforcing URLConnections security rules, only because it used the URLConnection APIs?!
My understanding is that WASM has no complex standard library that needs to work in both privileged and unprivileged mode. Avoiding that design mistake doesn't guarantee WASM security, but it makes it much easier to implement correctly.