Multi-factor authentication is based on the idea that if one authentication factor fails, you have others to protect you. If someone tricks you into giving up something you know (ex: a password), they will be thwarted by something they don’t have (ex: a hardware token generator).
If you used a good password and were sure that no one could steal (or trick you into giving up) your password, then having a hardware token generator doesn’t really make sense. If you were sure that no one could steal your hardware token generator, then having a password doesn’t make sense. But this defeats the idea of multi-factor authentication.
So to answer your question, if you are very confident that your Yubikey won’t be stolen or cloned, then having a strong password doesn’t offer a huge benefit. But you also limit the benefits of multi-factor authentication if your device does get stolen.
This answer also assumes that websites implement all of the authentication factors correctly. This isn’t always the case since some websites such as Apple don’t require the second factor for all of their services. Historically some services like “Find my iPhone” required only a password and not your second factor even if it was enabled on your account (this may still be the case, but I haven’t checked).