PAM does not just do authentication, but authorisation and session services. You probably want to keep it on as it adds quite a bit of flexibility.
PAM will be called for a successful pubkey authentication, because session and account services are still checked.
PAM can do things SSH cannot. This list is not exhaustive:
- Deny a user access if SELinux is not in enforcing mode (if thats your thing).
- Set resource limits like max processes and max logins allowed.
- Flexibly deny a user based off of their user and remote source IP (possible in SSH too, but is pretty terse in PAM)
- Setup a series of environment variables you may want to pass.
- Create a home directory for a user if it did not exist.
- Deny users based off of the time/date of their access attempt.
- Deny inactive users.
- Deny users using an invalid shell.
- Setup key logging facilities of input.