I guess this could also be phrased as JWT vs. Basic Auth, similar idea?
Passing a token for each API request seems to be the more common and recommended approach, and I have the gut feeling that it just "feels better and safer", but I am not sure why and how to clearly communicate it.
A username and password is a blunt instrument. They are hard to get people to change, and they represent the full authority of their bearer.
Tokens can represent authorization on a much finer scale. A username and password can be used to request a very limited token. A password might be good for 90 days or more, but a token could be issued that lasts only three minutes, making them harder to exploit. A service could issue a token good for read-only access to a specific resource, ensuring the token can’t be sent to a different server for higher access.
Limited tokens also reduce a system’s attack surface. The token issuing system can be secured, vetted, audited, inspected and carefully monitored, while an ordinary server handling mundane data is often ignored on all those fronts for cost reasons. If an ordinary server doesn’t handle passwords, it can’t be used to steal or harvest them and provide elevated permissions to an attacker.
In most use cases, a token is a form of identification, similar to a username without a password.
If you are certain that nefarious actors or incompetence won't compromise that identification, then many people will use that as a form of authentication, such as with session tokens inside of browser cookies, or as the token in JWT.
A token is typically generated from a much higher entropy source than a username or a password, and is unlikely to be reused or shared with third parties, unlike passwords.
Additionally, if a token is compromised, it only affects one account in one system. If a user's password were compromised, it might affect most of the user's accounts across many systems, possibly also giving access to the user's bank.
(Please use a password manager and generate random passwords for every account, people!)
If the username and password are generated randomly with high entropy, and assigned only to a specific account and not re-used on other systems, then there is little difference between using a username and password, and using a token. In the end, both are simply authentication using shared secrets. Tokens just generally don't get shared with third parties the way that security-ignorant-users' passwords are.
Multiple reasons. A partial list:
Per-user secrets (generally referred to as "passwords", although they needn't always be memorable) are re-usable after the session ends (even if the user doesn't re-use the password elsewhere, which users have a deplorable tendency to do), and are therefore more dangerous to the user if compromised. You cannot rely on secrets transmitted over the network or stored in the client (since they must be sent on every request) or in a server database staying secret forever.
Per-client tokens can also be scoped more tightly (for example, restricting their access to only a few functions, or to use only from a specific IP address), and can be revoked (if compromised, or even just if compromise is suspected) without invaliding all other sessions. Passwords generally need to give access to everything, from everywhere, and if they are compromised you need to change them and invalidate all sessions.
Finally, using session tokens means users can use a third-party identity provider without exposing their credentials to your application (this is very common; for example, you can log into StackExchange using single-sign-on via Google, without ever giving StackExchange your Google credentials). Even if you don't plan to use SSO or support third-party auth at this time, building your session management on tokens rather than credentials will make it much easier to do so in the future, without requiring your users to expose their credentials on those third-party sites to you.
For security, passwords (secrets the user chose or that are chosen for memorability) need to be passed through a computationally-expensive hash function (sometimes described as a key derivation function, such as PBKDF2) and this is slow. A site owner really wants to only do that once per session, and in response, issue a token that can be verified very quickly on subsequent requests.
Additionally, tokens can be made verifiable without any request to the database at all (for an example, see JWTs). This not only avoids DB queries (except when initially issuing or refreshing the token), it also means that a request can go to a different server in a cluster than the one that handled the previous request, and the user's session can nonetheless be verified without accessing a DB (or shared cache or anything like that).
