Currently I use a distributed revision control system called 'darcs'.
I've tried both approaches outlined below using that tool and linux servers.
I've also used approach 1 with Subversion and with CVS.
Your mileage may vary on different server and/or the revision control systems.
But the basic approaches may be feasible one way or another on whatever 
configurations you have selected.
1- Using a single user account without divulging its password to anyone, you 
authenticate connections to that account via SSH using public keys.  You confine
the privileges of that single user account to only have access to the repository.
Contributors to your repository must generate a private/public key pair and
ship you the public key so that you can add it to the authorized keys.
If you remove it, he loses access.
2- Require your patch contributors to sign and email patches.
You maintain a public keyring which holds the keys of authorized contributors.
all patch signatures are verified against that keyring before they are
applied.  So only patches signed by authorized contributors are applied.