To prevent him from getting access to you emails you would have to take control away from your colleague - what I mean is, changing the server's login username and/or password. That is the main thing, but there are other things to consider.
If you have web mail as well then it is possible there are administrative tools available from within that so you would have to change his rights and all the passwords he knows. This way, he only has access to his emails as a normal user (assuming he still needs the email).
You may also need to ensure that no forwarding is in place to take a blind copy of what you're offering (assuming you're using POP style email).
Or, if possible, create even more rights and roles - giving yourself the 'full admin' and giving your colleague the 'nearly full admin' rights.
2This depends entirely on the nature of the server hosting the email and what access he has to it. Can you update your question with more details about the environment: [edit]. – Paul – 2012-08-14T02:52:50.123
Generally speaking, if someone has the rights to administer an email server, then they have the ability to read the email. – Keltari – 2013-05-13T05:58:44.837