For a mail server, there are free services that will check the various security measures for your domain. Check this site as an example:
http://mxtoolbox.com/
Check your domain and implement the recommendations that the site suggests and your mail server should be secure.
Since you have other services running on the server (assuming Ubuntu, and exposed to the Internet), you might have any number of security weaknesses. Have you tried using penetration testing software, or researched ways to secure Ubuntu server? There are plenty of tutorials available for securing Ubuntu. Here is a highlight of things to implement in Ubuntu for security from this blog post:
https://www.thefanclub.co.za/how-to/how-secure-ubuntu-1204-lts-server-part-1-basics
- Install and configure Firewall - ufw
- Secure shared memory - fstab
- SSH - Key based login, disable root login and change port
- Apache SSL - Disable SSL v3 support
- Protect su by limiting access only to admin group
- Harden network with sysctl settings
- Disable Open DNS Recursion and Remove Version Info - Bind9 DNS
- Prevent IP Spoofing
- Harden PHP for security
- Restrict Apache Information Leakage
- Install and configure Apache application firewall - ModSecurity
- Protect from DDOS (Denial of Service) attacks with ModEvasive
- Scan logs and ban suspicious hosts - DenyHosts and Fail2Ban
- Intrusion Detection - PSAD
- Check for RootKits - RKHunter and CHKRootKit
- Scan open Ports - Nmap
- Analyse system LOG files - LogWatch
- SELinux - Apparmor
- Audit your system security - Tiger
You could follow this and other tutorials on the Web, just be aware that security is a moving target and you will need to stay on top of security updates for your hardware/software stack (server OS, application servers, firewalls, etc.) to prevent future attacks. This blog post is extensive, but created in 2012, so there are updates and things that will not be appropriate to your system.
This topic can get quite detailed, so give yourself time to learn, and check for answers to specific questions (e.g. how do I implement DKIM for my mail server?) on SF.