I have a machine running Mac OS X and I suspect there may be a backdoor installed.
I know that I could take it to a professional security firm for analysis, or nuke it from orbit, but I'm interested in dealing with it myself.
What forensic techniques might I use to determine if any backdoors exist in the system?
Additionally, what network configuration options might l want to put in place (firewall or IDS rules, for example) to help limit or detect the exploitation of backdoors in my system?
If some people have put some hidden backdoors in your system, and if they were competent at it, then you won't be able to find them. "Competence", here, means "having an Internet access and typing 'rootkit mac os x' in Google". See e.g. this. It is theoretically impossible to completely hide a backdoor, but only in the same sense that it is theoretically possible to write software without any bug. In other words, it is hard.
"Security labs", whoever they may be, can try to see in your machine if they find something fishy. Possibly, if you observe strange symptoms, they might be able to explain them without postulating a virus / backdoor / malware. If you do not want to show them your computer, then, no escape: you'll have to do the analysis yourself, which means acquiring a few years of technical skills.
Alternatively, reformat the hard disk, then reinstall from scratch. Very few malware can survive that.
If you are looking for an inactive backdoor, then good luck, you'd need years of computer forensics skills to trace it down. On the other hand, if you are looking for a backdoor that is in use, then using traffic analysis from another system or a hardware device on the network could allow you to see if your computer is emitting any unexpected packets.
This could help discover if someone is actively using your computer for something, but still requires a pretty high level of technical knowledge since there will be some background communication on your system even when it "isn't doing anything."
The efficiency is not a variable here. The trade off is assurance of integrity against resources expended. To achieve complete assurance that your system has perfect integrity (i.e. no one may use your system without your approval) you would need to expend an almost infinite amount of resources.
At the very least you would need an operating system with much stricter partitioning and then OS X. At the extreme you need a specialized processor which provides strict physical separation of data and control (Harvard architecture as opposed to von Neumann architecture). Given the number of system components outside your control (cpu, motherboard, network card, OS, and other software) even an expert would have difficulty achieving a high integrity assurance of a system without outside help.
Given that you are not a malware expert the best you can hope for is reduction of risk by reduction of exposure, vulnerability management, and reduction of visibility to threats.
Limiting exposure means steps like reducing the total number of hours the system is connected to a network, reducing the size and scope of sensitive data stored on the system, and reducing downloading and installation of software.
Vulnerability management means tracking all the components of your system and continually upgrading or patching any vulnerable components. This is primarily software, but could additionally be a network card or peripheral device. It means monitoring of sources of your OS and applications for alerts about vulnerabilities and reconfiguring or patching your system as needed.
Reduction of visibility to threats means not advertising where or what your system is and what it contains. The easiest way to illustrate this is by showing the opposite. Don’t post on facebook that you started a taking credit cards for your small business using your Macbook. This alerts a potential attacker to a valuable target (the credit card numbers) as well as what kind of vulnerabilities it may have.
Given a system in an unknown state, it is, as others have said, very hard to detect backdoors.There are various tools you can use but a proper discussion of them goes somewhat beyond the scope of a post here.
However, given a system known to be in a good state, it is possible to detect any modification of its state using a host based integrity detection system such as tripwire and ossec hids. This maintains a secure databaseof file hashes of the executable code on your system. Providing ameans for you to see any changes occurring outwith your patching/development activities.
