Oligomorphic and polymorphic software (Oligomorphy is a type of polymorphy, if I remember my classes right) can be detected heuristically by using behavioral analysis. Essentially, you can determine whether or not a piece of software is malicious by looking not at what it is, but rather what it does. You may have seen your firewall prompt you whether or not to allow a program to connect to the internet, that's an example of simple behavioral control of software. Most anti-virus programs have a heuristic engine that will observe the software on your computer and look for programs that do suspicious, virus-like things like:
- attempting to read or write memory allocated to other programs
- attempting to read or write files belonging to other programs
- opening spurious network connections, especially using protocols like IRC that are commonly used by botnets for C&C, or SMTP, commonly used by botnets to send spam
- taking actions to hide itself (renaming processes and files frequently, disabling the task manager, running under the same name as other known programs)
Heuristic detection is imperfect, though, because it essentially involves taking an educated guess (something that computers are bad at). So, there's a constant balance between missing viruses and bothering the user too often (a high false positive rate). Good antivirus programs will allow you to adjust the sensitivity of the heuristic engine.