Assuming that I have an infected image containing a self-executing malicious code that targets my image viewer and executes when I open the file. Supposing also that my computer is completely clean and not infected by another malware that can read the hidden code in the image, this code could not have been encrypted. Thus it must be unencrypted in order to execute, which means it is somehow visible.
What methods can I use to detect it ?
This is the easy case for anti-virus software.
However, do not suppose that a malware has to be unencrypted to be executable. Polymorphic code brings self-decryption to malwares, it can be helped by Metamorphism to ensure that even the very start of the decryption routine, since there must be indeed a unencrypted starting point, does not present any recognizable constant pattern.
It is trying to fight such kind of methods which make anti-virus software to be complex programs.
Such code would of necessity exploit a buffer overflow or other data-to-code jump trick. As such, it will only be effective against a limited range of hardware. So you can run a JPEG lint checker written for, say, ARM in a virtualized device. The exploit will be uneffective against the emulated CPU, which will then be able to report on the JPEG structure.... whether it contains illegal image blocks or, much likelier, corrupt APP tags. Even if the virtualized code is vulnerable, because the exploit targets a library which was ported to the emulated architecture - say, libexif - the exploit payload will be in the outer CPU machine language, and thus be ineffective. On some VMs you will be able to check what addresses contained the corrupted data.
