All browser exploits do not necessarily use Javascript to reach their needs.
A browser plugin can be used to escape a browser sandbox for example.
This means that a malicious person could create a website, knowing that his victim uses vulnerable plugin X (that parses the text of each visited page), and display text on his website that, once parsed by the vulnerable plugin, would escape the browser sandbox and execute code remotely (in this case an exploit in the plugin would be leveraged to exploit a flaw in the browser)
As soon as an external input is read by a browser, a risk exists. In theory a maliciously crafted JPEG could have nasty bits that would make the parser lose its head - in this case good luck finding such an exploit, but you get the point :)
Note that disabling Javascript prevents most exploits from executing. For example, many malware toolkits use Javascript to detect what version of Flash and Java are installed to serve the most appropriate exploit to the user - disabling Javascript would prevent such toolkit to even start fingerprinting the browser