As a general rule, there's no need for something to stop malware from executing. Malware, like any other software, doesn't execute itself - something needs to start executing it. Browsers don't automatically execute most kinds of software they download from the Internet - that would be an insanely insecure thing to do - but they do offer the ability to run downloaded software (e.g. an executable you just downloaded) and if you, the user, tell a browser to do so... it will[1]. At that point the software (malicious or not) will have access to all the data that you, the user, have access to.
There is one class of software that browsers download and automatically execute, and that is client-side script (these days just Javascript, historically also things like Actionscript for Flash). Client-side scripts have three major protections against being used maliciously:
- An API sandbox that simply doesn't give access to dangerous actions. For example, there's no function available to JS that would let it search your drive for a passwords.txt - or any other file - nor to read one even if it knew where it was. JS also can't do things like write a file to your drive - it can offer you a file that you can choose to save, but you'd choose where and what name, or choose to not save it at all - or execute anything except more client-side scripts.
- A managed runtime that doesn't expose raw memory access (pointers or manual memory management). This is important because, if scripts could access pointers, they could bypass the API sandbox by e.g. finding the
system
function in the C standard library (which is probably present in their address space, since JS engines are written in C/C++ or something built on top of it) and directly invoking that. They could also deliberately cause memory corruption that would break the safeguards on what data they can access.
- A permissions sandbox, enforced by the operating system, that prevents the JS engine from doing anything interesting. All modern browsers run scripts in special low-privilege processes, which have way less access to the rest of the OS than the user account running the browser normally has. The exact permissions vary by browser, OS, and version of each, but in general they restrict access to the file system, to other processes, to the network, to inter-process communication channels, and so on. There are very limited (and heavily verified) communication channels between the browser render processes (which include the JS engine) and the main browser process; these channels are used mostly just to feed the render processes code (HTML/CSS/JS), retrieve the rendered graphics to display, handle calls for things that HTML/JS are allowed to do like top-level navigation, and handle calls for things JS isn't allowed to just do but is allowed to ask the user for (like saving a file or accessing your microphone).
These protections work together, providing defense-in-depth against exploits. They aren't perfect, though, because JS engines and browsers and OS security are complicated and hard to get perfect. Sometimes there's a bug in the code that implements a JS API, and this bug can be used to overwrite a pointer even though JS normally doesn't have access to those, and that can be used to access and exploit a vulnerability either in the privileged part of the browser or in the OS itself to gain more privileges, and then you have malware running on your machine that can do things like find secrets stored on your drive. However, note the multiple things that have to go wrong; any one vulnerability won't get an attacker very far. That means there's more time for the browser and OS developers to notice each such vulnerability and patch it, before an attacker can build a full "kill chain" to get from the almost-no-permissions state of JS executing normally to running outside the browser sandbox like normal malware. This is why it's very important to install security patches to your operating system, browser, and anything else running either privileged or with exposure to potentially-malicious actors such as Internet servers.
Obviously, a modified browser might not have such protections. If it's modified specifically to remove them, though, the browser is malware. It doesn't need to browse to any malicious web content; if you run it at all, it can already read your secrets and encrypt your files for ransom and do other malware things. I suppose somebody might do it for plausible deniability - release a vulnerable browser and then, when it's exploited, claim that they didn't know it was vulnerable and have no idea who exploited it - but that seems awfully round-about and only people who visited a page hosting the exploit while using the vulnerable browser would be affected.
[1] Absent anti-virus software attempting to block it, AV being the main class of software that does specifically try to stop something from executing even when something else tries to start it.