Technically speaking, it is possible to spoof both headers using an intercepting proxy but that's useless because we are doing it ourselves as an attacker.
When we send an ajax request using JS from another domain with our spoofed referer and origin header it won't really be spoofed. The browser would still send the legit header to the server.
My question is, why is it so that we can't spoof both these headers while sending cross domain requests?
You can't set those headers, because the browsers ignore attempts to set them. Browsers ignore attempts to set them, because they aren't supposed to be script-controlled. The restriction on letting them be script-controlled is for security reasons. If you could set the Origin header, you could break the security guarantees of CORS. Since the whole point of CORS is to open gaps in the same-origin policy for trusted origins only, letting a script (which can be attacker-controlled) spoof the origin is obviously unsafe.
Another question would be: why should XHR/Fetch allow you to set those headers? What legitimate reason is there to do that? There's a good reason to disallow it (breaks a security guarantee, setting us back to the bad old days of JSONP where you couldn't be sure who sent the request), and no good reason to allow it, so of course it's not allowed.
These headers cannot be modified as a safety precaution, the ability to alter them would break a lot of assumptions about how browsers work. There used to be some bugs in flash that would let you do this: https://www.securityfocus.com/archive/1/441014/30/0/threaded
