We know that Intel processors have one (ME) so it is definitely possible.
In general, how could you even trust a piece of hardware like a CPU or network card, even if the manufacturer says there is no backdoor? For software, it is easy since you can access the source code verify it and compile it. But for hardware?
The short answer is, you can't. The longer answer: there are a few things that can be done to increase your trust in hardware, though they also just shift the root of trust elsewhere.
A first interesting question you pose is the software/hardware distinction. To not go into the discussion about the possibly blurred boundary between the two here, I'll understand "hardware" to be non-reconfigurable logic implemented in some physical device (i.e. I'll exclude firmware such as the intel ME or microcode).
Backdoors can be inserted into a physical device in a number of stages: from conceptual architecture, through logic design, up to fabrication. To ensure no backdoors are inserted, you would need to validate the whole process from the end product to the beginning.
The good news is that the initial stages are very similiar to software - in fact, logic is usually designed using hardware description languages (HDL). These designs can be audited in the same way software can be audited. The step from here to fabrication involves multiple conversions e.g. to lithography masks using synthesis software, in a similar way to how software is compiled using a compiler. These too can be audited just like a compiler. (As a tangent - the bootstrap problem is a really interesting problem where you consider the possibility of the compiler compiling your compiler being untrustworthy)
So this leaves the last step: fabrication. Validation at this stage is usually done both by inspecting the fabrication process, and by randomly sampling devices from the same production batch (produced using the same lithography masks). For instance, the masks used can be compared to a validated trustworthy copy to ensure that no backdoors are inserted at this stage. Similarly, randomly sampled devices can be delayered and inspected under an electron microscope.
However, as a consumer, these steps are usually not available to you. For most chip producers, this whole process involves a lot of closely kept trade secrets, and aren't publicly available. This is why there is a movement towards creating open-source hardware toolchains and HDL implementations of common logic modules and systems - though there are a number of problems here too.
Finally, as @knallfrosch correctly points out in the comments, backdoors may also be inserted after production, either at a distributor, while the product is being shipped to a customer, or in-place (c.f. evil maid attack). An example of such practices by the NSA has come to light through the Snowden affair. Tampering at this stage may range from hardware implants added to the device to editing the circuit on a silicon die e.g. using a Focused Ion Beam (FIB). Mitigations at this stage usually rely on these kinds of tampering leaving externally visible traces, which may be additionally enforced using e.g. tamper-evident packaging (something every-day users can do is the glitter and nailpolish technique). Furthermore, minute device-specific imperfections that are a side product of fabrication may be used to create so-called Physically Unclonable Functions (PUFs) which may be designed in a way that tampering will most certainly alter or destroy the PUF and therefore be detectable.
plonk's answer already outlines the technical options for increasing trust in your hardware, such as device inspection or open-source hardware.
However, at the end of the day, the thing is:
You need to trust someone.
Fundamentally, this is the same for any type of hardware (or service) use use: How do you know that the manufacturer or the installer of your door lock did not retain a key? How do you know that your physician does not secretly share your confidential health information with a shady company to get kickbacks?
The answer is: You cannot know for sure - but there are strong social and legal incentives for them to play by the rules.
So in the end, you can only try to choose trusworthy vendors. That's more of a social than a technical problem; some things you can look for are:
The last point specifically is where things like open-source hardware help:
Even if you cannot verify the hardware, someone else (or multiple someones) may, and the risk of being found out will help to keep a vendor honest.
Hardware backdoors are expensive. Very expensive. You need to influence complex supply chains, and the people responsible for them. Each time a backdoor is used it risks being discovered and published, becoming useless over time, and bringing down costly reputations of people and companies. For each of these times you need to account the benefit brought to the attacker vs the cost of the attack.
Hardware is usually the most secure part of systems. The security track record of software you typically have on your computer is frankly terrible. Even if you are an above-average user, there's good chances you'll have at least one critical vulnerability in your computer software within a year.
Also, there is no way to have absolute trust in anything. As the old saying goes: the only secure computer is the one that is turned off! And disconnected from any power sources.
But that doesn't really answer your question - How can you trust it (somewhat)? There's actually a number of practical things that you can do as an interested user or developer.
In most realistic attack scenarios, a hardware backdoor will have to escalate to a intrusion on the OS level or communicate over a distance (network, radio, ...). You can put systems in place to detect these.
Intrusion detection systems can help you detect backdoors. This can be done both within the potentially compromised system and at the network level. Obviously you can only trust the former so far (read on for mitigations to that problem).
Anti-virus and rootkit detection might offer some very baseline of protection if they're heuristic-based, but most likely would be evaded by an attacker that has a hardware backdoor if they're well known.
A SDR receiver let's you monitor the airwaves for anomalies if you have reason to suspect them. Easier said than done, but within the realm of possibility in a controlled environment.
The RTL SDR costs around $25
It's worth noting that a sophisticated attacker will likely keep the detectable time window very small, so detection alone might not help you a lot.
Using open source hardware is the easiest thing you can do. There's a number of vendors selling completely open systems, including BIOS and peripherals firmware, or disabling Intel's Management Engine, for example.
A well configured hardware firewall can prevent undesirable communication with the outside world.
Of course, the firewall itself can be compromised. Stacking several firewalls/IDS with different hardware, software and OS makes things more expensive for your attacker (and you!)
If your attacker is very sophisticated they might decide to build a mesh network to bypass your firewall entirely, so it might be a good idea to consider what other things your computer can communicate with on the same network.
Unplugging the network might not be a bad idea, actually. While you are at it, do the same to any plugged USB cables that might have embedded radio transmitters, including your keyboard. Or maybe just put your computer into a large safe, that could be safer and almost as convenient.
If you really have enough time money and willpower to do the whole computer-inside-safe thing, you're a real pain. Your attacker will avoid the network altogether. They will transmit data from your computer using sounds you can't hear, heating up your CPU or transforming your motherboard wires into a GSM antenna
Perhaps you should consider using a typewriter instead? :)
I've got a computer sitting on my bookshelf that has no possibility of having a backdoor.
The reason for this is quite simple. No remotely plausible backdoor design could know how I wired up the IO ports. My assembly code drives all IO manually including the IO clock, which is just another data pin as far as the CPU is concerned.
In addition, had there been a backdoor looking for an SD card, I would have noticed because I debugged my SD assembly with LEDs on the SD pins.
Having a machine that has no backdoor can be used to verify that another machine doesn't have one. I could bootstrap a compiler using this machine as the origin, and use that to cross-check something with an Ethernet port, and so on ladder up until I could check current gigabit Ethernet.
In the case of IME, it was quite defeatable for the simple reason that it couldn't work if you added a PCI card with an Ethernet controller it didn't have a driver for (which was pretty much any non-Intel).
But not-trusting is stupidly expensive and you probably can't afford it. Intel burned a lot of trust with IME (there should have been a simple on/off switch in bios settings that really worked) and it's going to be hard to get back.
In general, how could you even trust a piece of hardware like a CPU or network card [...]?
A backdoor is pointless unless it produces some effect noticeable from outside the computer. For example / most likely, it communicates information about the system over an internet connection. So one pragmatic check is to use another, separate piece of hardware to monitor the inputs and outputs of the system. For example, check that all the packets out of the original hardware are as expected, and there are no extra packets.
Now the concern is that the second computer is also compromised to hide this information. But you can increase your confidence by making it as different as possible from the original hardware: different age, different manufacturers of the components like the processor, created and purchased in a different country. The level of conspiracy needed for a backdoor coordinated between these products is quite a bit higher.
One wrinkle is that if you are not running open source software, then you may not even be able to know/check what the expected packets would be (for example DRM with some encryption key you can't access).
The other concern is that the system is communicating in a way you aren't monitoring. For example, you don't think it has a wifi connection but it does; or it uses high-pitched sound waves that you can't hear; or it waits for a USB stick to be plugged in and hides information on it; or something else. You'd want to make a list of such possibilities and check for them.
I'm not saying this is easy or accessible to the average person, but the question reads as more theoretical to me.
