Short answer
Where would this data live on the device if not on something like flash storage?
It probably is flash, but it's put in a chip that's hard to take apart and examine without destroying the data you're trying to read. It's not impossible to do, but it's really quite difficult and expensive. It's not like a hard drive or flash drive where you can just remove it and plug it into a new computer.
Long answer
It is stored on a tamper-resistant solid-state chip. Most likely it is flash, but is designed in such a way that it's difficult to non-destructively remove the IC cover to expose the actual die (a process known as decapping). There are many ways to achieve this, but they can all be bypassed with enough effort. You should assume that anyone with a sufficiently-equipped laboratory will be able to break into it.
The usual way to read data from a poorly-protected IC (such as an 8051 chip with the firmware read-protect bit set) is to decap it with acid and then take a detailed photograph of the die. This makes it possible to reverse engineer the design and find the relevant components that will be storing the data you want. If you know where the data is stored, you can simply read the bits off it by examining the electrical properties of those components. A protected IC (like a TPM or Yubikey) merely attempts to make this more difficult. Many barriers are put in place to slow this down, to obfuscate the location of the secret data, and to increase the chances that the die will be damaged during decapping.
For example, see this discussion on forensic recovery from microcontrollers. It has this to say:
Decapsulating the YK4 and sticking it under an optical microscope reveals, well, not a whole lot. Pretty much every active part of the die apart from the bond pads is shielded by a metal layer that prevents the kind of visual inspection and simple UV tampering I described earlier.
One would have to de-process the device further using more exotic (and dangerous) acids or by wet lapping to reveal the true structure hidden underneath. These are destructive processes that render the circuit non-functional.
That’s not to say these types of defenses can’t be bypassed, but getting around such protections usually requires significant investment. Other types of mitigations focus on the detection of decapsulation by placing light sensors on the die itself to detect when the chips powers up after the molding compound has been removed. SIM cards often include an active mesh layer that can detect when a single trace has been cut.
Another technique to retrieve data is to use glitching attacks, which are made possible when the voltage or clock are controlled externally. If the voltage or clock are varied, it can induce glitches that may result in side-channels that expose secrets from the otherwise "sealed" integrated circuit. A well-designed secure storage system will make this nearly impossible, but new attacks are discovered from time to time. You can never completely rule glitching attacks out, but they may not be easy. If someone is going to decap the chip and expose the die and run it while beaming lasers at individual transistors and traces for fault injection attacks, there's not much they won't be able to recover.
Regular passive side-channel attacks (monitoring power usage, heat, sound, or radiation during processing of sensitive material) are easier to protect against. This can be done by using constant-time operations which take the same amount of time regardless of the data being processed, or with masking, which ensures that an operation takes the same amount of electrical power regardless of what is being processed. This is very effective and protecting against passive attacks, but glitching attacks as described above may force the system into an undefined state that violates the security guarantees provided by constant-time operations and masking, and could reveal secret data.
