(Quick answer, I don't have the time to research all I remember and double-check my facts.)
There are multiple things to consider here:
USB sticks may carry payload that you do not know is there, e.g. a virus to infect the computer it's plugged in to.
Things like BIOSes, network cards, etc. can often be flashed to upgrade the firmware. You could also install viruses on here and they often have full memory access.
USB sticks are rewritable. There are very few that actually implement hardware write locks, e.g. by using a physical switch. I think Jacob Appelbaum (from The Tor Project) looked into this, he mentioned it somewhere... I forgot where. Such USB-sticks are basically unobtainium as far as I know.
RAM memory is not off and gone: it fades. Off the top of my head, it's gone after about a minute in room temperature. You can google this, search for cold boot attacks.
Be sure to power off the computer (if you aren't sure, pull out the cable and battery), in standby mode memory is obviously retained.
The data is only as secure as the host. If your offline computer contains a hardware keylogger that someone then later retrieves when you are out of the house, well, being offline didn't help much in this case.
I think it was with a microphone inside the same computer, but some chips on the motherboard may leak data. There was some sort of side channel attack to extract private keys... I don't really remember, but you might want to keep a bit of distance between the computer and other systems. This Google search seems to give some info.
So it depends what you want to protect against. Are you hiding state-secret information, or are it love letters that you are trying to keep private?
In theory you're right: under normal circumstances all data will be gone when the live USB-stick is removed and the computer is powered off. In practice, there are a lot of things that (powerful) adversaries can do to all parts of the system.
If you buy a USB-stick, assuming it's not backdoored to start with (which is a reasonable assumption under normal circumstances) and you only use it on your offline computer, then I don't see how it could get infected without someone physically touching it. Same goes for the computer. The problem is: how do you get your OS while offline? Somewhere you somehow need to connect to the internet to create the live boot. Perhaps this would be safest on a computer of a friend that is not tech-savvy and has no clue what you are talking about, and who is preferably not a close friend either. For bonus points, pick another friend and use his computer to verify the checksum. I think as individual, this is about as close as you can get to protect against extremely powerful adversaries.
By the way, lots of stuff in this post may be tinfoil hattery, but I'm trying to talk in terms of "what would be provably secure." For example one would normally assume that "of course a computer that you just bought is not backdoored", and it would be tinfoil hattery to assume otherwise, but how do you know? If you are not trying to protect against powerful adversaries like a government though, much of the advice in this post can be disregarded.
Edit: This answer is only about the first paragraph of the question, not the second that the asker added in an edit.
