If this is a new paste you just put, you only want it to be a uniform thin layer on top of the CPU or the heatsink (don't put it on both surfaces), so you don't need to scrape it all off completely. I usually use an old credit card as a spatula to spread the thermal paste thinly and evenly. You can use a similar soft plastic tool (you don't want to damage either the CPU or the heatsink).
If this is an old paste, it is better to scrape it off and put a new one if you removed the
heatsink from the CPU.
If you managed to make a big mess, it is sometimes easier to just scrape it off and re-apply. Some pastes contain silver particles, therefore it is slightly conductive. You don't want that paste on anything else except the surfaces it is intended to be as it might create a short circuit.
Update: According to Intel's instructions on how to apply thermal paste, soft, dry cloth or tissue should be used to wipe off previously applied/used thermal paste. My opinion is Isopropyl alcohol, Googone, or other substances are risky because they can either contain too much water or be too chemically aggressive. Soft, lint-free cloth works just fine.
3Yes remove ALL of the old paste. The best way to remove old past is to use Q-tips and isopropyl alcohol/googone. – jmreicha – 2011-10-19T13:35:04.450
2I would not use googone as it will leave an oily residue. Polish the cpu with a paper towel to finish the process. – music2myear – 2011-10-19T16:22:38.133