The most common cause of high-pitch audible noise is something loosely known as coil noise. Specifically, it's a vibrating toroidal inductor. As you can see in the picture below, they commonly apply glue at the factory to dampen the vibration. It is harmless to the electronics, but annoying to people and animals.
To answer your question, it's not "normal", just a common flaw. Personally, I find such noises unbearable, and a sign of poor manufacturing. If the computer is under warranty, I would ask for the power supply to be replaced. If it's not under warranty, and it uses a standard form factor, I would replace it with something that has met the criteria for 80 PLUS certification.
@Ivan Does the noise go away after a few minutes? – Jason – 2014-07-18T21:25:23.363
Nope, it stays there all the time when the PC is off and plugged in the electrical network. Through as I've said the noise is inaudible unless I bring my head very close(5-10 cm) to it. – Ivan – 2014-07-18T22:26:02.100
possible duplicate of Why do some AC adapters and power supplies generate a whining noise, and what can I do about it?
– bwDraco – 2014-10-28T11:08:34.957