FIPS compliance (or the European equivalents such as EAL levels) is a requirement in some markets or to achieve some legal properties; for instance, in France, time stamps are considered to be legal proofs (burden of proof lying on the party who claims the time stamp to be non-binding) only if the time stamp authority went through a certification process, in which it is required that a "certified" HSM (Hardware Security Module) is used to store and use the authority private key. If that was happening in the US instead of France, the "certified HSM" would be "a HSM which conforms to FIPS-140 level 4".
In private-owned areas (e.g. banking) requirements for certifications and/or FIPS 140 levels come from insurance companies: banks can get covered at insane-but-feasible rates because they can prove that they use strict security rules, and FIPS 140 levels are such rules. VISA and MasterCard are especially stringent about security (this is about money, thus it is important).
In more practical terms, FIPS 140 levels are obtained by demonstrating that some security characteristics are achieved (see the response from @bethlakshmi), which involves both hardware and software. Since, in general, correctness of software cannot be proven, the certification process involves showing that the system was developed with all due craftsmanship (unit test, code reviews, authenticated source code versioning, up to and including background checks on the developers themselves). This involves time, money (think 100k+ dollars), and an awful lot of paper. So the real, practical meaning of conformance to FIPS 140 is that someone spent a lot of time and money on the idea that the system is not blatantly insecure. This does not mean that the system is secure, but that at least there was some substantial effort and investment towards that goal.
Most opensource projects use a bazaar model which is efficient at producing good code fast, but is utterly incompatible with a certification process. Also, most opensource projects have no or very little funding. This explains why Bouncy Castle is not FIPS compliant, and even big companies such as Sun (now Oracle) have preferred not to run that process. There again, not being FIPS compliant does not mean "insecure" or even that the system could not be declared FIPS compliant, only that nobody found it worth the effort to go through the expensive administrative hoops of formal FIPS compliance.