Assume that I have a smart card that returns an 8 byte (for example) random value on reception of a command. The question is that:
How I can check if this value is really random? (I don't have any access to implementation and mechanism. I just see an 8 byte hex number.)
In the other world, Is there any way to check the randomness of output of the random generators based on the output? or the only way is analyzing the implementation and mechanism of generator?
NIST has defined in NIST Special Publication 800-22 an extensive set of statistical tests for verifying a source of randomness.
However, while this is good enough to assert that a RNG is not obviously broken, it isn't sufficient to assert that a RNG is "good". It is the nature of RNGs that very subtle flaws can lead to breaking it. The only proper way of checking the robustness of an RNG is looking at the implementation.
Real "randomness" comes down to the amount of entropy in the samples. And entropy is more or less impossible to qualify in blackbox testing, as others pointed out. A good conditioning function and a simple input with very little information can pass statistical tests. So the tests can only point out that there is or isn't some inherent statistical weakness in the way the numbers were produced, they can't really confirm that they're unpredictable.
There are several elements that are important in evaluating an RNG's randomness, unpredictability, or its entropy:
RNG evaluation has been getting a lot more attention recently, and the standards are becoming more stringent. You can look at the NIST SP800-90A/B/C series for a good overview of RNG design, and especially SP800-90B (still draft) for entropy source evaluation. BSI also has guidelines in AIS20/31 which explain how to evaluate RNGs.
And finally, NIST's STS 800-22a is a good test suite, as already pointed out. There's also the BSI's AIS20/31 test suite, and the Dieharder test suite as well.
You can't. It is impossible to verify that the output of a purported random generator is random enough for cryptography. (It is possible to verify that it's random enough for some applications such as Monte Carlo numerical methods, but not, say, to generate a cryptographic key.)
There exist cryptographically secure pseudorandom number generators, that is, random number generators which are deterministic but nonetheless indistinguishable from truly random sequences. We don't know how to distinguish the output of a CSPRNG from a truly random sequence: given part of the output of a CSPRNG, we don't know how to reconstruct the outputs we aren't given.
(We actually can't prove mathematically that a CSPRNG exists, but there are CSPRNG built on cryptographic primitives such as AES and SHA-2, which are CSPRNG under the same hypothesis that make AES suitable to derive encryption algorithms from, and SHA-2 a cryptographic hash function.)
Since a CSPRNG is deterministic, you know its output if you know its initial state; more generally, if you know its internal state at some point, you know all subsequent outputs. Thus a CSPRNG is a suitable way to generate cryptographic keys and other random values for cryptography only if the adversary doesn't know the internal state.
A CSPRNG whose internal state is known to an adversary (not suitable for cryptography) is not distinguishable from a CSPRNG whose internal state is not known outside the smartcard (suitable for cryptography). Thus it is impossible to test the quality of an RNG from its output alone. All you can see from the output is to detect some grave flaws, but not all grave flaws.
In order to verify that an RNG does its job, it is necessary to look inside. This is done in two parts:
(An RNG based on physical processes alone pretty much always has detectable biases, so physical RNG are not used directly, but fed as entropy input (i.e. seed) to a CSPRNG.)
The CSPRNG implementation is validated by examining the source code, checking that the algorithm is a good one (algorithms from standards such as NIST SP 800-90A are typically prefered or even mandated), and conducting a few tests to detect implementation errors such as storing the result at the wrong address.
The seeding of the RNG is more delicate to validate. Some systems include a “true random number generator”, i.e. a piece of hardware that returns unpredicatable bits based on unpredictable physical processes such as radioactive decay or chaotic oscillators. Validating these is conducted with statistical tests that demonstrate that the output of the TRNG has sufficient entropy.
Note that the TRNG may actually not be “truly random”: it can have biases. What is important is that it is sufficiently random so that the probability of correctly predicting its next output is so low that the adversary will not be able to make an informed guess. These biases are why a CSPRNG is applied to the output of the TRNG: the use of a cryptographic “mixing” function turns observable correlations (e.g. if a bit is 0 then the next bit has a 52% chance of also being 0) into correlations that are impossible to calculate.
It is also possible to make an RNG by seeding the CSPRNG once and for all with a secret value. This is viable only you are sure that the internal state of the PRNG will not leak out. The seed value itself must be generated randomly, ultimately from some form of TRNG; but this way, the TRNG can be in the factory, rather than in the device where the RNG runs.
As a purchaser of smartcards, you cannot look inside to see what the card is doing. (Not being able to look inside is an important property of smartcards.) However, the card you're using may have been evaluated by a security laboratory. There are standards to evaluate the security of smartcards, such as Common Criteria protection profiles. You can check if your card has been certified. CC aren't the only such certification; credit card companies such as Visa and Mastercard have their own certification process. If the card has been evaluated, your vendor should point you to the certificate — it's a marketing argument. There are many smartcard certifications and they test different things, so check exactly what the certificate guarantees. You won't find out exactly how the RNG works (that's usually a trade circuit), but you'll know that a security laboratory looked into it and found no flaw.
Use it to generate an image: random x, y, r,g,b... spot any pattern..?
Update: While I agree that 1) this is not a very scientific method, and 2) it doesn't prove that the RNG is cryptographically secure, it is a quick and easy way to check if a RNG is not random... E.g. here's a visualisation of .net's System.Random:
