The whole grading mechanism is more propaganda and public relations than actual security. If you want good security, then you must mind the details and understand how things work internally. If you want a good grade then you should do whatever it takes to have a good grade. An "A+" from SSL Labs is a very nifty thing to add at the end of a report, but it does not really equate with having rock solid security. Having an "A+" equates with being able to say "I have an A+".
The principle of the SSL/TLS handshake is that client and server advertise their maximum supported versions, so that the version which is selected is indeed the most recent that both client and server support. This property holds as long as active attackers cannot break immediately (in real-time) the handshake of any version that both client and server support. E.g. client and server can support all versions from SSL 3.0 to TLS 1.2 (TLS 1.2 is, internally, "SSL 3.3"), and attackers cannot force a downgrade because even the SSL 3.0 handshake is robust enough not to be immediately broken by active attackers -- and that handshake includes the advertised maximum version.
... Unless the client is doing something dumb, which is reconnecting and advertising lower versions than what they actually support. Existing clients do that because some must talk to servers that do something even dumber, which is crashing/disconnecting whenever the client advertises a version which is greater than the highest version that the server ever heard of. Normally, when the client says "SSL 3.42" and the server knows only up to "SSL 3.17", the server should answer "we'll use SSL 3.17". The standards and documentation on SSL 3.0 and later versions have always been very clear on that subject. But developers still got it wrong; some servers, when they receive "SSL 3.42", simply shrivel up and die on the spot.
Thus, in order to cope with possibly incompetent servers, clients have taken the habit to auto-dumb themselves in case of connection error. This is where TLS_FALLBACK_SCSV comes into play: it is an extra mechanism, smuggled in the handshake under the guise of a cipher suite, so that a client may tell to the server "btw I am not dumb, I am just sending a dumbed-down ClientHello
in case you, as a server, would be one of these buggy server that blow up at incongruous times".
If a client applies the auto-dumbing and does not support TLS_FALLBACK_SCSV, then there is nothing the server can do to prevent a downgrade attack: attackers will be able to force the client to use an old protocol version by simply killing connections that attempt to advertise anything newer. However, if the client and the server support TLS_FALLBACK_SCSV, then they will detect any downgrade attempt.
This long preamble was only to get the context for answering your question. Basically, if your server does not support TLS_FALLBACK_SCSV, then active attackers will be able to force a downgrade, even if the client would have supported TLS_FALLBACK_SCSV. This is a problem in the sense that if you support TLS 1.0, TLS 1.1 and TLS 1.2, then this is for a reason: you prefer using TLS 1.2 when available, so if attackers have a way to prevent that, then this is, stricto sensu, a successful attack: attackers can force a situation that is distinct from the preferred, non-attacked one.
Now this does not mean that the consequences are serious. TLS 1.0 has a few known shortcomings, basically boiling down to two kinds of attacks:
BEAST-like attacks that use chosen-plaintext to leverage the fact that in TLS 1.0, with CBC-based cipher suites, the attacker can predict the IV for the next record;
Padding oracle attacks relying on, for instance, small timing differences in the treatment of incoming records (described as the "Lucky 13 attack").
TLS 1.1 fixes the first kind, with per-record unpredictable IV; for the second attack, TLS 1.2 offers AEAD cipher suites that do not need padding at all, thereby avoiding any padding oracle. However, both kinds of attacks can be efficiently mitigated in "pure" TLS 1.0, by using the "1/n-1 split" for CBC-based cipher suites, and with constant-time MAC computation, respectively.
Thus, if the client is sufficiently well developed to do a 1/n-1 split, and the server is sufficiently well developed to compute MAC in constant-time, then using TLS 1.0 incurs, as far as we know, no security issue that would need fixing. It may be argued that a client and a server that implement TLS_FALLBACK_SCSV are necessarily recent and "probably" up-to-date with implementation recommendation, so chances are good that they will do everything correctly and be able to run a safe TLS 1.0.
However, there is always a nagging suspicion that a server that supports TLS_FALLBACK_SCSV might support it only to get an A+ from SSL Labs, and may have not taken the effort to do constant-time MAC computations, because SSL Labs cannot readily test for constant-time computations.
To sum up: not supporting TLS_FALLBACK_SCSV is not necessarily a serious issue, depending on how well the client and server implement TLS 1.0 (by not supporting SSL 3.0 you already avoid the most glaring problems). However, good implementations cannot be guaranteed, and not supporting TLS_FALLBACK_SCSV is formally a weakness, even if it is not necessarily a vulnerability. That the weakness cannot be turned into a full exploit by attackers does not mean it does not exist.
In any case, you won't implement TLS_FALLBACK_SCSV because you want security; you will implement TLS_FALLBACK_SCSV because you want an A+. If you do not, then you will spend inordinate amounts of time explaining to many people that the "A+" grade is meaningless in that respect and that you can afford not to take it. In the long term, not howling with the wolves is too expensive.