It depends how that figure is calculated.
Some tools include buffers and cache in the free memory figure, some don't. I/O caching has a huge impact on webserver performance. If you've got shell access, try running 'free' to see what the actual values are, e.g. from a system with virtually no load on it:
total used free shared buffers cached
Mem: 4135700 504028 3631672 0 52240 262880
-/+ buffers/cache: 188908 3946792
Swap: 4192944 156 4192788
Here I've actually got (4135700 - 188908) = 3946792 free
While you can never have too much memory, if you've got less than 50% available for buffers and cache and the total size of the content in the website is more than this amount then your website will slow down.