When I first got high speed access many sites on server data at 56kbs (high speed modem rates). However, I could browse multiple sites at the same time without slowing down any site. We now expect, and usually get much faster response.
Due to windowing it is unusual to get the full speed out of any high speed connection. Network round trip times have an inverse relation to throughput. All things being equal the close a site is the faster it will be. Increasing the window size may increase the download speed, but there are window size limits.
Network routing can greatly increase the network distance to a site. Usually I access sites in north-east USA via a fairly direct route that goes 100 miles north-east to Montreal then south to the site. Other times the route from Montreal goes to the west coast along the way. This significantly slows the transfers. Fortunately, most of these slow downs are not overly visible when viewing a web site. File transfer speeds are much slower with this routing.
That being said, if you only transfer one file at a time you will have unused (wasted?) bandwidth. You may find that you can download two or more files at the same time without a significant decrease in download speed. As you add more connections, the speed will begin to slow down as you begin to fully utilize the available bandwidth.
Having a high speed connection reduces the connection latency at your end. This will marginally increase your transfer speed. Google.com has a latency of 25ms for me, while another site has a latency of 90ms. All things being equal, I would expect better file transfer speeds from Google.