You might want to read about how Internet works and how packet switching works to add to the explanation here.
But for now, lets assume things are purely between your PC, to a very specific host server that you have the IP address.
The way packet switching works is "try our best" to send packets to the destination.
If for whatever reason the switches/routers between your PC and the server has issues (too much traffic on it, or it dies) it will simply try to route the packet using different route. The same request may ends up coming through multiple different routes just because it happens to be the better route 'at that time'.
Or if the router/switch that your packet arrives to is too busy, that router WILL drop your packets, and send a confirmation to your PC that they dropped it. Your PC will then re-send the packet, hoping it will arrive. (This whole process is completely transparent and you won't see it unless you know what you are doing / looking for)
Different physical link will definitely affect time elapsed, not only because you may be not on the most straight-line route, but also, maybe the straight-line route is highly congested and you may ends up faster using the route going round the world first.
For comparison, imagine you are driving your car from A to B. Same driving speed, same road, you still end up with different time elapsed due to network congestion. Then one day you decided to use different road. The smaller road, the backroad, the highway, you name it. You may arrive faster / slower - you really don't know. You may have an 'average' time, but the time elapsed will still be different.
Hope this helps.
Keltari, if one server is used at all times can't we exclude some factors from the list ? Such as distance, route, device/server itself ? I guess my question really is: what defines the difference in times when source and destination hosts stays the same ? Thank you. – user1634619 – 2013-10-28T02:46:49.470