On a compute cluster, I have several hundred jobs running simultaneously.
They each download a single different file (wget
) from the same remote FTP server, and then process the file and save some computations somewhere.
Problem: If the remote server gets too many download requests from my IP in a short amount of time, it will reject all of my requests and penalize future requests with slow download speed.
Question: Is there a way for me to query the current number of connections with the specific remote server?
My idea is:
Within my embarrassingly-parallel script, I can make a clause to check the current number of simultaneous downloads from the same remote server. If this number is, say, > 20, then the script will wait some time before checking again & trying again.
My current attempt: I am looking into netstat
, lsof
and ss
, but I can't figure it out. I have another terminal downloading from the server, but when I use lsof
or ss
, I don't see the active download in the list. Maybe I am looking wrong somehow?
For example: Terminal A:
wget ftp.remote.server/myfile.txt
And it says "resolving ftp.remote.server... 123.45.678"
And then while it is downloading, I got into another terminal and type:
netstat -n | grep "123.45"
But nothing is found...