0

On a compute cluster, I have several hundred jobs running simultaneously.

They each download a single different file (wget) from the same remote FTP server, and then process the file and save some computations somewhere.

Problem: If the remote server gets too many download requests from my IP in a short amount of time, it will reject all of my requests and penalize future requests with slow download speed.

Question: Is there a way for me to query the current number of connections with the specific remote server?

My idea is:

Within my embarrassingly-parallel script, I can make a clause to check the current number of simultaneous downloads from the same remote server. If this number is, say, > 20, then the script will wait some time before checking again & trying again.

My current attempt: I am looking into netstat, lsof and ss, but I can't figure it out. I have another terminal downloading from the server, but when I use lsof or ss, I don't see the active download in the list. Maybe I am looking wrong somehow?

For example: Terminal A:

wget  ftp.remote.server/myfile.txt

And it says "resolving ftp.remote.server... 123.45.678"

And then while it is downloading, I got into another terminal and type:

netstat -n  | grep "123.45"

But nothing is found...

cmo
  • 109
  • 2
  • 1
    This sounds like a ridiculously round-about way of avoiding keeping a simple counter in whatever your script does and not exceed N. It also sounds like you're looking for help to press as hard against an established configuration that's trying to limit abuse and excessive bandwidth usage. Netstat, lsof, and ss should all give you this information. If you can't figure it out, you should spend more time on the man pages. –  Aug 13 '16 at 02:14
  • Why not set up a proxy server and route your download requests though that. Then you have a single application that can reuse connections for multiple sequential downloads and from which you can enforce your download policies: be it in the proxy itself (for instance with a squid delay pool or an ACL) or traffic shaping at the IP level. – HBruijn Aug 13 '16 at 05:18

1 Answers1

1

If that's your ftp server, you need to fix it. If it's not, find a way to stop abusing it. Find out what the sites limits are and change your code to stay within those limits. Your code can be "self aware", i.e. you don't need to check netstat, etc... have your code keep tabs on it's own behaviour.

You can also connect via ftp and download multiple files in serial in a persistent connection. This may be preferred by the ftp host and is the way ftp was designed to be used.

Another option may be to mirror the ftp site to a local server and download from there.

I don't know why netstat -n and friends don't work for you. They work for me.

$ netstat -n | grep 192.168.1.254 
tcp        0      0 192.168.1.122:38868     192.168.1.254:80        ESTABLISHED
Ryan Babchishin
  • 6,160
  • 2
  • 16
  • 36