Tailing a File over FTP

Question

I am attempting to access large log files on a remote server from my Windows desktop. I only have FTP access to this remote machine, not SSH access.

At the moment, I'm using WinSCP to pull the entire file down over FTP. This means I have to transfer the full file every time. However, given it's a log file I am likely to only need the last few lines.

This is particularly frustrating, as my bandwidth is severely limited, so transferring the entire file takes a few minutes.

If I had shell access this could easily be achieved by using something like tail -100 to obtain the last 100 lines.

I would like to find a solution to perform this via FTP. Note that it does not have to be a continuous tail, just a one-off would be sufficient.

Answer 1

What you want to do, can definitely be done with FTP. Technically it's the same, what any FTP client does, when resuming an interrupted file download.

Though from a user perspective, I do not know, if any FTP client supports an explicit download of only given number of trailing bytes.

But some FTP clients will definitely allow you to download new trailing contents of file that you have previously downloaded.

Particularly with WinSCP, just initiate a log file download. Then, on the Overwrite confirmation prompt, select the Resume (it's in drop down menu of the No button). Note that the option is obviously available, only if the source file is larger than the destination file.

If you really want to download only the last few lines of the log, you can cheat WinSCP, by creating a dummy local file with a size bit smaller than the log file before you initiate download.

You can also easily automate the above trick:

fsutil file createnew mylog.log 100000000
winscp.com /command "open mysession" "get -resume /path_to_log/mylog.log" "exit"

For alternatives to the fsutil, see Quickly create large file on a windows system?

With a more effort you can modify the script to first check log file size and automatically calculate dummy file size few kilobytes smaller than the log size.

Answer 2

I don't know how you can do this using WinSCP, and indeed it may not be possible, but in general, you should be able to use the REST command. See RFC 3659. It is not clear to me whether you can start downloading from late in the file, though, or whether you need to already have received the markers. RFC 3659 indicates this is possible, at least in STREAM mode, though:

STREAM mode transfers with FILE STRUcture may be restarted even though no restart marker has been transferred in addition to the data itself. This is done by using the SIZE command, if needed, in combination with the RESTART (REST) command, and one of the standard file transfer commands.

Answer 3

I didn't hear about any software that supports your needs. But if you are programmer, shou should understand how to write this in Perl.

Here i found example how to get all file. This isn't your resolve, but it help you understand how to do: http://www.perlmonks.org/?node_id=907019

another, help for library Net::FTP : http://search.cpan.org/dist/libnet/Net/FTP.pm

using this you can open ftp connection to server (first example) make authorization (first example)

second URL help you how to get last part of file using methods: ->ascii set transfer mode to ascii because this is logfile ->size ( FILE ) get size of file for counting where you should start ->restart ( WHERE ) set file coursor where from in the file you want start reading ->read ( BUFFER, SIZE [, TIMEOUT ] ) directly read small SIZE part into BUFFER variable

another functions: length(BUFFER) tell you how much data you get. If it's ZERO, you propably reached EOF print ("$BUFFER\n") simple print buffer content to the screen

Notice, this is not completly resolution but a way how to do. Only one reason why I write this all is there probably are no software ready for your task.

If you don't know how to write that program, simply ask on stackoverflow.com , not superuser. There are many programmers that are ready for help.

If it's possible, check your ftp site supports download resuming. Whithout this reading end part of file is impossible.

Another partial resolution may be log file rotation, one file a day or a hour. this depends how fast log file grows.