Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [file-transfer]

308 questions
328
votes
18 answers

Copying a large directory tree locally? cp or rsync?

I have to copy a large directory tree, about 1.8 TB. It's all local. Out of habit I'd use rsync, however I wonder if there's much point, and if I should rather use cp. I'm worried about permissions and uid/gid, since they have to be preserved in the…
Amandasaurus
  • 30,211
  • 62
  • 184
  • 246
129
votes
4 answers

robocopy transfer file and not folder

I'm trying to use robocopy to transfer a single file from one location to another but robocopy seems to think I'm always specifying a folder. Here is an example: robocopy "c:\transfer_this.txt" "z:\transferred.txt" But I get this error…
Bill McKay
119
votes
27 answers

How to copy a large number of files quickly between two servers

I need to transfer a huge amount of mp3s between two serves (Ubuntu). By huge I mean about a million files which are on average 300K. I tried with scp but it would have taken about a week. (about 500 KB/s) If I transfer a single file by HTTP, I get…
nicudotro
  • 1,521
  • 2
  • 13
  • 14
96
votes
11 answers

Transfer 10 TB of files from USA to UK datacenter

I am migrating my server from the USA to the UK from one data center to another. My host said I should be able to achieve 11 megabytes per second. The operating system is Windows Server 2008 at both ends. My average file size is around 100 MB and…
Paul Hinett
  • 1,205
  • 3
  • 11
  • 19
81
votes
11 answers

Transfer 15TB of tiny files

I'm archiving data from one server to another. Initially I started a rsync job. It took 2 weeks for it to build the file list just for 5 TB of data and another week to transfer 1 TB of data. Then I had to kill the job as we need some down time on…
lbanz
  • 1,579
  • 4
  • 18
  • 30
72
votes
8 answers

Quickest way to transfer 55GB of images to new server

I currently have two CentOS servers. I need to know how and what the quickest way would be to "tar" up the images directory and SCP it over? Is that the quickest way that I just suggested, because tarring is taking forever... I ran the command: tar…
Andrew Fashion
  • 1,635
  • 7
  • 22
  • 26
60
votes
18 answers

Favorite rsync tips and tricks

The more I use rsync the more I realise that it's a swiss army knife of file transfer. There are so many options. I recently found out that you can go --remove-source-files and it'll delete a file from the source when it's been copied, which makes…
Amandasaurus
  • 30,211
  • 62
  • 184
  • 246
56
votes
11 answers

Is there an equivalent to ssh-copy-id for Windows?

Is there any equivalent or port of ssh-copy-id available for Windows? That is, is there an easy way to transfer SSH keys from a local machine to a remote server under Windows? In case it helps, I'm using Pageant and Kitty (a Putty alternative)…
Matt V.
  • 837
  • 1
  • 9
  • 12
53
votes
1 answer

rsync - Exclude files that are over a certain size?

I am doing a backup of my desktop to a remote machine. I'm basically doing rsync -a ~ example.com:backup/ However there are loads of large files, e.g. wikipedia dumps etc. Most of the files I care a lot about a small, such as firefox cookie files,…
Amandasaurus
  • 30,211
  • 62
  • 184
  • 246
22
votes
4 answers

Rsync show progress for individual file

I'm using rsync to upload/download a pretty large file (20Gb). I'm aware of --progress option, but what it does in my case is just showing me the result at the end. My problem is that during some unstable connection - I don't know whether it's still…
Salmon
  • 321
  • 1
  • 2
  • 6
21
votes
5 answers

What is the best way to transfer a single large file over a high-speed, high-latency WAN link?

This looks related to this one, but it's somewhat different. There is this WAN link between two company sites, and we need to transfer a single very large file (Oracle dump, ~160 GB). We've got full 100 Mbps bandwidth (tested), but looks like a…
Massimo
  • 68,714
  • 56
  • 196
  • 319
18
votes
2 answers

What a pros and cons of FTP vs SSH for file transfers?

Why to consider setting up an FTP service on a server when files transfers work fine by means of SSH (with Midnight Commander on a Lunux and FileZilla on a Windows client)? What are pros and cons of both? Except of the fact tat FTP is more widely…
Ivan
  • 3,288
  • 19
  • 48
  • 70
17
votes
7 answers

Get list of transferred files from rsync?

I'm currently using rsync on a script that deploys a PHP application from a staging to a production server. Here is how: rsync -rzai --progress --stats --ignore-times --checksum /tmp/app_export/ root@app.com:/var/www/html/app/ This is currently…
Mauro
  • 356
  • 1
  • 3
  • 13
16
votes
10 answers

Easiest way for fast transfer of files between linux servers?

I need to transfer files from one CentOS server to another. Will transfer 5MB files about every 10 minutes. Do not need encryption. What is an easy was for fast transfer of files? Is there something simpler than ftp? Thanks!
Alex L
  • 581
  • 2
  • 5
  • 11
16
votes
4 answers

ASCII vs Binary vs Auto?

I'm transferring files between servers and just started noticing that some of them are getting modified to be one long continuous line as opposed to having the returns and line-breaks they originally had. I'm assuming this has something to do with…
Sampson
  • 520
  • 2
  • 6
  • 18
1
2 3
…
20 21