226

There's been a number of questions regarding disk cloning tools and dd has been suggested at least once. I've already considered using dd myself, mainly because ease of use, and that it's readily available on pretty much all bootable Linux distributions.

What is the best way to use dd for cloning a disk? I did a quick Google search, and the first result was an apparent failed attempt. Is there anything I need to do after using dd, i.e. is there anything that CAN'T be read using dd?

Skyhawk
  • 14,149
  • 3
  • 52
  • 95
falstro
  • 2,675
  • 3
  • 18
  • 10
  • I'm aware how dd works, my question was more in the direction of any known problems related to dd when cloning disks (as described by the link), maybe this wasn't very clear. What his answer contains and yours doesn't is "I've never once had any problems with it". I did upvote your answer too, as you did definitely present some interesting points (I like the one about no progress indication). – falstro May 06 '09 at 20:25
  • 1
    Looks like you got the Spolsky Bump: http://www.joelonsoftware.com/items/2009/05/29.html – Kyle Cronin May 29 '09 at 18:04
  • didn't see this on here when I asked (and answered) a similar question on superuser - http://superuser.com/questions/11453/how-to-move-from-one-drive-to-another-thats-a-different-size – warren Aug 31 '09 at 05:58
  • 7
    It's ironic that Joel linked to the question as a good example of server-fault, although none of the answers were good. There was not one answer among 25 (excluding comments) with the right `dd` options for skipping bad blocks - which is essential when cloning disks for recovery. I added a better answer, which can clone disks having bad blocks: `dd if=/dev/sda of=/dev/sdb bs=4096 conv=sync,noerror` – Sam Watkins Nov 14 '14 at 06:52
  • I think dd restore might "fail" if talking about drive geometry dependent file systems and restore is done on non identical hard drives? I experienced some failures on dd restore, and I think this was the problem in my case. – Marco Jun 08 '17 at 13:03
  • 1
    This method will also save the unallocated part of a disk, if any. If you don't need it, you can tell it where to stop (`count` parameter for `dd`, or pipe the data through `head -c …` for anything else; but I'm not sure how to find the exact number of bytes yet, or at least a good upper bound) – Display Name Jun 18 '18 at 06:46
  • 1
    Also note than most of the times, `dd` program is unnecessary. You can use `head` or `cat` for reading from a block device, and write to a regular file using a redirect. And you can write to a block device with `tee`. Then you won't have to guess a good block size, and it's likely going to be faster too. And if the source disk is in bad condition, then `dd` isn't a really good option too, use `ddrescue` to save data from damaged disks instead. – Display Name Jun 18 '18 at 07:43
  • Also take a look at this question: https://superuser.com/q/1050894/910769 Two more resources to be considered: https://tecmint.com/clone-linux-partitions and https://cyberciti.biz/faq/unix-linux-dd-create-make-disk-image-commands – Cadoiz Jan 19 '21 at 12:50

29 Answers29

203

dd is most certainly the best cloning tool, it will create a 100% replica simply by using the following command. I've never once had any problems with it.

dd if=/dev/sda of=/dev/sdb bs=32M

Be aware that while cloning every byte, you should not use this on a drive or partition that is being used. Especially applications like databases can't cope with this very well and you might end up with corrupted data.

Izzy
  • 786
  • 2
  • 8
  • 29
Adam Gibbins
  • 7,147
  • 2
  • 28
  • 42
  • 13
    Of course, as long as /dev/sdb is at least as large as /dev/sda... – Eddie May 05 '09 at 21:09
  • 28
    add a "bs=100M conv=notrunc" and it's much faster in my experience. – Tim Williscroft May 06 '09 at 22:14
  • 7
    @Eddie - and of course the partition table will be copied too, so if sdb is larger you'll have unused space at the end. – Alnitak May 07 '09 at 08:41
  • 4
    @Tim, What does conv=notrunc do exactly? – Svish May 19 '09 at 13:19
  • 8
    notrunc means (according to the man page) : do not trunc the output file. I don't understand how it can be faster. You can follow the progression of the operation with : # dd if=/dev/sda of=/dev/sdb & pid=$! # kill -USR1 $pid; sleep 1; kill $pid – Emmanuel BERNAT May 29 '09 at 19:09
  • 3
    Play with bs a little when you start. I've encountered systems where any bs > 4k slowed the process down for some reason. If both drives are internal (ide/sata) it's probably not an issue, but if there's a network share or a USB disk involved, take care on the block size. – Michael Kohne May 29 '09 at 19:30
  • 144
    just be very careful with the 'i' and 'o' letters... – bandi May 29 '09 at 21:52
  • 2
    Oh yes. I've used dd a great many times and it always gives me shivers, when I think just how will that feel when I'll eventually get those "i" and "o" wrong.. – shylent May 30 '09 at 15:22
  • 25
    Nobody seems to know this trick... dd is an asymmetrical copying program, meaning it will read first, then write, then back. You can pipe dd to itself and force it to perform the copy symmetrically, like this: `dd if=/dev/sda | dd of=/dev/sdb`. In my tests, running the command without the pipe gave me a throughput of ~112kb/s. With the pipe, I got ~235kb/s. I've never experienced any issues with this method. Good luck! – mistiry Oct 19 '10 at 20:50
  • 8
    Added bs=32M, to save future readers the agony of cloning a disk with the obsolete default block size. – Skyhawk May 10 '12 at 00:02
  • 15
    @Mistiry, that's not the meaning of the word symmetric. – psusi May 10 '12 at 01:53
  • 6
    @bandi Time seems to slow to a crawl in the second before you hit enter.. – Rag Mar 27 '13 at 19:10
  • 10
    What is the advantage of using `dd` compared to `cat /dev/sda > /dev/sdb` or `pv /dev/sda > /dev/sdb`? – Marco Apr 12 '13 at 10:59
  • @Marco - `dd` allows you to specify a block size (http://stackoverflow.com/questions/150697/is-dd-better-than-cat) – Ben Jul 22 '13 at 02:48
  • 2
    @bandi I have scripted dd so that before executing it displays a prompt saying "Are you absolutely sure about if and of?". It requires to type "yes" in order to continue. Of course this is after I got bummed out of 3Tb of perfectly good data. – dlyk1988 Sep 09 '13 at 12:57
  • If you're restoring a partition, don't forget to run `partprobe -s`, or the OS won't be informed about changes to the partition table. – Gui Ambros Sep 02 '14 at 04:31
  • 6
    it's very important to use the options `conv=sync,noerror` so that it skips bad blocks properly... unless you happen to be certain that your disk has no bad blocks, which is not true in general. – Sam Watkins Nov 13 '14 at 02:49
  • @Mistiry, try again with a block size bigger than the default 512 bytes. It will be even faster and no pipe needed. Add this argument: `bs=1M`. I tried the pipe method and it did not change the transfer rate. – drzymala Feb 23 '20 at 13:24
  • 2
    Add `status=progress`. This parameter will show you where `dd` is at in the cloning process. Like so: `dd if=/dev/sda of=/dev/sdb bs=32M status=progress` – Joe Jan 07 '22 at 21:27
135

To save space, you can compress data produced by dd with gzip, e.g.:

dd if=/dev/hdb | gzip -c  > /image.img.gz

You can restore your disk with:

gunzip -c /image.img.gz | dd of=/dev/hdb

To save even more space, defragment the drive/partition you wish to clone beforehand (if appropriate), then zero-out all the remaining unused space, making it easier for gzip to compress:

mkdir /mnt/hdb
mount /dev/hdb /mnt/hdb
dd if=/dev/zero of=/mnt/hdb/zero

Wait a bit, dd will eventually fail with a "disk full" message, then:

rm /mnt/hdb/zero
umount /mnt/hdb
dd if=/dev/hdb | gzip -c  > /image.img.gz

Also, you can get a dd process running in the background to report status by sending it a signal with the kill command, e.g.:

dd if=/dev/hdb of=/image.img &
kill -SIGUSR1 1234

Check your system - the above command is for Linux, OSX and BSD dd commands differ in the signals they accept (OSX uses SIGINFO - you can press Ctrl+T to report the status).

Fidel
  • 363
  • 1
  • 4
  • 18
David Hicks
  • 2,258
  • 2
  • 15
  • 12
  • 2
    Does this also work with "modern" fs such a BTRFS, NILFS, [whatever you can dream of] ? – Steve Schnepp Jun 26 '09 at 14:41
  • DD works on block devices, a level of abstraction lower than the file system, so it should, yes. I haven't actually tried it, though. Hmm, NILFS looks interesting, I'll have to take a look at that. – David Hicks Jun 28 '09 at 22:47
  • Sorry, just checked out NILFS' homepage and realised what you might have meant - can you use DD to copy a snapshot from a NILFS filesystem? I don't know, but it'd be interesting to find out. – David Hicks Jun 28 '09 at 23:01
  • 4
    +1 for the `kill -SIGUSR1 %1`, and the OSX dd command happily accepts SIGUSR1... super useful, thanks! – stuartc Nov 03 '12 at 10:17
  • 1
    +1 for `Kill -SIGUSR1 1234` I was looking for that. – John K. N. Dec 12 '12 at 13:17
  • 3
    Should it be: `dd if=/dev/hdb | gzip -c > /image.img.gz` ? – Mike Causer Oct 07 '15 at 10:18
  • You could also use `status=progress` to have it report its progress on stderr while it's cloning. – haliphax Sep 29 '17 at 00:44
  • 1
    Pay attention: Using that `gzip` in between (even with --fast) ramped my cpu to 100% and slowed down transmission speed by a factor of at least 7. For real HDD imaging, it could be worth considering to use at least a parallel implementation: https://superuser.com/questions/400634/fastest-gzip-utility – Cadoiz Jan 19 '21 at 12:55
47

CAUTION: dd'ing a live filesystem can corrupt files. The reason is simple, it has no understanding of the filesystem activity that may be going on, and makes no attempt to mitigate it. If a write is partially underway, you will get a partial write. This is usually not good for things, and generally fatal for databases. Moreover, if you screw up the typo-prone if and of parameters, woe unto you. In most cases, rsync is an equally effective tool written after the advent of multitasking, and will provide consistent views of individual files.

However, DD should accurately capture the bit state of an unmounted drive. Bootloaders, llvm volumes, partition UUIDs and labels, etc. Just make sure that you have a drive capable of mirroring the target drive bit for bit.

jldugger
  • 14,122
  • 19
  • 73
  • 129
  • You can always use 'sync' to sync the file system to the hdd before running dd. – LiraNuna Jul 17 '09 at 20:08
  • 8
    I suspect that `sync` is not the answer to file corruption problems. What happens if a deamon or something writes more files after the `sync`, during the `dd` operation? – Deleted Aug 13 '09 at 10:47
  • 6
    It's a good idea to umount the drive first (or remount as read-only) but it's not always possible – Alex Bolotov Aug 22 '09 at 09:36
  • 1
    In which case, you use rsync and let it do file handle magic to get a consistent file and let Copy On Write semantics handle the incoming writes. – jldugger Aug 22 '09 at 21:18
  • I'm not sure what having multiple CPUs/cores has to do with using rsync to copy files? – davr Mar 19 '10 at 20:22
  • Apologies, I intended to refer to the concept of multitasking. – jldugger Mar 19 '10 at 21:11
  • 4
    I'd like to add that running dd on a mounted filesystem WILL NOT CORRUPT the files on the mounted filesystem, but what is meant here is that the copy of the filesystem will necessarily be in a known good state. – 3molo Apr 12 '13 at 10:11
  • 1
    Using `rsync` will ensure that the *internal data* in the destination filesystem is consistent. It *will not* ensure that the data in the files is consistent — to do that, you would need to lock the files and any programs that write to the files would need to respect these locks. – Martin Geisler Dec 16 '14 at 22:37
  • This is not an answer. – BonsaiOak Mar 21 '18 at 13:28
32

When using dd to clone a disk which may contain bad sectors, use conv=noerror,sync to ensure that it doesn't stop when it encounters an error, and fills in the missing sector(s) with null bytes. This is usually the first step I take if trying to recover from a failed or failing disk - get a copy before doing any recovery attempts, and then do recovery on the good (cloned) disk. I leave it to the recovery tool to cope with any blank sectors that couldn't be copied.

Also, you may find dd's speed can be affected by the bs (block size) setting. I usually try bs=32768, but you might like to test it on your own systems to see what works the fastest for you. (This assumes that you don't need to use a specific block size for another reason, e.g. if you're writing to a tape.)

Cadoiz
  • 135
  • 5
TimB
  • 1,430
  • 2
  • 15
  • 19
  • 16
    If you have a disk with bad sectors, you really should be using 'ddrescue' instead of dd. It's much more efficient, and has a much better chance of recovering more data. (Don't get it confused with dd_rescue, which is not as good) – davr Mar 19 '10 at 20:27
  • 6
    should not use a large block size if attempting to skip bad blocks, or it will skip too much. 4096 is large enough. – Sam Watkins Nov 13 '14 at 02:54
  • Consider this for discussion of the fastest `bs` settings: https://serverfault.com/questions/147935/how-to-determine-the-best-byte-size-for-the-dd-command – Cadoiz Jan 19 '21 at 13:00
19

To clone a disk, all you really need to do is specify the input and output to dd:

dd if=/dev/hdb of=/image.img

Of course, make sure that you have proper permissions to read directly from /dev/hdb (I'd recommend running as root), and that /dev/hdb isn't mounted (you don't want to copy while the disk is being changed - mounting as read-only is also acceptable). Once complete, image.img will be a byte-for-byte clone of the entire disk.

There are a few drawbacks to using dd to clone disks. First, dd will copy your entire disk, even empty space, and if done on a large disk can result in an extremely large image file. Second, dd provides absolutely no progress indications, which can be frustrating because the copy takes a long time. Third, if you copy this image to other drives (again, using dd), they must be as large or larger than the original disk, yet you won't be able to use any additional space you may have on the target disk until you resize your partitions.

You can also do a direct disk-to-disk copy:

dd if=/dev/hdb of=/dev/hdc

but you're still subject to the above limitations regarding free space.

As far as issues or gotchas go, dd, for the most part, does an excellent job. However, a while ago I had a hard drive that was about to die, so I used dd to try and copy what information I could off it before it died completely. It was then learned that dd doesn't handle read errors very well - there were several sectors on the disk that dd couldn't read, causing dd to give up and stop the copy. At the time I couldn't find a way to tell dd to continue despite encountering a read error (though it appears as though it does have that setting), so I spent quite a bit of time manually specifying skip and seek to hop over the unreadable sections.

I spent some time researching solutions to this problem (after I had completed the task) and I found a program called ddrescue, which, according to the site, operates like dd but continues reading even if it encounters an error. I've never actually used the program, but it's worth considering, especially if the disk you're copying from is old, which can have bad sectors even if the system appears fine.

Kyle Cronin
  • 1,218
  • 14
  • 26
  • 1
    You can actually also use a read-only mount. A filesystem can be remounted with: mount -o remount,ro /path/to/device – Paul de Vrieze May 30 '09 at 15:11
  • Good point, I added a note in my answer about that. – Kyle Cronin May 30 '09 at 18:07
  • I used ddrescue to scrape data off a dying hard drive, and can confirm that it's awesome. – sleske Jun 23 '09 at 00:45
  • 7
    **...dd provides absolutely no progress indications...** - well this is not true - there is kinda tricky way how to show progress - you have to find out pid of dd process ('ps -a | grep dd') and then send signal USR1 to this process - 'kill -USR1 '(without <>) which force dd to show progress information. – Michal Bernhard Jun 19 '11 at 15:51
  • 4
    "several sectors on the disk that dd couldn't read": I think that `conv=sync,noerror` would help. – Gauthier Nov 30 '11 at 10:00
  • 1
    @MichalBernhard I just registered to upvote that comment. SO awesome. – kaoD Oct 26 '12 at 17:11
  • 1
    For status info on OS X, use `kill -SIGINFO ` since `-USR1` on OS X will kill dd. Thanks for the tip @MichalBernhard! – Adam Franco Oct 15 '13 at 15:35
  • 2
    The `conv=sync,noerror` options are essential, they allow dd to skip bad blocks and zero them out in the image so things are aligned correctly. Props to the very few people who commented something about that. – Sam Watkins Nov 13 '14 at 02:51
  • 1
    GNU `ddrescue` provides progress indicator without any special options, and you can stop the copy and resume where you left off. – endolith Feb 25 '15 at 02:46
  • 6
    A less tricky way to get progress with dd is to add the option `status=progress` – James Jul 30 '17 at 16:49
  • @michael you can also change the dd command to directly obtain the pid: `dd if=/dev/zero of=/dev/null& pid=$!`. – Cadoiz Jan 19 '21 at 13:06
  • @james The option `status=progress` is not supported everywhere, but a good call if so. – Cadoiz Jan 19 '21 at 13:06
14

If the source drive is damaged at all, you'll have more luck using dd_rhelp with dd_rescue (my personal preference) or GNU ddrescue.

The reason behind this is that, on read errors, dd keeps trying and trying and trying - potentially waiting for a long time for timeouts to occur. dd_rescue does smart things like reading up to an error, then picking a spot further on on the disk and reading backwards to the last error, and dd_rhelp is basically a dd_rescue session manager - cleverly starting and resuming dd_rescue runs to make it quicker again.

The end result of dd_rhelp is maximum data recovered in minimum time. If you leave dd_rhelp running, in the end it does the exact same job as dd in the same time. However, if dd encountered read errors at byte 100 of your 100Gb disk, you'd have to wait a long time to recover the other 9,999,900 bytes*, whereas dd_rhelp+dd_rescue would recover the bulk of the data much faster.

tshepang
  • 377
  • 2
  • 12
Ben Williams
  • 739
  • 1
  • 7
  • 17
9

The source disk must not have any mounted filesystems. As a user able to read the block device (root works), run 'dd if=/dev/sda ....'

Now, one of the neat things here is that you're creating a stream of bytes... and you can do a lot with that: compress it, send it over the network, chunk it into smaller blobs, etc.

For instance:

dd if=/dev/sda | ssh user@backupserver "cat > backup.img"

But more powerfully:

dd if=/dev/sda | pv -c | gzip | ssh user@backupserver "split -b 2048m -d - backup-`hostname -s`.img.gz"

The above copies a compressed image of the source harddrive to a remote system, where it stores it in numbered 2G chunks using the source host's name while keeping you updated on progress.

Note that depending on the size of disk, speed of cpu on source, speed of cpu on destination, speed of network, etc. You may want to skip compression, or do the compression on the remote side, or enable ssh's compression.

retracile
  • 1,260
  • 7
  • 10
7
dd if=/dev/sda of=/dev/sdb bs=4096 conv=sync,noerror

This will copy the disk, and skip blocks with errors, which is very important.

These are the basic and essential options for using dd to clone or rescue a disk.

I did not want to post another answer, but there were no good answers with the essential "conv=sync,noerror" options among the 25 already posted.

Sam Watkins
  • 272
  • 3
  • 7
  • 1
    Actually it was mentioned before, in [one answer](http://serverfault.com/a/5584/126632) and at least two comments. – Michael Hampton Nov 13 '14 at 02:51
  • 1
    @Michael the answer you linked does not contain a full example command, and has other faults - it's not a good idea to use large block size with `conv=sync,noerror` as it will skip too much data for each bad block. These options are essential for "recovery cloning", and it's no good having to search the comments for them. The most popular answer is adequate if the disks have no bad blocks, e.g. for cloning a pristine disk, but not for recovery. – Sam Watkins Nov 14 '14 at 06:32
  • @Michael the example I gave is a command I have used several times for professional disk recovery. While there are other tools which might do a slightly better job, the example I gave is better at disk recovery cloning than every other `dd` example here, while also being suitable for cloning an error-free disk. Therefore, I consider my answer to be the best here on how to "use DD for disk cloning". I did not add info on monitoring progress, compression, etc., because I wanted to keep it simple, and focus on providing a short answer which gets the basics right. – Sam Watkins Nov 14 '14 at 06:38
6

To clone a disk, all you really need to do is specify the input and output to dd:

dd if=/dev/hdb of=hdb.img

Of course, make sure that you have proper permissions to read directly from /dev/hdb (I'd recommend running as root), and that /dev/hdb isn't mounted (you don't want to copy while the disk is being changed). Once complete, hdb.img will be a byte-for-byte clone of the entire disk.

There are a few drawbacks to using dd to clone disks. First, dd will copy your entire disk, even empty space, and if done on a large disk can result in an extremely large image file. Second, dd provides absolutely no progress indications, which can be frustrating because the copy takes a long time. Third, if you copy this image to other drives (again, using dd), they must be as large or larger than the original disk, yet you won't be able to use any additional space you may have on the target disk until you resize your partitions.

You can also do a direct disk-to-disk copy:

dd if=/dev/hdb of=/dev/hdc

but you're still subject to the above limitations regarding free space.

The first drawback can be resolved by gzipping the data as you make the copy. For example:

dd if=/dev/hdb | gzip -9 > hdb.img.gz

The second drawback can be resolved by using the pipeview (pv) tool. For example:

dd if=/dev/hdb | (pv -s `fdisk -l /dev/hdb | grep -o '[0-9]*\{1\} MB' | awk '{print $1}'`m) | cat > hdb.img

I know of no way to overcome the third drawback.

Additionally, you can speed up the copy time by telling dd to work with larger chunks of data. For example:

dd if=/dev/hdb of=hdb.img bs=1024
James Sumners
  • 493
  • 3
  • 7
  • 17
  • 1
    You already told the way to overcome the third drawback...resize the partitions. Enlarging a partition is generally a safe and fast operation (versus shrinking or moving, which is slow and more dangerous since it's moving data around). – davr Mar 19 '10 at 20:28
  • gzipping will not work with a disk that has been used for some time, as it will be filled with either current or deleted data. gzip will only work if the empty space is zero'ed, which is only the case with a brand new disk. – Tozz Feb 22 '12 at 20:51
  • 3
    @Tozz: You can improve the compressibility of a filesystem image by filling the filesystem with a file filled with 0's, syncing it to disk, then deleting it. `dd if=/dev/zero bs=1M of=/balloon; sync; rm /balloon` (Modulo extra intelligence in the filesystem layer.) – retracile Jul 26 '12 at 14:15
5

Another nice thing you can do with dd and rescue disks is copy data over the network:

remote_machine$ nc -l -p 12345

local_machine$ dd if=/dev/sda | nc remote_machine 12345

You can stick gzip in both these pipelines if the network is not local. For progress, use pv. To make local_machine's netcat quit after it's done copying, you might add -w 5 or something.

user5692
  • 77
  • 1
  • 9
    This is not quite correct. The 'remote_machine' command is missing something, such as `> disk_backup.img` or `|dd of=/dev/sdb` or something else, depending on what you want to do. I'm guessing you don't want to dump a disk image to stdout. – davr Mar 19 '10 at 20:26
  • 1
    And throw in gzip on both ends to further minimize the sent data. – 3molo Apr 12 '13 at 10:12
4

This is kind of a cheap hack, but it's a quick and dirty way to monitor your DD process.

Run your dd command. Open a new shell and do a ps awx to find your dd process' PID. Now in the new shell type watch -n 10 kill -USR1 {pid of your DD process}

This will do nothing in the watch output window, but back in the original DD shell, DD will start outputting status reports every 10 seconds. You can change the -n 10 in the watch command to any other time frame of course.

Tachyon

  • OS X doesn't have `watch` available and `-USR1` kills dd. The following command works though: `while [ true ]; do killall -INFO dd; sleep 30; done` – Adam Franco Aug 09 '11 at 23:04
  • I don't think this is very practical for a novice, they may be better served with the `pv` command. – Robbie Mckennie Jul 19 '13 at 09:36
  • I found that you can also send a SIGINFO useing CTRL-T in dd. It's easier than the while loop and the OSX 10.6 cd I have does not have killall. Learned this from http://en.wikipedia.org/wiki/Unix_signal#Sending_signals – Citizen Kepler Sep 02 '14 at 16:34
  • Nowadays, there's `status=progress` (GNU Coreutils version 8.24 or above) – fuero Apr 27 '21 at 08:56
4

Keep in mind that dd makes an exact copy, including all the blank space.

That means:

  1. 2nd drive must be at least as big as first
  2. If 2nd drive is larger, extra space will be wasted (filesystem can be expanded mind you)
  3. If the source drive is not full, dd will waste alot of time copying blank space.
  4. You can copy either the entire drive, or a single partition this way.
  5. If this is a bootable drive, I'm pretty sure you need to install the bootloader after using dd

Hope that is helpful

Brent
  • 22,219
  • 19
  • 68
  • 102
  • 9
    If you're cloning the whole hard disk, you're also cloning the boot loader. – Cristian Ciupitu May 29 '09 at 22:16
  • welll, just a thouhgt, but couldn't u just use gparted to resive the partition/disk being copied down to whatever is used- then drop dd? Assuming it's a onetime image it should mitigate this issue. – bbqchickenrobot Mar 24 '12 at 19:40
3

Another grand feature is copying MBRs, partition tables and boot records.

Just

dd if=/dev/sda of=parttable bs=512 count=1

and the other direction around when you're writing it. Polish with fdisk after.

You feel much more safe when you have your partition table backed up.

Also, it makes migrating to another hard drive (while changing partion structure) a joy.

alamar
  • 199
  • 5
3

How to copy using dd (in this case to a remote machine, but the same principle applies to a local copy) which shows progress.

It works by storing the pid via file descriptor 3 in /tmp/pid, which is then used for the subsequent kills with signal USR1. A wrinkle was to filter the output of the progress on stderr to only one line via filtering stderr through a subshell.

(dd bs=1M if=$lv-snapshot & echo $! >&3 ) 3>/tmp/pid  2> >(grep 'copied' 1>&2) | gzip --fast | ssh $DEST "gzip -d | dd bs=1M of=$lv" &
# Need this sleep to give the above time to run
sleep 1
PID=$(</tmp/pid)

while kill -0 $PID; do
  kill -USR1 $PID
  sleep 5
done
  • Not really relevant to the question, but its a neat shell trick using a sub-shell and high (higher than stderr) file descriptors to convey data out of it, +1 – falstro Jul 31 '13 at 06:10
  • I was referring to this page myself for different dd options when cloning disks, so it seemed a suitable place at the time to put the end result of what I used for cloning, especially since I thought it was rather neat myself :) – Edward Groenendaal Jul 31 '13 at 23:29
  • echo $! > /tmp/pid – mug896 May 29 '20 at 18:54
3

For future reference it might be of interest to check out ddrescue. It has saved my day a couple of times.

Anders Hansson
  • 179
  • 1
  • 4
3

You could actually try something like this

dd if=/dev/sda2 of=/dev/sdb2 bs=4096 conv=sync,noerror

to skip all errors and have exact clone of a partition or hard drive

Sam Watkins
  • 272
  • 3
  • 7
  • It should be conv=sync,noerror The sync option is needed or else blocks with errors will be removed rather than copied as zeros. – Sam Watkins Nov 13 '14 at 02:35
3

The most info was described in previous inserted recipies, but not all was described.

Under linux you can clone hard drive or partition by dd command. Attention, when you'll make a mistake, you will lost all your data.

At first, destination should not be in use, secondly source should be not used, or remounted into read only mode. Otherwise copy will be damaged. If remounting is impossible, please make bootable drive (hdd/ssd/pendrive) any linux live distro. I prever knoppix, but this is your choose. If it is possible, you can boot or change system level into 1, for single user mode, or you can directly reboot system into single user mode, it is distro depended. If you'll clone only one partition, this partition should be unmounted or remounted into RO:

umount /mountpoint_or_device

or

remount -o,ro /mountpoint_or_device

If you want clone entire hard drive, you must umount or remount all partitions.

You must identify source and destination device. please look at the dmesg, here is stored all needed info about device, with vendor etc. alternatively identifying can be based on device size, if it is different. Next, destination should be the same or bigger than source. you must calculate source, for example: fdisk -l /dev/sda except partition geometry (there can be GPT), you will fetch: 1. total disk size wigh GB and bytes 2. historical geometry and total sector number, very important info 3. block size in bytes, usually it is 512.

for example:

# fdisk -l /dev/sda

Disk /dev/sda: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000f1d1e

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048    40136703    20067328   83  Linux
/dev/sda2        40138750    41940991      901121    5  Extended
/dev/sda5        40138752    41940991      901120   82  Linux swap /     Solaris

next let's try bigger than 512 divider, we have 41943040 physical sectors:

41943040 / 256 = 163840 , very good, we can do bulk copy of 256 sectors. can we more? let's try: 41943040 / 1024 = 40960 , I think this is enough, we will select this one. Let's count size of sector group: 512(sector size) * 1024 = 524288 bytes eq 512K . Then we can use parameter bs=512K or less, but divide this by 2^x. For modern hard drives with big internal cache, this is practical enough. for older drives with much smaller cache, value 32K or less is enough.

Then after preparation we can do a copy: dd if=/dev/source_devide of=/dev/destination_device bs=32K and copy will be done. Pay attention, any mistake will overwrite your importand data. On destination all will be overwritten.

If you try rescue data on damaged source disk, better use native sector size, usually this is 512 bytes, and add option conv=notrunc . otherwise holes in source dropped by bad sectors will be joined by sector shifting on destination. This will damage copy with few chance for repair. then command will be:

dd if=/dev/source of=/dev/destination bs=512 conv=notrunc  

, and wait long time when drive and system will give up and will walk sector by sector to the end.

dd is usefull tool for moving partition into new place. Simply create partition, make dd to new partition (this can be bigger, much bigger), and if it is possible, expand copied file system for filling all new partition, ext3/ext4/xfs/zfs/btrfs have this facility. Finally you must change /etc/fstab , then umount/mount if it is possible, or reboot system.

Of course you can clone any type of partition. dd command doesn't look into file system type, it do nothing with its structure. then this command can be usable for cloning NTFS or other partition types.

There is any trick. When you didn't set of parameter, then dd will put output into its stdout. then you can make compressed raw copy of disk or partition, for example:

dd if=/dev/sda bs=512 | gzip >/any/place/computerOne_sda.gz

Of course this should be done offline. you can restore this by:

zcat /any/place/computerOne_sda.gz| dd of=/dev/sda bs=512   

, then all sda hard drive will be overwritten by this backup, and all current data will be lost. You can do this also with NTFS windows partition and hard drive used by this. Of course you can use other compression command, depended by your choose.

Znik
  • 338
  • 1
  • 3
  • 12
2

dd does provide progress information - well most versions in linux. I've seen some which don't but don't recall the unix flavour.

The man page says: Sending a USR1 signal to a running ‘dd’ process makes it print I/O statistics to standard error and then resume copying.

I use this feature regularly.

Steven
  • 3,009
  • 18
  • 18
2

Someone had to say this: give Clonezilla a try (http:// clonezilla.org/)

What do you get? To copy only used parts of the filesystem. Clonezilla uses dd, grub, sfdisk, parted, partimage, ntfsclone, and/or partclone. Depending on the options you choose.

Decent documentation can be found at: http:// clonezilla.org/clonezilla-live/doc/

  • I found the documentation a little rough, and cloning a linux PATA drive to a SATA drive did Not leave me with something I could boot (yet). But much faster to same result as dd, and it worked great for my laptop drive upgrades. – jbdavid May 30 '09 at 00:43
1

You can create a compressed image file of the partition (or disk) on the fly using bzip2 or gzip instead of dd. This is nice for storing away images in removable media:

bzip2 -c /dev/sdaX >imagefile.bz2
or
gzip -c /dev/sdaX >imagefile.gz

If the disk has been heavily used before, you can enhance the compression by filling all unused space with zeros before the imaging:

mkdir /mnt/mymountpoint
mount /dev/sdaX /mnt/mymountpoint
cat /dev/zero >/mnt/mymountpoint/dummyfile.bin
(Wait for it to end with a "disk full" error)
rm /mnt/mymountpoint/dummyfile.bin
umount /mnt/mymountpoint

To restore the image into another disk, all you have to do is:

bzcat imagefile.bz2 >/dev/sdbY
or
zcat imagefile.gz >/dev/sdbY
JCCyC
  • 670
  • 2
  • 13
  • 24
0

To compress the image using gzip, I use David Hick's excellent answer.

I'd like to post the equivalent using XZ.

The benefit is that I can then use 7-Zip to inspect the contents of the image without decompressing the entire archive. That's made possible by the fact that xz supports random seeking, and that 7-Zip can interpret many file system types.

To capture the image:

dd if=/dev/sda status=progress | xz > image.img.xz

To restore the image:

xzcat image.img.xz | dd of=/dev/sda status=progress

If you are running an old version of dd, do the following to see progress:

Alt + F2 (to open another terminal)

watch -n 1 "kill -USR1 `pidof dd`"

Alt + F1 (to change back to the original terminal)

It should now display progress.
Fidel
  • 363
  • 1
  • 4
  • 18
0

To compress the image using gzip, I use David Hick's excellent answer.

I'd like to post the equivalent using Zstandard, which is very fast.

Unfortunately the output file can't be inspected with 7-Zip.

To capture the image:

dd if=/dev/sda status=progress | zstd > image.img.zst

To restore the image:

zstdcat image.img.zst | dd of=/dev/sda status=progress
Fidel
  • 363
  • 1
  • 4
  • 18
0

For some reason, dd fails when imaging CDs with audio tracks. You need to use cdrdao or something similar to get an image + TOC file.

Matt
  • 101
  • 2
0

A note on speed: in my experience dd is twice as fast if you specify bs=1024 instead of the default bs=512. Using an even larger block size gives no noticeable speedup over bs=1024.

  • 4
    disk clusters are generally around 4k now, so using 4096 is probably a good option, and even 8192 if you want to read 2 clusters at a time. Don't go too big though, as you run into fragmented memory problems – user4767 May 29 '09 at 19:32
0

One thing you must be aware of when dd-ing a full disk is that doing so will overwrite the master boot record of the receiving disk. This contains the partition table and other vital information. If the new disk is not the same as the old disk, this can create all sorts of tables. Copying over partitions is generally safer (and swap partitions do not have to be copied over)

0

I've been out of the admin role for many years now, but I know that 'dd' is up to the job. I used this technique regularly in the late 80s on Sun Sparc and 386i computers. I had one client order over 30 386i systems running CAD software that was distributed on multiple QIC tapes.

We installed on the first computer, configured the app, ran SunOS' sys-unconfig, placed the drive in a shoebox with a different SCSI address and then proceeded to 'dd' to the other 30 drives.

pbrooks100
  • 251
  • 1
  • 4
0

For NTFS volumes, I prefer using ntfsclone. It's part of the ntfsprogs package.

Ed Brannin
  • 111
  • 5
0

As others have mentioned above, one of the gotchas with cloning a mounted file system is potential data corruption. This obviously won't apply to full drive clones, but if you're using LVM you can Snapshot the LogicalVolume and dd from the snapshot to get a consistent image.

Ophidian
  • 2,158
  • 13
  • 14
0

Just a warning to beginners that needs to be said: At least with some Versions, bs=X means that memory to the size of X will literally be allocated. bs=2GB on a system with 1GB of RAM and insufficient swap WILL cause bad things to happen.

rackandboneman
  • 2,487
  • 10
  • 8