Clone file, partition with dd, pv, nc locally or over network with SSH

Clone file, partition or disk using pv

A popular way as of 2013 is to use pv:
apt-get install pv
pv < /path/to/source > /path/to/target

It can be used to copy files, partitions or whole disks and it shows progressbar. Some say it's faster than dd. It is faster than Gparted anyway.

Note: if you copy different source and target sizes and are using GPT instead of MBR, then you might have to do this to move the GPT backup partition table to the end:

apt install gdisk
sgdisk -e /path/to/target

Changelog:
2025-10-11 16:57:00
2016-01-28 13:11:48

Clone Debian Linux hosts

Get two bootable USB drives with Debian Live.

Boot both, ensure both are online.

Setup the power management settings on both so they do not turn off when lid is closed or on other energy related events.

Open a terminal and install necessary tools on them:
sudo bash
apt update
apt install ethtool gparted netcat net-tools pv

Target:
Open gparted and resize the partitions so you have space for the Linux partitions.

Source:
Check partition sizes - note the sector values for each partition to clone:
fdisk -l /dev/sdX
Or
sfdisk -d /dev/sdX 

Target:
Re-create the Linux partition structure - for example an extended partition with the partitions from the step above.

Use cfdisk to make new partitions, then use the <sector number>S in the size field to make the sizes exact the same.

Connect both hosts with a network cable from RJ45 port to RJ45 port.

Find the network interface and set an IP on it:
ifconfig -a
ifconfig <interface name> 192.168.20.2

Source:

Find the network interface and set an IP on it:
ifconfig -a
ifconfig <interface name> 192.168.20.1

Ping the target:
ping 192.168.20.2

Check that the interface is running at gigabit speed:
ethtool <interface>|grep Speed

Check partitions to share:
fdisk -l /dev/sdX

Share them one by one - repeat this for each partition when the previous has completed:
cat /dev/sdXY | pv | nc -l -p 5555

Press Ctrl+C when they have reached 100% and the speed has gone down to 0.00B/s.

Target:

Check partitions to fill:
fdisk -l /dev/sdX

Read from the source one by one - repeat this for each partition when the previous has completed:
nc <source-host> 5555 | pv > /dev/sdXY


Source/target:
Check each partition with md5sum and compare results so they are the same:
md5sum /dev/sdX

Target:
partprobe

mount /dev/sdaXY /mnt

Now mount partitions according to source /etc/fstab, for example:
mount /dev/sdaXY /mnt/home
mount /dev/sdXX /mnt/boot

if UEFI:
mkdir -p /mnt/boot/efi
mount /dev/sdNN /mnt/boot/efi

mkdir -p /mnt/run/udev

Mount necessary parts:
for i in /dev /dev/pts /proc /sys /sys/firmware/efi/efivars /run/udev; do mount -B $i /mnt/$i ; done

OR:

mount -t proc proc /mnt/proc
mount -o bind /dev /mnt/dev
mount -o bind /dev/pts /mnt/dev/pts
mount -o bind /sys /mnt/sys
mount -o bind /sys/firmware/efi/efivars /mnt/sys/firmware/efi/efivars
mount --bind /run/udev /mnt/run/udev

Go into the root partition:
chroot /mnt

Install GRUB:
grub-install /dev/sdX
update-grub

exit

reboot

Changelog:
2023-04-08 18:32:29

dd / nc netcat disk clone over SSH

ssh root@sourcehost 'dd if=/dev/sda'| dd of=/dev/target status=progress

Source: https://www.maketecheasier.com/ssh-pipes-linux/

Combined with the rsync tip, to copy 32 GB:

dd if=/dev/sda status=progress bs=4M count=8192 | ssh -T -c arcfour -o Compression=no -x user@targethost 'dd of=/folder/on/target/file.img'

Got 42,5 MB/s with this from SSD to HDD over 1 Gbit.

Verify the file

dd if=/dev/sda status=progress bs=4M count=8192|md5sum

md5sum /folder/on/target/file.img

Netcat - higher speed, unencrypted

On the source:
cat /path/to/source | pv | nc -l -p 5555

To only use one IP on the source and not all:
cat /path/to/source | pv | nc -l -s 192.168.1.2 -p 5555

On the target:
nc <source-host> 5555 | pv > /path/to/target

Speeds were ~10-90 MB/s with this method.

Regarding VirtualBox VM:s

To increase speed between a host and a virtual machine switch network adapter on the host from Intel PRO 1000/MT Desktop to PCnet-FAST III, also replace NAT connection with a bridged connection if possible.

This increased speed by around +10 MB/s.

Synchronize only changes on block devices

If you already have an outdated copy of the block device you want to copy, then you can save by just updating it with the changes.

However, rsync does not work with block devices, but there are alternatives that do.

rsync alternative - bscp

Installation:
cd /opt
git clone https://github.com/dop251/diskrsync
ln -s  /opt/bscp/bscp /usr/bin/bscp

Usage:
bscp <source> <host>:<path> <block size> <hash, defaults to sha1>
bscp /dev/sdXY host:/path/to/device-or-file

Notes:
bscp does NOT need to be installed on the remote side.

It is NOT possible to specify a user with user@ before the hostname, it tries to log in with the username of the current local user.

Specifying a directory that does not exist results in cryptic errors.

What hashes that are available is not sure.

Speed test 6 GB file over a wireless network: 07:15.

rsync alternative - diskrsync

Installation - note, the binary MUST be installed on both the client AND the host:
apt install golang
cd /opt
mkdir diskrsync
cd diskrsync
export GOPATH=$(pwd)
go get github.com/dop251/diskrsync/diskrsync
ln -s /opt/diskrsync/bin/diskrsync /usr/local/bin/diskrsync

Usage:
diskrsync --no-compress --verbose <source> <host>:<target>
diskrsync --no-compress --verbose /dev/sdXY host:/path/to-device-or-file

Notes:
diskrsync binary must exist on both client and server the client it runs it on the server.
Login must be done using a private/public SSH key.
By default it stores files compressed in a spgz format, to avoid this use --no-compress.

Speed test 6 GB file over a wireless network: 10:17.

Changelog:
2023-12-25 00:50:48