Recursively mirror or copy files (local copy)

This is to copy directory trees on the same host. For remote copy see further down.

Copy a directory tree

There are many ways to do this and most people would probably suggest "cp -ap" but I prefer to use commands where I can do a dry-run to see what would happen before I actually copy the files.

cpio looks like a complicated command with many options but all the options we need to preserve attributes and creation times are -d -u -m -p and you can conveniently write this as "-dump" which is very easy to remember. You can optionally add -V to show progress or -v for a verbose printout.

cd /source/mydirectory find . | cpio -dump -V /destination/directory Dry-run this with: cd /source/mydirectory find .

You can as well go one level up and include the source directory (this will create /destination/directory/mydirectory):

cd /source find mydirectory | cpio -dump -V /destination/directory Dry-run this with: cd /source find mydirectory

The tar command can be used to copy a directory tree:

cd /source/directory tar cf - . | (cd /destination/directory; tar xvf -) Dry-run this with: cd /source/directory tar cf - . | (cd /destination/directory; tar tvf -)

Copy a directory tree, ignore file if it is already there regardless of time stamps

Normal mirroring takes file modification times into account and copies files even of the same name if the one in the source directory is more recent.

Sometimes you just want to not look at time stamps and just mirror based on file names. That is: if a file with the same name is already there then don't copy.

cp -prnv sourcedir /destination/directory This will create /destination/directory/sourcedir and all the files and directories inside sourcedir.

Mirror a directory tree

Mirror means to copy only files that have changed or are new.

Mirror the content of /source/mydirectory to /destination/directory/mydirectory

cd /source rsync -v -a --del mydirectory /destination/directory Dry-run this with: cd /source rsync -v -a --del -n mydirectory /destination/directory

Rsync offers the possibility to exclude certain file are directories. To use this feature you create a rsync.exclude file which contains all the files or directory trees that need to be excluded, one per line. Like this:

# rsync.exclude exclude file
mydirectory/.cache/
mydirectory/.bashrc

You use this exclude file like this:

cd /source rsync -v -a --del --exclude-from=/some/place/rsync.exclude mydirectory /destination/directory Dry-run this with: cd /source rsync -v -a --del -n --exclude-from=/some/place/rsync.exclude mydirectory /destination/directory

Mirror or copy from/to a remote machine

We use a SSH tunnel to stream the files between the computers.

Copy a directory tree, remote computer to local computer, tar

"cd" to the directory on the local computer were you want to have the files and use ssh + tar to stream the files over the network.

cd /where/this/should/go/on/local/host ssh remote_computer tar -cf - . | tar -xvf - Dry-run this with: ssh remote_computer tar -cf - . | tar -tvf -

This copies all files and directories in "." on the remote machine (whatever directory you end up when you ssh to that machines). To copy the content of the directory /dir/on/remote/host you can use the -C option of tar. "-C" (not -c) causes tar to change directory before doing anything else:

cd /where/this/should/go/on/local/host ssh remote_computer tar -C /dir/on/remote/host -cf - . | tar -xvf - Dry-run this with: ssh remote_computer tar -C /dir/on/remote/host -cf - . | tar -tvf -

Copy a directory tree, local computer to remote computer, tar

You are standing in-front of the local directory "somedir" (or some files) and you want to copy those recursively to the remote computer.

tar -cf - somedir | ssh remote_computer tar -C /dir/on/remote/host -xvf - Dry-run this with: tar -cf - somedir | ssh remote_computer tar -C /dir/on/remote/host -tvf -

This creates /dir/on/remote/host/somedir on the remote host.

Mirror a directory tree, remote computer to local computer, rsync

Mirror means to copy only files that have changed or are new. We use rsync for this but without running rsyncd permanently. We just tunnel the data over a ssh connection.

rsync -avHz -e ssh user@remotehost:/source/mydirectory /where/this/should/go

The above command creates /where/this/should/go/mydirectory on the local computer (including any sub-directories).

Mirror a directory tree, local computer to remote computer, rsync

Mirror means to copy only files that have changed or are new. We use rsync for this but without running rsyncd permanently. We just tunnel the data over a ssh connection.

rsync -avHz -e ssh /source/mydirectory user@remotehost:/where/this/should/go

The command is executed on the local computer and it creates /where/this/should/go/mydirectory on the remote computer (including any sub-directories).