Benchmarking backup tools

Test logs

Introduction

This document keep trace of the tests performed that lead to the summarized results presented in the benchmark document. Several helper scripts detailed at the end of this document have been necessary to obtain these results.

Testing Context

Software under test

• dar version 2.7.0
• rsync version 3.1.3
• tar GNU tar 1.30 under Linux and GNU tar (gdar) 1.32 under FreeBSD

Hardware used for testing

Performance tests has been performed on an HPE Proliant server (a ProLiant XL230a Gen9 running two Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz processors) running a Devuan beowulf Linux system. Other tests have been run from Virtual Machines (FreeBSD 12.1, Devuan 3.0.0) of a Proxmox hypervisor running one on an Intel Core i5-7400 (3 GHz) based computer.

Test logs

Completness of backup and restoration

Dar

We simply performed backup of SRC directory with dar's default options, then restore this backup into the DST directory, let's now compare SRC and DST contents:

The space used by DST is less than the space used by SRC! At first we could beleive that not all data could be restored, let's looking for the explanation:

All files are present in DST and use the expected space usage, as reported by the ls command. We can also see that the hard linked inode were properly restored for plain file, named pipe and unix socket: the inode number in first column is the same (see colorized output above).

Maybe something is missing elsewhere?

To summarize:

• user and group ownership are restored
• permission are set correctly
• ACL are properly restored
• Extended Attributes also
• file system specific attributes are too
• hard links are restored

So what? Let's rerun du file by file:

OK here is the explanation: plain_zeroed file was using 1048576 bytes of disk space in SRC and consumes only 4096 bytes in DST, but it has its file size is still officially 1048576, it has thus become now a sparse file (not all zeroed bytes are stored).

But nothing changes from the user point of view, the restoration process with dar just optimized the space usage.

Let's continue checking the inode dates. As you know, Unix inode have several dates:

• atime, the access time, gives the last time the file's data has been accessed (read)
• mtime, the modification time, gives the last time the file's data has been modified (wrote)
• ctime, the change time, gives the last time the file's metadata in other word the inode properties (ownership, ACL, permissions, dates, ...) have been modified
• btime, the birth time or yet creation time, gives the time the file has been created on the current filesystem, this date is not present on all Unix system

The ls -iRl command we used so far does only show the mtime date moreover with a time accuracy of only one minute, while modern systems provide nanosecond precision. For that reason we will use the stat command instead to have all dates at the system time accuracy:

From the above output we see that:

• atime is restored
• mtime is restored
• ctime is not restored

As we targeted this benchmark mainly for Linux which has not yet the btime available (Well some Linux file systems support btime but its access is not yet fully available to applications), we will thus momentarily change to a BSD system to play with btime. BSD systems include MACOS X, FreeBSD, NetBSD, butterflyBSD,... we will use FreeBSD here. Under FreeBSD, the stat command is not as easy to read as under Linux, however it is very flexible which we will leverage to mimic the Linux output:

In conclusion dar also saves and restores btime properly.

Rsync

Let's do the same we did previously using rsync. We start by copying SRC directory to DST:

First note, the backup and restoration is done in one step, where dar was decorelating the backup operation from the restoration operation. The resulting backup needs not software to be restored (DST is a copy of SRC). For dar to reach the same result (without using storage for the backup) this implies two dar commands: dar -c - -R SRC | dar -x - --sequential-read -R DST. The situation is similar with tar, you need two commands to perform the same task: tar -cf - | tar -xf -

Here too, the restored data uses less space than the original data, sparse file have been taken into account (need specifying -S option) and space optimization of non sparse file is performed.

All files are present in DST and use the expected space usage, as reported by the ls. We can also see that all three hard linked inode (plain file, socket and named pipe) are restored properly. So we can suspect the cause of the size difference to be linked with sparse files:

Let's now check file's metadata:

So in summary:

• Permission are restored,
• user and group ownership are restored too,
• mtime is restored,
• File ACL are restored,
• Extended Attributes are restored

But

• filesystem specific attributes are not restored,
• atime is not restored,
• ctime is not restored

For btime as we did before, let's test under a FreeBSD system:

So, birthtime is properly restored.

Tar

As done with previously, let's save and restore the SRC directory to DST... Note that by default no sparse file is taken into account (this is why we added the -S option), same with acl (so we added the --acl option) and Extended Attributes (unless --xattrs is added). The tar command-line becomes thus a bit longer:

Now let's compare the restored data with the original:

The sparse file has been properly restored (thanks to the -S option for that) but not space optimization has been performed.

The warning was not vain, SUB/hard_linked_socket and log are missing in DST. This is however a minor problem as usually unix sockets get recreated by the process using them. However we might have some permission and ownership to set back, by hand. A possible use case is syslog daemon, when let available for a chrooted process or container (MTA, or other network service).

The second problem is a bit more annoying: the hard linked fifo (aka named pipe) is silently restored as two independent named pipes (the inode number are different in the first column for pipe and SUB/hard_linked_pipe and their respective link count was 2 in SRC but is now 1 in DST. If two processes in different namespaces or chrooted environment, exchange data by mean of such hardlinked pipe, after restoration, if you are not aware of this failure, it will thus be difficult to identify why the two processes are just locked out, one waiting for data that will never come from the pipe, the other stuck for the pipe to be read.

Let's continue by checking the file's metadata:

Note that without --xattrs at creation time the timestamp accuracy of tar is 1 second: root@terre:/mnt/localdisk/Benchmark_tools# stat SRC/random DST/random File: SRC/random Size: 1048576 Blocks: 2048 IO Block: 4096 regular file Device: 802h/2050d Inode: 414841 Links: 1 Access: (0644/-rw-r--r--) Uid: (65534/ nobody) Gid: ( 0/ root) Access: 2020-10-27 14:03:46.064046436 +0100 Modify: 2020-10-27 14:03:42.016050420 +0100 Change: 2020-10-27 14:03:44.048048418 +0100 Birth: - File: DST/random Size: 1048576 Blocks: 2048 IO Block: 4096 regular file Device: 802h/2050d Inode: 414890 Links: 1 Access: (0644/-rw-r--r--) Uid: (65534/ nobody) Gid: ( 0/ root) Access: 2020-10-27 19:08:14.932424226 +0100 Modify: 2020-10-27 14:03:42.000000000 +0100 Change: 2020-10-27 19:08:14.932424226 +0100 Birth: - root@terre:/mnt/localdisk/Benchmark_tools#

From the above output we see that:

• permission are restored,
• user and group ownership are restored too,
• mtime is restored but it needs --xattrs to take into account today's system common time accuracy of one nanosecond
• ACL are restored,
• Extended Attributes are restored

But

• filesystem attributes are not restored,
• atime is not restored,
• ctime is not restored

For the last date, birthtime again we will perform the test under FreeBSD:

gtar saved and restored the birthtime

Feature set

Historization

To evaluate this feature, in a first time we will create two files A.txt and B.txt and make a first backup. Then we remove A.txt and add C.txt then make a second backup. We should be able to restore the data in both states (A+B and B+C). To simplify the operation we use the historization_feature script described at the end of this document.

Dar

Historization is present, we can get back from backup both saved states

In complement dar proposes a manager dar_manager to easily locate file's status between the archives the database has been feeded with, as well as the file's data present in each archive:

dar_manager can even take for you the actions to invoke dar as many time as necessary get the file's status of a given date for a given set of subset of the saved files:

Rsync

the "backup" contains all three files, A.txt, B.txt and C.txt while the first and the later never existed at the same time. Such backup does not allow to have neither the state of the phase1 nor the state of the phase2.

We added the --delete option and as result we got to be the phase2 state only. But then we cannot restore to the phase1 state as the file A.txt has been deleted from the backup.

To have both states with rsync, we should call rsync to a different destination directory at each new backup time, which would consume a lot of space and would also defeat one of the main features of rsync which is its ability to synchronize two directories exchanging only the minimal information that was modified.

Tar

We could restore from backup both the phase1 and phase2 status, historization is available with tar.

Data filtering by directory

Dar

We want to save /lib except the content of /lib/modules:

What if we want to exclude all of to exclude /lib/module except /lib/module/4.19.0-12-amd64?

OK, we can mix included directories and excluded directories

Rsync

We could exclude /lib/modules as expected. As previously, let's exclude it except /lib/modules/4.19.0-12-amd64:

OK, we can mix included directories and excluded directories

Tar

Let's save /lib and excluding /lib/module again:

Now let's exclude /lib/modules except /lib/modules/4.19.0-12-amd64:

root@terre:/mnt/memdisk# tar -cf backup.tar /lib/modules/4.19.0-12-amd64/ --exclude /lib/modules /lib tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets root@terre:/mnt/memdisk# tar -tf backup.tar | wc -l 6017 root@terre:/mnt/memdisk# tar -tf backup.tar | grep -v "lib/modules/4.19.0-12-amd64" | wc -l 1626 root@terre:/mnt/memdisk# tar -tf backup.tar | grep "lib/modules/4.19.0-12-amd64" | wc -l 4391 root@terre:/mnt/memdisk# tar -tf backup.tar | grep "lib/modules" | wc -l 4391 root@terre:/mnt/memdisk#

The backup contains a total of 6017 entries, 1626 are out of the lib/modules/4.19.0-12-amd64 directory, the rest is all in that previous directory, nothing else is found in lib/modules while there was lib/modules/4.19.0-11-amd64 and lib/modules/4.19.0-10-amd64 subdirectory. We can thus mix included and included directories.

Data filtering by filename

Dar

we would exclude all file having the ko extension, what if we do not want to exclude those that start with ext?

OK, we got what we wanted

Rsync

Same as previously, we don't want to exclude ko files starting by ext:

OK, we got what we wanted

Tar

Same as previously, let's filter out kernel object files

Now, we want to keep only those kernel object files starting with ext

Well, argument passed out of option do not seem expanded by tar thus using mask is not possible to include some pattern. It seems the only option is to use file listing, thing we will evaluate below.

Data filtering by filesystem

We will use a tmpfs filesystem mounted twice thanks to mount's --bind option. The objective is first to save every thing except a few given filesystems, or only one or save inside a few given filesystems. Here is the preparation phase:

Dar

We could exclude a filesystem, and its second appearance in D2 was also excluded, whithout having to mention it. Let's include only D1 now:

OK, we got what we wanted

Rsync

rsync has ony one option about filesystems, it sticks recursion to the filesystem of the source directory, we cannot exclude specifically some filesystems, they are all excluded, and we cannot include specifically some filesystems, none is excluded (default behavior without this option)

Tar

tar does not behaves better than rsync on that topic

Data filtering by tag

by tag we mean any mark the user can add to a file that will drive its fate when backup will be done. The most common is the dump flag, but it is not always available, using some other mechanisms (Extended Attributes,...) can be an interesting alternative.

Dar

We have two mechanisms, one based on the dump flag and an arbitrary extended attribute. However dar only supports exclusion of file, not inclusion for backup based on a tag.

Rsync

rsync does not seem to be able to filter based on an arbitrary mark

Tar

rsync does not seem to be able to filter based on an arbitrary mark

Data filtering by files listing

We build a file listing and expect to either have only those file saved or excluded from the performed backup. Here is the listing preparation:

Dar

file inclusion is available, let's see file exclusion:

Rsync

file inclusion is available. However if we can exclude a list of pattern defined in a file, we cannot exclude a list of files. We should prepend each entry by "- " seen the filtering syntax of rsync:

So we are good with this sed additional listing adaptation.

Tar

the include.txt file does not contain any file with the ko extension, however tar saved all of them. Reading back the man page concerning this --files-from option The names read are handled the same way as command line arguments explains that in the listing where all "*.ko" files have been removed, remain their parent directory, which implies saving all its content. In consequence we must not list directories only their content (which will restrict us saving empty directories as such). Let's modify the include.txt file that way:

the difference between the 1532 entries saved by tar and the 4123 saved by rsync or dar comes from the many empty directories that cannot be saved as such by tar using this method.

The file listing exclusion works as expected

Slicing

For this test we will backup the content of /usr/bin of the running system. We select a slice size smaller than the biggest file under backup. The use case for slicing implies compression (remote storage, cloud storage, limited removable media storage...).

Dar

We can also specify a different size for the first slice when using dar, this was used in the past to fulfill a disk partially filled by a previous incremental backup when saving onto CD-RW and DVD-RW, but that may still make sense when using USB keys or any other removable media.

Rsync

rsync cannot split any file in slices, and it does not generate any backup, but it copies files. You cannot thus split data into slices to fit a particular restricted storage space.

Tar

As reported by tar above, if a multi-volume support exists, it is quite restrictive as one cannot use compression at the same time.

But even without compression, tar is still restrictive: it does not produce different files, you have each new volume around and hit return at each time.
Note also that without compression, the space required passes from 8 volumes with dar to 21 volumes with tar.

The multi-volume support for tar seems well defined for local tape removable devices, but will cost more than twice more tape than what you can do with dar even if tape media is your only target. Here is an example with dar on how to write to mutli-volume and compressed backup to tape and pause between each volume as tar does:

Symmetric encryption

Encryption has for target relatively long term lifetime, having compression at the same time to increase security as it increases data "randomness" of the data to cipher. So we will use both in our tests (gzip with a compresion level of 6).

A point to pay attention to concerns the way the password/passphrase can be provided. Putting this to the command-line could let other users on this same system read it. Having interactive prompt is better as well as having the password set in a read access restricted file, which in addition allows automation.

Dar

We can provide password either on command-line (not recommended), prompted by dar once launched or from a protected configuration file. In the following we add slicing to encryption to see whether or not dar deciphers the whole backup to recover a single file:

As seen above, dar does not need to uncipher nor uncompress the whole backup to recover a single file, the use of slicing let us see which slice it accessed to, but the behavior is the same without slicing and can be measure by the execution time (see the performance tests logs).

Rsync

rsync cannot cipher data, it can rely on ssh to cipher the data over the network but data is finally always stored in clear text.

Tar

There is no native support for ciphering with tar. You can however pipe tar's output to openssl to cipher the generated backup on fly as a whole.

with openssl, tar has both the ability to provide the password/passphrase from an interactive prompt and from a protected file. However you will have to remember which algorithm you used in adition to the passphrase. The ciphering being done as a whole, you will have to decipher the whole backup even to just restore a single file. If the backup is large, this may take a long time and may require to download a lot of stuff from a remote storage.

We see that ciphering with tar is possible at the cost of some complex command-line. But this is error-prone as we see the shown warning that the key derivation function is deprecated and we should switch to another one. Moreover you will have to remember which key derivation function and its parameters in addition to the passphrase you provided and in addition to the ciphering algorithm used.

Note: you can also use openssl with dar as we did for tar but it brings all the drawbacks we saw with tar

Asymmetric encryption

The objective is to create a backup ciphered using GnuPG public/private key pair, restore the whole backup and restore a single file from it. We will also use compression (gzip level 6) as it may make sense for the corresponding use cases (data exchange over Internet for example).

Dar

As displayed in the backup header output above the underlying encryption is a symmetric encryption (AES 256 by default), but the AES key is stored ciphered using the private key of the backup recipient which email address is provided (or email adresses, if more than one recipient is expected). This key is randomly chosen by dar and stored ciphered in the archive header. Thus the overall behavior, performance and security of GnuPG withing dar is equivalent to the one of the symmetrical algorithm chosen, with the ability to quickly restore some or all files from an archive, and not waiting/downloading first the whole backup to unciphered it.

Seen above no password or passphrase is asked as the recipient email is ourselves (root@terre.systeme-solaire.espace). Let's cipher for another recipient:

Here we saw that ciphering for a recipient different than ourself does not allow us to read the resulting backup, however we can define several recipients and if we add ourself, we can read the backup as well as our primary recipients.

Rsync

Tar

Tar cannot hold asymmetrical encryption alone, as for symmetrical encryption we must use an external tool that performes the ciphering operation outside the backup.

Same as for symmetric encryption, the fact that the whole backup is ciphered at once implies to download back the whole backup even to recover just one file.

Protection against plain-text attack

Dar

When ciphering the same data several times (with symmetric or asymmetric encryption), the resulting backup size changes each time. This is due to the garbage (the elastic buffer) dar adds at the beginnning and at the end of the data to cipher. This way, even if a dar backup has well known structure it is not easy to know precisely where they are positionned in the backup file, which makes plain-text attack much more difficult to succeed if even possible in a reasonable time.

Rsync

rsync does not provide any way to cipher the backup, it is thus not concerned by protecting against plain-text attack.

Tar

tar by itself does not provide any ciphering mechanism, however you can cipher the tar generated backups with external tool (for example openssl for symmetric encryption or gpg for asymmetric encryption). However none of these mechanism protect against plain-text attack: tar backup have somehow predictable header contents.

Key Derivation Function

Dar

dar uses argon2 by default, with 10,000 iterations. It can also use pkcs5 v2 (pbkdf2) with md5, sha1 or sha512 algorithm. The user is able to set the KDF function and iteration count, so we are able to measure the execution time variation added by the iteration count (taking into account that the data to cipher also changes depending on the amount of random garbage dar wraps it with):

Rsync

rsync does not provide any way to cipher the backup, it is not concerned by KDF.

Tar

As of today (year 2020) openssl only supports PBKDF2: no support for argon2 is available. Argon2 was the winner of the Password Hashing Competition in July 2015. PBKDF2 has been published by the IETF in September 2000 with the RCF 2898

File change detection

In order stress each backup software on that aspect, we will use an ugly script always_change that loops forever permanently invoking touch on a given file. For the test, we create a source tree to backup, containing a file of 1 MiB on which we will apply this script:

Dar

• dar detects properly the file change and issues a warning during the backup.
• It even retries to save the file several times (3 times by default).
• the resulting backup keeps trace of this context by flagging the file as DIRTY
• When restoring the data, a warning shows (default behavior) and the user is requested for confirmation.

Rsync

rsync does not shows anything nor behaves differently (no retry, no change notification).

Tar

• tar detects properly the file change and issues a warning during the backup.
• it does not tries to save the file
• the resulting backup keeps no visible trace of this possible data corruption
• When restoring the data, no warning is issued and the restoration proceed as if the file was saved properly

Multi-level backup

For this test we make a full backup of a Linux source tree, then rename the Documentation directory as doc and make a differential backup of the whole. Renaming files is expected to do at worse the same as removing some and adding new ones, whe should not see all data saved again:

Dar

We can see that the restoration of the full and differential backup over it lead to the exact same directory tree as the source saved files.

Rsync

We see that after the modification the amount of data pushed to the backup by rsync passes from 214 MiB to only 12 MiB we can consider this as a differential backup, thus this part of the multi-level backup aspect is addressed, but we have lost the access to the first backup: it has been overwritten by the new one, so we lose history but that's a different feature.

Tar

Here too, we got the exact same directory as original and modified data

Binary Delta

To evaluate the ability the support for binary delta, we will make a first backup of a Debian ISO image, of which we will modify one bit using the bitflip script, then make a differential backup of it. We expect to see the differential backup not resaving the whole file, and though the restoration of the full and differential backup matching the modified file.

Dar

For dar the backup we used:

• 4.3 GiB for the full backup (compression ratio of 0,21 %)
• 614 bytes for the differential backup (compression ratio of 99,99998%)

Rsync

We had to use --no-whole-file to see the binary delta in action with rsync. This feature is not activated when copying on local disk as it does not makes sense (for rsync) because the computation time needed for the binary delta takes more time the the byte to byte copy and because rsync does not store just the delta (no backup history) but modifies the existing backup. Anyway, binary delta is supported (of course!) by rsync.

Tar

For tar the backup used:

• 4.3 GiB for full backup (compression ratio of 0,22 %)
• 4.3 GiB for the differential backup (compression ratio of 0,22 %)

Binary delta is not supported by tar

Detection suspicious modifications

For this test we will use the hide_change script that rely on the bitflip script seen above and try to hide the modifications performed, as a virus, keylogger or rootkit would tend to do. We will make a full backup before the modification and a differential backup after, then observe the behavior.

Here follows the script in action, we see no change using ls -l while stat shows the exact same information:

• file size
• block used
• Inode number
• permissions
• user and group ownership
• last access time
• last modification time

Dar

dar issues a warning because of this suspicious condition. Note that we still have the sane file in the full backup, in case of doubt, we can compare it with this modified version:

The previous test reports that the first byte to have changed is at offset 0, thus this is not just a metadata change that lead to this warning. We can if necessary restore the sane data from the full backup.

Rsync

rsync has not reported the problem, but hopefully it has not synchronized the backup, thus we end in a sane version in the DST backup directory though, as user is not aware of this potential risk, the virus/ransomware can spread silently.

Tar

As seen above tar does not see any problem, but the file has been resaved as a whole (while its last modification time was unchanged) which lead to corrupt the new backup with potential harmful data. The good point is that you have still the full backup with the sane data. But at a next backup cycle, as you were not notified of the risk, you will lose it and keep only the corrupted version of this file.

Snapshot

Dar

As seen above,a snapshot can be created:

• as part of a backup process (full, differential, incremental or even decremental backup) (--on-fly-isolate)
• from a existing backup (-C)
• alone by a dedicated operation (-A +)

Once created a snapshot can be used:

• to create a differential or incremental backup
• to list the files that would be saved (thus which have changed, were added or have been removed) thus all changes since the time the snapshot was made (using the --dry-run option)
• rescue a corrupted backup when the corruption falled into the backup table of content located at the end of the dar backup (only if it has been created based on the backup to rescue either using -C option maybe long after the backup was made or using --on-fly-isolate at the same time the backup was created.

Rsync

This feature is not supported by rsync.

Tar

tar can generate snapshot:

• alone redirecting the backup output to /dev/null
• as part of a backup process (in fact tar cannot do else)

If a snapshot can be used (and is in fact required) to make a differential backup, it cannot really be used to see the difference a current living filesystem has with a given snapshot. Worse, doing so modifies the snapshot, so you have first to make a copy to not screw up your backup process. Worse, if incremental backup fails and you have not created a copy of the backup, your snapshot being modified you will mostly have to remake the whole backup process from the full backup to be sure to not miss backing up some modified files. Same thing if you lose by mistake the snapshot file.

On-fly hashing

Dar

Rsync

not supported by rsync

Tar

not supported by tar

Custom command during operation

As an example (but there is much more thing that can be done), we take the case of a automounted directory. Such type of volume is mounted only when used, if not used no mount point directory shows and unless you know it exists, no backup of its content is performed. The idea, is when entering the parent directory at backup process to trigger the mount point for the backup to include them.

Dar

In the previous example we see that the /mnt/Externe directory is a mount point containing three auto-mounted volumes: Espace, Commun and Backup. At first only Espace was mounted. Performing a backup without care will skip the two other directories.

In a second time, thanks to the -< and -= options, we instructed dar to run the file command on the two missing directories when entering /mnt. As a result, we now see both of them in the backup. We could do that before executing the backup, but as the backup may include many other directories the time between such operation done before starting the backup and the time the backup finally saves the automount point at /mnt/Externe may exceed the automount timeout leading them to be unmounted and disappear before the backup process reaches them.

In this example, this we used slicing with on-fly hashing which generated for each slice the corresponding sha512 hash file. Then we tested the archive content and at the same time the hash files thanks to the -E option. Of course any user command or shell or python script, can be used instead, and for backup, restoration, testing, snashotting,...

Rsync

not supported by rsync

Tar

tar has the -F option to launch a command after each tape, but it is only available with multi-volume tar archive, which in turn cannot be used with compression. Thus we won't test it, as it is quite restrictive and does not match any common use cases.

Dry-run execution

Dar

Rsync

Tar

does not seem supported by tar

User message within backup

Dar

The use of the -aheader let one see the archive header that is always in clear-text. The usual listing operation provides some additional informations from the ciphered table of content and thus in that context requires the passphrase:

Rsync

not supported by rsync

Tar

not supported by tar

backup sanity test

Dar

Rsync

It does not seems possible to let rsync check that the target or destination directory is sane and usuable. All operation modify the destination file or save modified files in either the destination directory (the backup) or an alternate directory (--compare-dest option).

Tar

Comparing with original data

Dar

Rsync

Does not seems supported by rsync

Tar

Tunable verbosity

Dar

dar has several options to define which type of message to show or not to show: -v, -vs, -vt, -vd, -vf, -vm, -vmasks, -q. They can be combined.

Rsync

-v option leads to a more verbose output, while -q remove the non error messages. Using both at the same time seems not to be different than using -q alone. However rsync has a very rich set of additional options like --info, --debug that can be added on top.

Tar

tar only provides the -v option to increase verbosity.

Modify Backup content

We will perform two types of tests:

• remove one or several files from an existing backup without having to make a new backup process
• add some forgotten files to an existing backup without performing a new full backup process

Dar

dar does not modify a existing backup but creates a copy of it with the requested files or directory removed. The process can be quick even with compression thanks to the -ak option that avoid uncompressing and recompressing file that are kept. Before removing the old backup you can test the sanity of the new generated one.

Here to add files to a existing backup we must make a small backup of these files only, then merge this backup with the backup we want to modify. Nothing of the source data is touched in this operation, is something goes wrong or if you made an error, you can fix and restart without taking the risk to lose data.

Rsync

The backup made by rsync is just a copy of the save files, removing a file from the backup is as simple as calling rm on that file in the repository that is considered the backup.

While adding a new file in the backup can be done by using rsync as usual including the directory tree where this file resides.

Tar

Well, tar cannot manipulate compressed archives. What the point then to remove a file from a backup if storage space is not an issue, else, would compression be used?

stdin/stdout backup read/write

Dar

dar can read a backup from stdin and write a backup to stdout.

Rsync

Using stdin/stdout to send to or read from backed up data does not seems possible with rsync

Tar

tar can read a backup from stdin and write a backup to stdout.

Remote network storage

For Remote Network storage, if you use a personal NAS you may avoid generating ciphered backup, though you still should transfer it using a secured protocol if the underlaying network is not your own from end to end (for example a part of the path goes over Internet without IPSec or equivalent).

Why ciphering backup if using secure transfer protocol?

• Secure transfer avoids one to read your data in transit in particular your credentials to the remote site, but at the end the transferred data is stored in clear at the other end.
• ciphered data avoids the owner of the remote storage to access to your data without your consent, though it does not protect one to intercept your transfer and read the credentials you used to connected to the remote storage. This one could then connect later and delete all your backup... which is surely not what you want.

In the following we will use both: secure protocol and ciphered backup, without using local storage. We will also need compression to save precious space (usually you pay for the cloud storage you use) and maybe slicing depending on the constraints imposed by the remote storage (some provider ask you to pay an extra amount to store larger files, having slicing avoids you paying extra cost in such context). Another use case of slicing is when the file transfer protocol is not able to continue an interrupted transfer, you will then only need to restart it for the last slice, not the whole backup.

Dar

The backup results in four ciphered slices located on the remote sftp server. Let's add a -E option to see which slice are being read while testing the archive.

We see that all slices have been read as expected, now let's restore /etc/fstab in the current directory and compare the restored files with the real /etc/fstab

As seen above only one slice (slice #4) has been necessary to restore /etc/fstab. But let's save two files, a huge one and a small one into a single sliced backup and measure the transfer time of backup and restoration of this ciphered an compressed backup through sftp. We have added public key authentification for precise time measurement:

While restoring the whole backup needs 15 seconds of transfer time, restoring fstab alone requires only 0.87 second, as there is only one slice, this shows that dar is reading only the necessary part of the archive even within a slice to perform the operation.

Rsync

We can backup to a remote sftp server, we can compress the data on-fly but it is not stored compressed nor ciphered. Restoration operation is possible per file or for the whole backup

Tar

tar has no way to perform sftp or other secured transfer protocol, nor encryption by itself. When time comes to restore a particular file, the whole backup has to be retrieved, unciphered and uncompressed to restore even just a sigle file

Backup Robustness

The objective of this test is to measure the way the backup tools under test behave when the backup has been corrupted. We will here just flip one byte of the backup, at the beginning, in the middle or at the end of the backup and observe the consequences in term of ability to restore the backup.

Dar

Modifying the first bit dar has seen the corruption. We can use the lax mode (-alax option) to bypass this corruption and then the restoration proceeds normally. We can try a bit further for example somewhere in the middle of the archive, thus at offset 876354144 (half of the size expressed in bit, not byte):

One file could not be restored properly as reported by dar, but all other files could be and are identical to their respective originals. Let's modifying the last bit for completness:

By chance it did not affected the ability to restore the backup. However if it ever had, we have several fallbacks: the --sequential-read mode, the use a already created snapshot (aka isolated catalogue) as seen about the snapshot feature to backup the internal table of content, or as last resort the -alax option eventually combined with the --sequential-read mode and a backup snapshot.

Rsync

modifying the backup (the directory we sync with), rsync does not report any difference and the backup stay corrupted.

Tar

Modifying the first byte leads to a completely unusable backup. Nothing got restored at all. Let's see what going on when modifying a single bit in the middle of the backup:

Only 32615 files on the 74726 that were saved could be restored. Assuming the problem is due to the fact the backup is compressed, let's see tar without compression:

Without compression, a tar backup is much more reliable, however we now need more than 5 times storage space to hold the backup. Let's see what happens when we modify a single bit in the midle of the backup:

the backup restoration suceeded according to tar but the corruption has been completely ignored!!! The result is both a corrupted backup and a corrupted restored data, with no notification at all...

Parchive

We can increase the robustness of any file or set of files by mean of Parchive software. If its use is adapted to tar and dar it is not adapted to rsync due to the directory tree structure it uses for its backup. We will thus here measure the par2create (Parchive) execution time compared to backup time of tar and dar.

We see that the redundancy process is not negligible: for 5% of data redundancy, we need around 10% of extra CPU time. Things get worse when the size of the data to process by Parchive increases and when the disk I/O comes to play (here the /mnt/memdisk was an in-memory tmpfs filesystem). This is the case when the amount of data to backup is larger than the available RAM space, which is a quite frequent situation:

We are now placed in this context, having a 27 GiB data to backup on a machine having only around 16 GiB of RAM.

This execution time of around 36 mn above can be improved by using multiple slices. Choosing a slice size smaller than the available RAM let Parchive compute parity data right after each slice has been generated, while it is still in the disk cache (RAM), bypassing the corresponding disks I/O we had previously in a second time:

The total execution time dropped to around 22 mn! Which makes a 40% time reduction. By the way, having here 27 files of 1 GiB is also easier to manipulate (file transfer, copy to removable media,...) than a huge equivalent file of 27 GiB.

We get the same execution time (18 mn) for both tar and par2 for thus a total of 36 mn. This same time for both software while the real CPU usage is much more important for par2, clearly shows that the slowest operation was the disk I/O. Else the overall time of the operation is similar to what we say with dar above, except that we cannot use multi-volume to speed up the operation as we did with dar: tar is not able to compress *and* produce multi-volume backup: What we would gain on one side would be lost on the other side...

Execution Performance

In order to compare performance in a fair manner, we have to take into consideration that some CPU intensive features are not implement by all softwares or have different default values:

• The default compression level differs: dar uses level 9 by default while rsync and tar use level 6 which is faster. Only dar and rsync seems able let the user set this value, so we will have to manually set dar to use level 6 too: -z6 option
• the disk usage optimization is not supported by tar so we will not activate sparse file detection and optimization (for rsync no -S option) and disable it for dar (activated by default): -1 0 option
• dar spends non negligible CPU cycle to duplicate metadata along the backup, this is one of the root of exclusive robusness brought by dar backups, but this is an optional feature. We will disable it using the -at option
• rsync and dar calculate checksum on the saved data, while tar completely skips this data protection. Unfortunately this is not possible to disable for both, thus tar will have a speed advantage thanks to this difference.

All performance aspects are not interesting in all use cases. We can distinguish two main types of use cases:

• A first set of tests will cover the copy operations, they will not use any compression, focusing only on execution time.
• A second set of tests will cover the backup operations, use compression and focus on the data reduction mainly but also on the execution time.

When using rsync as a backup tool (at the opposite of a copy operation), we assume the remote (or local) copy is the backup, and thus restoring implies syncing back this remote (or local) copy to the place the original data was located and has been lost.

To prepare the data under test we used:

• a big ISO file
• a full linux system

All data to backup or to copy has been stored in a tmpfs, which is also the destination of the created backups and restored data. The swap has been disabled to avoid any disk I/O penalty, in the intention to provide a fair comparison environment (avoiding disk cache variable performance).

To prepare the Linux system under backup we installed the Devuan system a few days before in a VM. On day D, a full backup has been executed, we updated/upgraded the system using the disto package manager and we made a differential backup based on the first one, both backup being wrote to the testing machine (bare-metal server) that was used for the performance tests:

Back on the testing host (the bare-metal server at 10.13.30.163) we restored the data for the performance test, the following way: excluding FSA and EA to avoid a tone of warning as those are not supported on tmpfs filesystem:

This leads us to have two directories state-1 and state-2 corresponding to the state of the Devuan machine has two days ago and today respectively.

Performance of copy operations

To perform the copy operation, we have decomposed the operations to precisely measure the execution time. We could have decided to pipe the backup to a second instance of the backup tool restoring the data (tar and dar only would benefit from this). But the time measurement was less easy to obtain and doing that way does not seem to provide any noticable speed improvement. The data used here in SRC1 is a single big ISO file (a Devuan installation DVD image).

Dar

The overall copy time for dar is:

• 9.18 seconds for the big file in SRC1
• 16.78 seconds for the full linux system in SRC2

Rsync

The overall copy time for rsync is:

• 15.28 seconds for the big file in SRC1
• 16.59 seconds for the full linux system in SRC2

Tar

the overall copy time for tar is:

• 6.51 seconds for the big file in SRC1
• 8.04 seconds for the full Linux system in SRC2

Cp

The overall copy time for cp is:

• 2.58 seconds for the big file in SRC1
• 5.15 seconds for the full Linux system in SRC2

cp is always the fastest and does reject the unix sockets as tar does. However it requires slightly more storage than all other softwares tested here. And if metadata (ACL, Extended Attributes, filesystem specific attributes, ...) need to be copied with data, it does not match the need.

Performance of Backup operations

For reference:

Dar

Minimal features

for dar with minimal features (metadata no redundancy -at, no sparse file consideration -1 0):

• data reduction on storage for the full backup is: 61.41%
• data reduction on storage for the diff backup is: 98.80%
• data reduction over the network is the same as on storage
• execution time to restore a single file is: 0.98 s
• execution time to restore the full backup: 22.94 s
• execution time ot restore the diff backup: 3.48 s
• full backup time: 149.73 s
• diff backup time: 9.93 s

sparse file

for dar with default features (metadata no redundancy -at, with sparse file consideration (activated by default))

• data reduction on storage for the full backup is: 61.46%
• data reduction on storage for the diff backup is: 98.80%
• data reduction over the network is the same as on storage
• execution time to restore a single file is: 1.13 s
• execution time to restore the full backup: 30.36 s
• execution time ot restore the diff backup: 3.48 s
• full backup time: 157.99 s
• diff backup time: 10.37 s

Sparse file and binary delta

for dar with advanced features (metadata no redundancy -at, with binary delta computation --delta sig):

• data reduction on storage for the full backup is: 60,87%
• data reduction on storage for the diff backup is: 99.42%
• data reduction over the network is the same as on storage
• execution time to restore a single file is: 1.27 s
• execution time to restore the full backup: 30.35 s
• execution time ot restore the diff backup: 1.27 s
• full backup time: 162.62 s
• diff backup time: 7.11 s

Rsync

minimal features

for rsync with minimal features (no sparse file consideration):

• data reduction on storage for the full backup is: 0%
• data reduction on storage for the diff backup is: 0%
• data reduction over the network for the full backup is: 61.23%
• data reduction over the network for the diff backup is: 99.29%
• execution time to restore a single file is: 0.003 s
• execution time ot restore the full+diff backup: 157.81 s (this is not due to the --no-whole-file see next text)
• full backup time: 156.98 s
• diff backup time: 7.33 s

sparse file + binary delta

for rsync with advanced features (sparse file consideration -S option, binary delta --no-whole-file)

• data reduction on storage for the full backup is: 0%
• data reduction on storage for the diff backup is: 0%
• data reduction over the network for the full backup is: 60.64%
• data reduction over the network for the diff backup is: 99.28%
• execution time to restore a single file is: 0.003 s
• execution time to restore the full+diff backup: 158.39 s
• full backup time: 183.44 s
• diff backup time: 7.04 s

Tar

minimal features

Doing that way, the tar differential backup is empty: it only contains empty directories, no file data. We will apply the changes over state-1 rather than already setup changes at state-2. This seems to mean that if the system clock is wrong or was wrong at the time a file was modified (like daylight saving? or before NTP synchronization at system startup), which is the same as here, were the changes have been brought before full backup was done, those changes will not be backed up by tar, while the file's attributes (file size, last modification date,...) changed.

for tar with minimal features (no sparse file consideration):

• data reduction on storage for the full backup is: 62.16%
• data reduction on storage for the diff backup is: 98.92%
• data reduction over the network is the same as on storage
• execution time to restore a single file is: 25.15 s
• execution time to restore the full backup: 26.72 s
• execution time ot restore the diff backup: 1.48 s
• full backup time: 148.59 s
• diff backup time: 6.40 s

sparse file

Whe had to recreate state-1 as we needed modifying it at previous test

for tar with advanced features (sparse file consideration -S option):

• data reduction on storage for the full backup is: 62.16%
• data reduction on storage for the diff backup is: 98.92%
• data reduction over the network is the same as on storage
• execution time to restore a single file is: 25.0 s
• execution time to restore the full backup: 26.27 s
• execution time ot restore the diff backup: 1.50 s
• full backup time: 149.38 s
• diff backup time: 6.55 s

Ciphering performance

We evaluate here ciphering and deciphering performance. To compare on the same base we use the following parameters:

• AES-256 algorithm with CBC mode
• pkcs5 v2 (pbkdf2) key derivation function (KDF) algorithm
• KDF with 100,000 iterations
• salt
• password provided on command-line (insecure) to not depend on user or disk access
• local system to backup and backup repository on a tmpfs filesystem
• swap has been disabled to avoid tmpfs latency in case it would have been swapped out

The content that will be backed up is a copy of /usr directory tree. We will measure:

• the time to backup
• the time to restore the whole backup
• and the time to restore just the "diff" binary

For dar the operation took:

• 5.38 seconds to backup with encryption
• 3.75 seconds to restore the whole ciphered backup
• 0.42 seconds to restore a single file from the ciphered backup

dar is twice quicker to uncipher than to cipher the whole archive, but restoring a particular file is quite immediate. By default, dar uses argon2 for KDF, which is the most secure algorithm as of year 2020 to derive a key, but we had to adapt to openssl used with tar that does not (yet) support this algorithm.

To avoid plain-text attack a variable length elastic buffer containing random data is encrypted with the rest of the backed up files at the beginning and at the end of the backup, this has some performance penalties (time to generate and time to cipher/decipher). This explains why two identical invocations of dar produce backups of different sizes and execution times:

rsync has no way to store the backup ciphered. Testing directly tar now:

tar has not support for ciphering. Though it seems the some use openssl workaround this restriction. To measure the execution time we have to create as script that pipes tar and openssl so we can measure the execution time of this script as a whole. There is thus one script for backup and one for the restoration of tar+openssl.

For tar the operation took:

• 4.69 seconds for backup
• 2.70 seconds to restore the whole backup
• 1.79 seconds to restore a single file

tar as dar is also twice longer to cipher than to decipher, this seems to be related to the algorithm itself. Though tar is a bit faster than dar but lacks protection against clear-text: the generated encrypted backup have the exact same sizes at one byte precision, this means the blocks boundaries and tar file internal structure always lay at the same file offset for a given content:

Scripts used in this benchmark

The following scripts are also available for download from the directory

historization_feature script

always_change script

bitflip script

build_test_tree.bash script

hide_change script