Cognitive Overhead

In this final article, I'm going to talk about the little extras that can't be included in my comparison as a measurable criterion.

I'm going to talk briefly about graphical interfaces, because for some users, this can make a big difference. Application sizes are given for the macOS version.
Of the three programs, Restic has the richest and least mature ecosystem in terms of graphical user interfaces. There are several of them, but they're all independent projects more or less in the alpha or beta phase, and they don't all offer the same level of functionality.
I have only tested Restic Browser. This lightweight graphical interface (13 MB) is designed solely for browsing snapshots and restoring files. The application works perfectly and does exactly what's expected of it. Nothing to say, except that it's a very pleasant complement to a stand-alone backup script running on the user's machine.

Borg's main GUI project is called Vorta. It's a full-featured application that lets you fully configure backups, with scheduling, exclusion lists, compression parameters and more. This software weighs in at around 127 MB, but offers full functionality (configuration, scheduling, cleanup, restore, repository verification, etc.). On macOS, you'll just need to install the Fuse drivers to mount a remote archive locally. As Borg works without a configuration file, and without storing its configuration on the repository, it is unfortunately impossible for Vorta to retrieve the settings for backups, which are done on the command line. It can, however, connect to an existing repository, list archives, perform backups, restores and all other tasks. Just bear in mind that Vorta has no knowledge of the parameters you pass to Borg in any backup script. It's therefore a program that's best suited to stand-alone use: you set your backup/restore parameters directly in Vorta, and Vorta takes care of the rest.

In contrast, Kopia offers a totally unified backup experience between its command-line client and its KopiaUI interface. All backup, cleanup and other parameters are configured in a policy accessible to both environments. This means you can switch from the command line to the GUI, no questions asked. Still, KopiaUI weighs in at 313 MB and offers far less advanced functionality than Vorta. It is impossible, for example, to launch a verification task on a snapshot, or to make a comparison between two snapshots. Note that it is possible to display this user interface without even installing KopiaUI: in fact, kopia's command line integrates a server mode that displays exactly the same graphical interface in a Web browser.

In addition to graphical interfaces, there are "wrappers" that facilitate the use of Restic and Borg. For Kopia, this type of tool probably doesn't exist, or exists only marginally, due to its integrated management of backup policies. On the other hand, Restic and Borg do not benefit from complete configuration management, and these command-line "wrappers" can be very useful. I haven't tested one for Restic, but I've been using Borgmatic for some of my machines for several years. Using YAML files, this tool allows you to configure all backup, retention and other parameters.

Example of use:

@daily borgmatic -v 1 create --files prune ; borgmatic list
0 2 16-23 * 2 borgmatic -v 1 check
0 2 16-23 * 3 borgmatic -v 1 compact

kopia's server mode and Restic's REST-server mode make it easy to manage a multi-user backup repository. This is a type of operation that is not trivial to reproduce with Borg, as the latter requires users to be provided with SSH access. Of course, SSH access can be limited (must be!), but this entails management complications that go far beyond a simple login and password, as is the case with Restic and Kopia.
In multi-user mode, Kopia is superior to the REST-server. In fact, the latter only allows users to be managed via an .htpasswd file, whereas Kopia provides rather elaborate management of users and their access authorizations. I couldn't find any mention in the Restic documentation of its ability to share duplication when several users are backing up their data on the same server. However, this is an official feature of Kopia in server mode. It's important to note that this raises questions in terms of security. If users share a common repository, they can theoretically retrieve data from other users. Fortunately, this is only possible if the user knows the 256-bit hash corresponding to the file he wishes to retrieve. On the other hand, it could very well be used intentionally by Alice to share with Bob a file she has saved via Kopia. All Alice has to do is transmit the 256-bit hash to Bob, who can then restore it via his Kopia client.

Of these three backup solutions, only Borg doesn't offer a wide range of storage modes, including several in cloud environments that don't require deployment of the application. It is no doubt because of this difference that the market offers little or no storage for Restic or Kopia-based backups. There are a few commercial services offering storage for Borg backup repositories, such as rsync.net or borgbase.com. The latter is the only one, as far as I know, that also offers Restic backup hosting. I haven't found anything for Kopia, so don't hesitate to correct me in the comments. There are, however, a few turnkey solutions based on consumer NAS boxes, such as QNap, for which these three applications have been implemented.

To conclude this series of articles, I'll summarize the qualification tables for all three products here.

Please note that functional coverage (portability, storage and transport options, etc.) weighs heavily in the final score. If these aspects are of no importance to you, the final score will be significantly altered.

My analysis of the Borg, Kopia and Restic solutions is now complete. Unfortunately, I'm no further ahead than before, as these solutions are relatively equivalent and all have attractive aspects. I still haven't decided whether to replace Borg with something else and, if so, with what.

Related posts

Borg, Kopia, Restic: restoration and maintenance

Borg, Kopia, Restic: backup and resource utilization

Borg, Kopia, Restic: from configuration to first backup

Borg, Kopia, Restic: functional scope and general information

[This is the English translation by DeepL]
[Version originale en français]

Restoration

From the previous article, we know how the backup process works in Borg, Kopia and Restic. Now we'll take a look at restoration and maintenance.
Restoration is a stage in the life of a backup, which - when all goes well - is never used. So it's understandable that the benchmark for this stage isn't overly interesting or representative. Tracing was done in the original backup script, but restoration was not performed every day.
The test consisted in restoring a 39 MB zip archive from the oldest snapshot available in the archive, as well as restoring a Library/Preferences directory from the oldest of the last 10 snapshots. The size of the directory varies from one backup to the next, but is generally around 290 MB for 951 directories and ~15K files.

During this restoration test, the difference between the three programs is once again very marked. Borg receives around 575 MB of data from the server. It sends around 6 MB of data. Kopia is a slightly better pupil, performing the same restoration with only 450 MB of data received and around 20 MB sent. But it's Restic that has the smallest network footprint, managing to perform the expected full restore with just 120 MB of data received and 0.5 MB of data sent.

In terms of restoration time, Restic is also in first place, with a task completed in 8 seconds, while Kopia takes between 12 and 14 seconds and Borg between 18 and 19 seconds. These are still relatively short times, and although the differences may be significant in proportion, they remain fairly small in absolute terms.

As far as restoration is concerned, Restic is the clear winner, well ahead of Kopia. Borg comes last, not least because of its network consumption.

Maintenance

The first maintenance task, and also the easiest to measure, is the "plum" phase. This task aims to remove obsolete archives, snapshots and restore points. It is essential to ensure that the backup repository does not explode out of control. It is this step that is responsible for ensuring compliance with the user's retention policy.
I detailed in my previous article how retention management can be full of surprises. I won't return to that particular point. Here, I'll focus on the resource consumption involved in these operations.
Note that Kopia spontaneously performs the cleanup operation when a new backup is launched, without needing to be asked to do so. To be able to measure this operation independently, I had to schedule it just before the backup operation (and not after, as initially planned).

The bandwidth consumption of the cleanup operation is surprising, to say the least. Borg is the software that stands out here for its great regularity, with around 180 MB of incoming data and 1.3 MB of outgoing data. For the other two, the situation is much more chaotic. Kopia seems to behave in a fairly stable way over time, with incoming data extremely limited to ~92 Kb and outgoing data to a volume of around 70 Kb, but there are some huge anomalies, with certain clean-up operations showing incoming transfers approaching 1.4 GB and outgoing transfers close to 1 GB. It seems that these peaks can be attributed to other maintenance tasks (see below).
Restic also shows great irregularity in the volumes of data exchanged with the server during cleaning operations. Nevertheless, volumes remain under control, well below the 40 MB mark for outgoing data and the 70 MB mark for incoming data.

4-day figures, mid-March

Over the duration of the cleaning stages, Borg and Kopia are relatively well-placed, with Kopia showing some significant peaks and Borg a certain regularity. Restic, on the other hand, is in last place, with a significant gap to these two competitors.

4-day figures, mid-March

To conclude on the subject of cleaning, I'll give first place to Kopia, as the anomalies observed are most certainly due to automatic snapshot/repository verification operations, which are handled automatically by this software.

Other maintenance tasks relate to checking backups and the backup repository. Like cleaning, checking snapshots and the health of the repository is handled automatically by Kopia (at least in theory). For Borg and Restic, these operations have to be planned in addition to the rest.
These operations are considered rare, some even exceptional (such as depot repair operations). I have therefore decided not to measure them over the long term, nor to draw any statistics from them. I've simply taken a few measurements on specific tasks.

Borg

$ time borg check --rsh "${SSH}" ${HOMEREPO}

real 8m36.258s
user 4m49.125s
sys 0m25.353s

This operation consumes 100% CPU on the client side (python) and 0% CPU on the server side.

$ time borg check --verify-data --rsh "${SSH}" ${HOMEREPO}

real 26m0.151s
user 15m9.882s
sys 2m37.629s 

This operation has phases with 100% CPU on the client side (python) and 1% CPU on the server side (python) and other phases with 50% python + 35% ssh on the client side and 50-60% sshd + 30-40% python on the server side.

$ time borg check --repair --repository-only --rsh "${SSH}" ${HOMEREPO}
This is a potentially dangerous function.
check --repair might lead to data loss (for kinds of corruption it is not
capable of dealing with). BE VERY CAREFUL!

Type 'YES' if you understand this and want to continue: YES

real 3m29.619s
user 0m0.252s
sys 0m0.088s

The latter loads almost exclusively the server: 0% CPU on the client side and 85% CPU on the server side.
Note that this repair was carried out in a vacuum, as the repository is in perfect health.

Kopia

$ time kopia maintenance run
Running full maintenance...
Looking for active contents...
Looking for unreferenced contents...
GC found 16436 unused contents (1.3 GB)
GC found 16550 unused contents that are too recent to delete (1.2 GB)
GC found 409930 in-use contents (73.3 GB)
GC found 28 in-use system-contents (71.6 KB)
Previous content rewrite has not been finalized yet, waiting until the next blob deletion.
Found safe time to drop indexes: 2024-04-06 15:51:09.649426 +0200 CEST
Dropping contents deleted before 2024-04-06 15:51:09.649426 +0200 CEST
Looking for unreferenced blobs...
  deleted 100 unreferenced blobs (2 GB)
Deleted total 138 unreferenced blobs (2.5 GB)
Cleaning up old index blobs which have already been compacted...
Cleaned up 8 logs.
Finished full maintenance.

real	0m9.883s
user	0m15.166s
sys	0m6.066s

$ time kopia maintenance run
Running full maintenance...
Looking for active contents...
Looking for unreferenced contents...
GC found 0 unused contents (0 B)
GC found 18882 unused contents that are too recent to delete (1.6 GB)
GC found 409930 in-use contents (73.3 GB)
GC found 23 in-use system-contents (56 KB)
Rewriting contents from short packs...
Total bytes rewritten 649.4 MB
Found safe time to drop indexes: 2024-04-06 15:51:09.649426 +0200 CEST
Dropping contents deleted before 2024-04-06 15:51:09.649426 +0200 CEST
Skipping blob deletion because not enough time has passed yet (59m59s left).
Cleaning up old index blobs which have already been compacted...
Cleaned up 0 logs.
Finished full maintenance.

real	0m27.121s
user	0m29.134s
sys	0m11.207s

Note the difference between the two runs, which are only a few seconds apart.

$ time kopia snapshot verify --verify-files-percent=10 --file-parallelism=10 --parallel=10
Listing blobs...
Listed 4141 blobs.
Processed 0 objects.
Processed 12137 objects.
Processed 17883 objects.
Processed 20581 objects.
../..
Processed 365793 objects.
Processed 368714 objects.
Finished processing 389361 objects.

real	3m26.325s
user	0m52.439s
sys	0m48.059s

Kopia has the interesting feature of being able to check only a percentage of the files in a backup (in this case 10%).
CPU consumption is distributed as follows: 30-60% CPU on client and server 80-90% CPU for sshd + 15-20% CPU for sftpd.

Checking 100% of files simply lengthens the operation:

$ time kopia snapshot verify --verify-files-percent=100 --file-parallelism=10 --parallel=10
Listing blobs...
Listed 4034 blobs.
Processed 0 objects.
Processed 4566 objects.
Processed 4577 objects.
Processed 4594 objects.
../..
Processed 372956 objects.
Processed 373370 objects.
Finished processing 393794 objects.

real	15m54.926s
user	5m37.337s
sys	4m3.843s

Restic

$ time restic -r rest:http://192.168.0.22:8000/ check
using temporary cache in /var/folders/1t/fbr0pvvc8xjg8f006s7cd4280000gp/T/restic-check-cache-2552920101
repository 9c062709 opened (version 2, compression level auto)
created new cache in /var/folders/1t/fbr0pvvc8xjg8f006s7cd4280000gp/T/restic-check-cache-2552920101
create exclusive lock for repository
load indexes
[0:00] 100.00%  6 / 6 index files loaded
check all packs
check snapshots, trees and blobs
[3:46] 100.00%  55 / 55 snapshots
no errors were found

real	3m48.649s
user	2m24.891s
sys	0m31.295s

$ time restic -r rest:http://192.168.0.22:8000/ check --read-data
using temporary cache in /var/folders/1t/fbr0pvvc8xjg8f006s7cd4280000gp/T/restic-check-cache-448461585
repository 9c062709 opened (version 2, compression level auto)
created new cache in /var/folders/1t/fbr0pvvc8xjg8f006s7cd4280000gp/T/restic-check-cache-448461585
create exclusive lock for repository
load indexes
[0:00] 100.00%  6 / 6 index files loaded
check all packs
check snapshots, trees and blobs
[3:46] 100.00%  55 / 55 snapshots
read all data
[10:36] 100.00%  4588 / 4588 packs
no errors were found

real	14m23.868s
user	6m55.821s
sys	1m45.116s

Data verification consumes 45-55% CPU on the client side and around 67% CPU on the server side.

$ time restic -r rest:http://192.168.0.22:8000/ repair snapshots
repository 9c062709 opened (version 2, compression level auto)
found 1 old cache directories in /Users/patpro/Library/Caches/restic, run `restic cache --cleanup` to remove them
[0:00] 100.00%  6 / 6 index files loaded

snapshot 09e89c9e of [/Users/patpro] at 2024-04-04 11:38:02.747322 +0200 CEST)

snapshot 0cf2fd62 of [/Users/patpro] at 2024-04-01 19:20:10.583797 +0200 CEST)

../..

snapshot fbeb283e of [/Users/patpro] at 2024-03-17 19:17:04.390071 +0100 CET)

snapshot fec2f8b1 of [/Users/patpro] at 2024-04-07 17:15:45.588862 +0200 CEST)

no snapshots were modified

real	3m22.149s
user	3m15.453s
sys	0m9.031s

Here, CPU consumption is around 100% on the client side and 0% on the server side. In principle, it works entirely on the client from the local cache. If the local cache is unavailable, Restic rebuilds it and server CPU consumption peaks at over 80%.

Overview
Relative to the number of snapshots/archives and the size of the repositories, the exhaustive verification times are as follows:

Borg and Restic are fairly close in terms of performance when it comes to verifying backed-up data, but Borg is a hair slower while consuming more CPU on the client. In the end, Kopia is much slower than the other two.

I briefly mentioned Restic above, and the same applies to the other two: these programs use a local cache system to speed up some operations. The size of this cache can be very important in some contexts, and seems to be proportional to the number of unique data blocks saved on the repository (at least this is the case for Borg). I'm unable to present reliable figures for Borg due to an oversight on my part: having not relocated Borg's cache for my experiment, it got mixed up with the cache of my other Borg repositories (those of my real backups), so its volume is not comparable with that of other softwares.
That said, here's some information:

The "Borg cache size, test, isolated late" corresponds to a cache for Borg, dedicated to my test script, created 2 days earlier. It immediately reached 828 MB and hasn't grown since.
In any case, Restic is the big loser here, and if we relate the size of the cache to the number of snapshots/archives on the repository, then Borg is the winner.

During this experiment, I was pleased to discover that Kopia manages maintenance operations for the user, which the other two programs have to launch explicitly. I also found Kopia's option to check a fraction of the files interesting. Finally, I like the fact that Restic offers an option for the cleaning operation, allowing the user to vary the cursor between performance and storage optimization.

Related posts

Borg, Kopia, Restic: backup and resource utilization

Borg, Kopia, Restic: from configuration to first backup

Borg, Kopia, Restic: functional scope and general information

Borg, Kopia, Restic: a comparison

[This is the English translation by DeepL]
[Version originale en français]

The functional scope of the three backup solutions is relatively equivalent. Each operates in "push" mode, i.e. the client initiates the backup to the server. This mode of operation is undoubtedly best suited to people who manage a small handful of machines. The "pull" mode is available for Borg, but requires a few adjustments.
All three programs use duplication and compression mechanisms to limit the volume of data transmitted and stored. With Kopia, compression is not activated by default. All three also implement backup retention and expiration management, repository maintenance mechanisms, etc.

Borgbackup

Borgbackup, or Borg, is a solution developed in Python. It is relatively easy to install, but does not allow Windows workstations to be backed up. Its documentation is relatively exhaustive, and its ecosystem includes automation and GUI solutions. It evolves regularly. Tests were carried out with version 1.2.7. Branch 1.4 (beta) is available and branch 2 is under active development.

To use Borg, the software must also be installed on the destination server. This is a constraint that may prevent some people from using Borg. Remote backup requires an SSH connection. In addition, access to the backup repository is exclusive (lock), i.e. it is not possible to back up several machines to the same repository at the same time. Furthermore, backing up multiple clients to the same repository is discouraged for various reasons.

Should the client machine be compromised, there is cause for concern that the backup software present on the machine will allow remote archives to be destroyed. By default, this is the case. Nonetheless, Borg's client-server operation allows you to add controls. It is possible, for example, to configure the client so that it can only add and restore files, but not delete archives.

All Borg settings are made using command-line arguments. There are no parameter files, although dedicated files can be used to deport lists of exclusions and inclusions, for example.

Official website: https://www.borgbackup.org/
Official documentation: https://borgbackup.readthedocs.io/en/stable/
Cooperative development platform: https://github.com/borgbackup/borg

Kopia

Kopia is a solution coded in GO. The version used here is 0.15.0 and the project is active. It is a monolithic binary available for different architectures and operating systems. Kopia is Windows-compatible and offers a GUI version. It also works with many different types of remote storage, such as S3, Azure blob Storage, Backblaze B2, WebDAV, SFTP and others. You don't need to install any server-side components to use Kopia.
It is also possible to back up several clients in the same repository. Deduplication can then be achieved across backups of all clients.

Kopia also features an optional "server" mode, which can be configured to add user management when users share the same repository, for example. This feature can also be used to limit the client's rights over stored archives, for example, by prohibiting their deletion. If you don't use the server part of Kopia, you won't be able to prevent a compromised client from deleting its backups.

Kopia lets you manage backup policies that are saved and then applied to backup jobs (or cleanup jobs, since these policies also include the notion of retention). This backup policy management is rather elegant, and allows you to limit the number of command line arguments each time a job is launched.

Official website: https://kopia.io/
Official documentation: https://kopia.io/docs/
Cooperative development platform: https://github.com/kopia/kopia/

Restic

Restic is also coded in GO. The version used here is 0.16.4 and the project is active. It is available for various architectures and operating systems, including Windows. The range of remote storage solutions supported by Restic is broadly similar to that of Kopia. Restic also offers its own high-performance HTTP server, which implements Restic's REST API. As with Kopia and Borg, this can ensure secure backups by limiting client rights.

This secure backup protocol, commonly referred to as "append-only", poses constraints on use that can be discouraging for users. Indeed, backing up machines to repositories on which old backups cannot be deleted poses a problem of volume management. For my part, I'd much prefer the client to have full rights over the backups, but for the repository of these backups to be stored on a file system that allows regular snapshots to be generated. In my case, all backups are written to a ZFS file system and automatic snapshots are taken every day with relatively long retention times.

Like Borg, Restic relies exclusively on command-line parameters to configure the various tasks.

Official website: https://restic.net/
Official documentation: https://restic.readthedocs.io/en/stable/index.html
Cooperative development platform: https://github.com/restic/restic

In a nutshell

Everyone has to make up their own mind according to their own needs, but if you don't rule out any scenario (Windows client backup, shared repository or S3 storage...), Borg doesn't have the edge. If, like me, you back up a handful of macOS/FreeBSD/Linux machines to servers on which you can install software and to which you can connect via SSH, then Borg is on a par with Kopia and Restic.

Related posts

"builtin" : quelques subtilités

Quelques notions sur les boucles dans bash

Gadget Apple pour le monitoring : BigTop

Les variables dans Bash

[This is the English translation by DeepL]
[Version originale en français]

I've been using Borg Backup for several years now. However, at the end of 2023, I encountered major problems with the backup of one of my machines. So I decided to explore alternative solutions. A quick search revealed two solutions that were relatively equivalent to Borg Backup: Restic and Kopia.

In the absence of a comparison of these three solutions (there is an "old" comparison between Borg and Restic, but it doesn't go as far and doesn't mention Kopia), I decided to carry out some tests myself, to be able to compare simple metrics common to all three software packages.

Finally, once a regular backup is automated, the important metrics for me are: backup duration, network exchange volume and storage volume on the backup server. Similar metrics are worth exploring for backup restore operations and backup server maintenance operations.

My test environment consists of the following components:

• client: macmini M2 Pro running macOS 13.6.3, 32 GB RAM
• backup server: Intel(R) Core(TM) i3-3220T-based PC, running FreeBSD 14, 16 GB RAM, ZFS storage on SSD
• switched 1Gbps ethernet network
• Splunk instance to ingest logs and produce statistics and graphs

To make sure I was working with representative data, I decided to back up part of my user directory. I've excluded a huge directory of digital photos that rarely changes, as well as numerous cache directories that it's not relevant to back up. The final size of my backup target is around 140GB.

I based my backup protocol on my experience with Borg. However, as each software has its own specificities, it took some time to document and adapt. In spite of these preparations, the backup process did have a few surprises in store for me.

Backups were automated using a single bash script, which launched the creation of backups via Borg, Restic and Kopia in random order. I had in mind that the first backup task would be potentially penalized by access to files on the client workstation that were not in the system cache, but after 15 days of testing, I found that there was no visible difference in backup time attributable to the order of the backup tasks. I therefore subsequently decided to freeze the order of backup jobs, which also makes it much easier to track metrics.

The client machine has scheduled sleep periods at night. During these sleep periods, no backup takes place. In addition, the script is scheduled in Launchd by setting the StartInterval argument to 3600 seconds. Unlike a crontab schedule, for example, this one ensures an incompressible delay of one hour between two script executions. In other words, one hour must elapse between the end of the previous execution and the start of the new one. This means that the script is actually run just under once an hour. For example, if each execution lasts 5 minutes, then it takes 26 hours, not 24 hours, to run the script 24 times.

Important note: I'm not going to go into detail in these articles about the differences in the software's ergonomics, the availability or otherwise of a graphical management interface, etc. Some of the solutions presented here have several different graphical interfaces, the functionalities of the three products vary, and I don't wish to make a catalog comparing each software point by point.

Here's the table of contents of my comparison. The link to each article will become active upon publication.

• Functional scope and general information
• From configuration to first backup
• Backup and resource utilization
• Restoration and maintenance
• Going further

Related posts

"builtin" : quelques subtilités

Quelques notions sur les boucles dans bash

Gadget Apple pour le monitoring : BigTop

Les variables dans Bash