S4E5: "Join the Borg? Restic is Futile"

Posted 2019-04-29 03:59
Modified 2019-08-18 13:48
Comments 8

Borg vs. Restic for performing backups.

• Just the Tip
• Notes
• Results
  • Single File
    • Repository Initialization
    • Backup (Non-Compressed)
      • Borg Compression (1GB File)
    • Differential (100MB change) (single file)
    • Restoring/Pulling a Backup
  • Multiple Files
    • Backup
    • Differential
    • Restoring/Pulling a Backup
• 15 Clams
• Errata
• Music

Just the Tip

• Per-user resource limits
  • ulimit is dead. Jthan says it isn’t but he’s probably the type of person that still uses net-tools instead of iproute2.
  • cgroups are the way to go.
    • They can be used for a lot of different resources.
    • There is also a newer, systemd-explicit management.
      • You can access the RHEL 8 beta for free.

Notes

Starts at 15m40s.

I was drinking Victory DirtWolf again (and a Snapple Apple). Paden was drinking Absolut Vodka. Jthan was drinking Nikka Whisky From the Barrel.

• Borg vs. Restic
  • Paden:
    • Borg:
      • Smaller resulting snapshot (even with no compression enabled)
      • Requires an archive/snapshot name (restic generates checksum-based IDs)
    • Restic:
      • Faster
      • Progress display by default (borg requires -e or --verbose)
  • Me (Brent):
    • Borg:
      • Has wider maintenance base; more people working on it.
      • In python; easier to directly access. No public python API, but written cleanly enough to import and directly interact with.
      • Tends to have features before restic copies them (e.g. append-only mode – borg had this way before restic did).
      • The remote server forces use of SSH tunneling via piping, thus a secure and built-in authentication method (with more flexible ssh connection options than restic)
      • One would need to write their own REST API with Flask or the like.
      • The same binary (python script) is used for server and client; this is not the case for restic.
      • Several different backup protection (encryption-at-rest) models available.
      • Borg has support for reporting etc. in JSON, which can help with interacting in a programmatic manner.
    • Restic:
      • Has an HTTP REST API, which can be nice for maintenance…
        • Except it’s the default method for backing up. which means backup traffic over HTTP. SFTP is supported, but most of the emphasis for self-hosted remote backup servers is HTTP.
      • Written in Go, which has a fraction of accessibility of python.
      • It does not force HTTPS or auth for the server component.
      • It is not packaged for CentOS/RHEL; it requires COPR (gateway for third-party repos/packages); borg is in EPEL for CentOS/RHEL 7.
      • rest-server hasn’t had a release since Feb 2018, and examples in their docs do not match latest release, just master:HEAD.
      • No compression.
  • Jthan:
    • If you have data that will be shared across repositories or if you are using a cloud-storage provider (the bigger names), restic has some benefits.
    • He liked the nomenclature that restic uses more.

Results

Please reference the tables below for results to tests that I (Brent) performed.

Single File

A test backup file of 1GB of random data was created via dd if=/dev/urandom of=junkdata.dat bs=32M count=32 iflag=fullblock.

Repository Initialization

Repositories were initialized via the following:

export RESTIC_PASSWORD="test"
export BORG_PASSPHRASE="test"
export BORG_RSH="ssh -l restic -p 22"
restic -r sftp:restic@[server]:testrepo1 init
restic -r rest:http://test:test@[server]:8000/testrepo2 init
borg init -e repokey restic@[server]:testrepo3

New repositories were created fresh for all tests (except for tests that were based on previous existing snapshots for e.g. differentials).

Backup (Non-Compressed)

Backups were then performed via:

restic backup -r sftp:restic@[server]:testrepo1 junkdata.dat
restic backup -r rest:http://test:test@[server]:8000/testrepo2 junkdata.dat
borg create restic@[server]:testrepo3::test1 junkdata.dat

Borg Compression (1GB File)

The difference in time consumption when using the highest compression was then tested with borg with a single 1GB file. A new repository was created and initialized (testrepo4), and the command:

borg create --compression lzma,9 restic@[server]:testrepo4::test1 junkdata.dat

was used. This process took 5m24.973s, which is a big difference from the 1m36.724s it took the uncompressed snapshot for the same data file. However, do keep in mind this is for binary data which does not compress very well.

Differential (100MB change) (single file)

A change of 100MB of binary data was then created via dd if=/dev/urandom bs=100M count=1 iflag=fullblock >> junkdata.dat. The compression test for borg was not done (as this would be inaccurate representation for size, since restic does not support compression).

The following commands were executed:

restic backup -r sftp:restic@[server]:testrepo1 junkdata.dat
restic backup -r rest:http://test:test@[server]:8000/testrepo2 junkdata.dat
borg create restic@[server]:testrepo3::test2 junkdata.dat

Restoring/Pulling a Backup

The restoration of a backup for a single ~1.1GB file was then compared.

Commands used:

mkdir restore
cd restore
restic -r sftp:restic@[server]:testrepo1 restore latest --target restic_sftp
restic -r rest:http://test:test@[server]:8000/testrepo2 restore latest --target restic_http
borg extract restic@[server]:testrepo3::test3

Multiple Files

For this test, it was attempted to mimic more realistic backup data – the number of files was changed from 1×1GB to 1000×1MB, and were text-only (to emulate logs, config files, MySQL dumps, LDAP LDIF dumps, etc.). You can find the script I wrote to generate the test text files here. Borg compression was tested in this case for comparison (as text compresses better than binary data, so the differences would be more easily noticed). Intialization was done the same as before, with the addition of a second borg repository for separate compression testing:

export RESTIC_PASSWORD="test"
export BORG_PASSPHRASE="test"
export BORG_RSH="ssh -l restic -p 22"
restic -r sftp:restic@[server]:testrepo1 init
restic -r rest:http://test:test@[server]:8000/testrepo2 init
borg init -e repokey restic@[server]:testrepo3
borg init -e repokey restic@[server]:testrepo3_compressed

Backup

restic backup -r sftp:restic@[server]:testrepo1 /var/tmp/dirtest
restic backup -r rest:http://test:test@[server]:8000/testrepo2 /var/tmp/dirtest
borg create restic@[server]:testrepo3::test1 /var/tmp/dirtest
borg create --compression lzma,9 restic@[server]:testrepo3_compressed::test1 /var/tmp/dirtest

Differential

restic backup -r sftp:restic@[server]:testrepo1 /var/tmp/dirtest
restic backup -r rest:http://test:test@[server]:8000/testrepo2 /var/tmp/dirtest
borg create restic@[server]:testrepo3::test2 /var/tmp/dirtest
borg create --compression lzma,9 restic@[server]:testrepo3_compressed::test2 /var/tmp/dirtest

Restoring/Pulling a Backup

It’s worth noting that restic via HTTP REST actually failed this test, giving the following errors:

ignoring error for /var/tmp/dirtest/0/1/8: decrypting blob 4cb6d10fbd7ca23759d050f117255f70ee73be74d2e5b494eee8a5a4e593f8c9 failed: ciphertext verification failed
ignoring error for /var/tmp/dirtest/6/3/2: decrypting blob c743cbe9f85d28bb4eb9d06b8687fc6463596db1f9955a8b8d78a8f8d0c32a75 failed: ciphertext verification failed
ignoring error for /var/tmp/dirtest/9/4/7: decrypting blob 726423b01e90f98b294a7f3e360a6f0038812e1e2416a6c30c750d554d7acad6 failed: ciphertext verification failed
ignoring error for /var/tmp/dirtest/9/3/6: decrypting blob 61b5e1a10587c44205c83d75b6b13db4dc1b9d0e09541dfc7563ee2e46eae112 failed: ciphertext verification failed
ignoring error for /var/tmp/dirtest/9/4/5: decrypting blob 00022b138ea6057ca3a9392b35de85f4eaa2f389052c07ba8e02b6fbc42177e9 failed: ciphertext verification failed
ignoring error for /var/tmp/dirtest/3/2/7: decrypting blob 6c799cffa51bb1c4341df2665654e9e83fa6b7dda44b1064a4ec9b9785f7b370 failed: ciphertext verification failed
ignoring error for /var/tmp/dirtest/7/2/7: decrypting blob 3dc9a3b5690f134506a8ce5a6a33a3b4118b3784ffc92caea7641e8b1523a3b4 failed: ciphertext verification failed
ignoring error for /var/tmp/dirtest/0/1/8: UtimesNano: no such file or directory
ignoring error for /var/tmp/dirtest/3/2/7: UtimesNano: no such file or directory
ignoring error for /var/tmp/dirtest/7/2/7: UtimesNano: no such file or directory
ignoring error for /var/tmp/dirtest/9/3/6: UtimesNano: no such file or directory
ignoring error for /var/tmp/dirtest/9/4/5: UtimesNano: no such file or directory
ignoring error for /var/tmp/dirtest/9/4/7: UtimesNano: no such file or directory
There were 13 errors

So. There’s that. But here’s the results, keeping that in mind:

restic -r sftp:restic@[server]:testrepo1 restore latest --target testrepo1/sftp
restic -r rest:http://test:test@[server]/testrepo2 restore latest --target testrepo2/http
borg extract restic@[server]:testrepo3::test2
borg extract restic@[server]:testrepo3_compressed::test2

15 Clams

In this segment, Jthan shares with you a little slice of life. The title is a reference to this video. (2m16s in)

Starts at 1h20m10s.

Jthan has been trying to turn up Vault (we talk about it in S4E0). Jthan wants to know why time-based access for protecting digital resources as a security layer is pointless.

Errata

• The “Jthan is a Neopet“ reference is because he kept calling himself a compukoala during pre-show and I said “that sounds like a neopet”.
• No, Jthan, that actually is NOT a “double”.
• Jthan wants you to visit git.jthan.io.
• This is Jthan’s award he got for the podcast.
• I’ll write up an article on how to set up a hardened central borg server in about a week or so. I finally wrote one up!
• Jthan asked about whether borg can be configured to mark certain snapshots (“archives”) as “read-only”, or persistent. You can’t do this on a per-archive level, but you CAN do it on a per-repo/per-pubkey level.
• The “how/where do you want to pay?” discussion is in S4E3.
• Paden, it is indeed 100% egg man.
• Thanks to amayer for catching a typo in the shownotes! You da man.

Music

Author r00t^2
Categories Season Four