Sysadministrivia

Linux, Lagers, and Late Nights

HOWTO: Set up a Locked-Down Borg Server

Posted
Modified
Comments 0

I’m sitting here around 0300 after dental surgery the previous day, and I don’t feel like working on actual work because I can’t focus enough. I realized “hey, I forgot to write up that article I said I would in S4E5!” Specifically, how to set up a remote borgbackup server in such a way that it prevents giving unwanted access and preventing machines/users from accessing others’ backups.

So! Without ado.

Why?

Using the methods outlined here, you will be able to provide a minimal-trust backup environment:

  • Backup users will not be able to execute commands on the backup server
  • The authentication for backup users vs. “real” (system/administrative) users is segregated
  • Machine A cannot access backups for Machine B (unless explicitly enabled)

Prep

Hardware

You’ll need a box that’s WAN-accessible via SSH and has a lot of storage space available. It should be remote, because it’s good practice for contingency planning (bit pointless to have backups on a local machine and some sort of natural disaster occurs, right?).

Distro

Distro doesn’t matter too much here, but we recommend RHEL or CentOS on the newest release available. Namely being that it’s easy to forget to perform updates sometimes on your backup server, so having SELinux available is going to help provide an additional stop in security. (We talk about SELinux in S1E6, S2E6, and S3E19).

Software

So do your install, ensuring that SELinux is set to enforced/enforcing, get your network set up, harden your SSH, and install borgbackup (it’s in EPEL in recent versions of RHEL/CentOS, you can simply do yum -y install epel-release && yum -y install borgbackup – it should be in the repos for all major distros at this point, however). You’ll need borgbackup installed on all the backup clients as well. You’ll also want to install policycoreutils-python on the server (yum -y install policycoreutils-python).

You may want to install and configure a client-side wrapper I wrote, BorgExtend, as well. The default config is heavily commented; customize it to your own desires and place it in ~/.config/borgextend/backup.xml on the client for the user that will be performing backups (9 times out of 10, due to permissions, this is going to be the root user). I recommend setting it as mode 0600, as it will contain passphrases.

For our purposes, we will assume that borgextend is installed on the clients, and have the following config in /root/.config/borgextend/backup.xml:

<?xml version="1.0" encoding="UTF-8" ?>
<borg xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xmlns="http://git.square-r00t.net/BorgExtend/tree/storage/backups/borg/"
      xsi:schemaLocation="http://git.square-r00t.net/BorgExtend/plain/config.xsd">
    <server target="YOUR_BACKUP_SERVER_HERE" remote="true" rsh="ssh -p 2222" user="someuser">
        <repo name="testrepo" password="SuperSecretPassword" compression="lzma,9">
            <path>/root</path>
            <exclude>/root/.ssh/known_hosts</exclude>
        </repo>
    </server>
</borg>

Configuration

SSH Alternative Port

In order to really do things well, here, you need to run two (yes, Virginia, two) SSH daemons:

  • an “administrative” SSH daemon
  • a “backup-dedicated” SSH daemon

This is done so we can ensure different privilege sets (only root, or a specific sudo-enabled user, can log into the “administrative” SSH, and only backup users can log into the SSH daemon for backups). In the example configs I provide, it doesn’t matter so much which port is used for which, but I’ll use 22/TCP for the “administrative” SSH and 2222/TCP for the “backups” SSH.

Before we get started, though, we need to ensure that we grant SELinux permissions to run SSH on an alternative port.

semanage port -a -t ssh_port_t -p tcp 2222

(Remember to change 2222 to whatever your second port will be if not 2222.)

Then, create a copy of the hardened SSH config you made above:

cp -a /etc/ssh/sshd_config /etc/ssh/sshd_backups_config

And edit it with the following changes (in diff -u format; you can save the below to /tmp/sshd.patch, install the patch utility and do patch < /tmp/sshd.patch):

--- /etc/ssh/sshd_backups_config	2019-08-15 09:29:06.793313270 +0000
+++ /etc/ssh/sshd_backups_config	2019-08-15 09:43:58.277737977 +0000
@@ -14,7 +14,7 @@
 # SELinux about this change.
 # semanage port -a -t ssh_port_t -p tcp #PORTNUMBER
 #
-#Port 22
+Port 2222
 #AddressFamily any
 #ListenAddress 0.0.0.0
 #ListenAddress ::
@@ -77,7 +77,7 @@

 # GSSAPI options
 GSSAPIAuthentication yes
-GSSAPICleanupCredentials no
+GSSAPICleanupCredentials yes
 #GSSAPIStrictAcceptorCheck yes
 #GSSAPIKeyExchange no
 #GSSAPIEnablek5users no
@@ -95,26 +95,26 @@
 # problems.
 UsePAM yes

-#AllowAgentForwarding yes
-#AllowTcpForwarding yes
+AllowAgentForwarding no
+AllowTcpForwarding no
 #GatewayPorts no
-X11Forwarding yes
+X11Forwarding no
 #X11DisplayOffset 10
 #X11UseLocalhost yes
 #PermitTTY yes
-#PrintMotd yes
-#PrintLastLog yes
+PrintMotd no
+PrintLastLog no
 #TCPKeepAlive yes
 #UseLogin no
 #UsePrivilegeSeparation sandbox
-#PermitUserEnvironment no
+PermitUserEnvironment no
 #Compression delayed
-#ClientAliveInterval 0
-#ClientAliveCountMax 3
+ClientAliveInterval 10
+ClientAliveCountMax 3
 #ShowPatchLevel no
-#UseDNS yes
-#PidFile /var/run/sshd.pid
-#MaxStartups 10:30:100
+UseDNS no
+PidFile /var/run/sshd_backups.pid
+MaxStartups 100:60:200
 #PermitTunnel no
 #ChrootDirectory none
 #VersionAddendum none
@@ -145,9 +145,13 @@
 Protocol 2
 HostKey /etc/ssh/ssh_host_rsa_key
 HostKey /etc/ssh/ssh_host_ed25519_key
-PermitRootLogin without-password
+PermitRootLogin no
+
 PasswordAuthentication no
 ChallengeResponseAuthentication no
 PubkeyAuthentication yes
 Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
 MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
+
+DenyUsers root
+ForceCommand /usr/local/bin/borg-restricted.py ${SSH_ORIGINAL_COMMAND}

We also need to restrict the administrative access: echo 'AllowUsers root' >> /etc/ssh/sshd_config (obviously replacing root with whatever user you use for sudo access if desired, or use AllowGroups, etc. The important thing to note is you use a matching DenyUsers and/or DenyGroups in /etc/ssh/sshd_backups_config).

systemd unit

RHEL and CentOS 7.x and above use systemd. Moden Debian, Ubuntu, and Arch systems do as well — most modern distros do. This example provided is RHEL/CentOS-specific, however; it should be easy enough to figure out for your distro. You probably shouldn’t be a sysadmin if it isn’t.

You need to create a custom systemd unit that will read the config you just created:

cp -a /usr/lib/systemd/system/sshd.service /etc/systemd/system/sshd_backups.service
sed -i -re 's/^(EnvironmentFile=.*)$/\1_backups/g' /etc/systemd/system/sshd_backups.service
cp -a /etc/sysconfig/sshd /etc/sysconfig/sshd_backups
echo 'OPTIONS="-f /etc/ssh/sshd_backups_config"' >> /etc/sysconfig/sshd_backups
systemd enable sshd_backups
systemd start sshd_backups

Restricted borgbackup script

And then, of course, we need to create /usr/local/scripts/borg-restricted.py:

touch /usr/local/bin/borg-restricted.py
chmod 755 /usr/local/bin/borg-restricted.py
curl https://git.square-r00t.net/BorgExtend/plain/tools/borg-restricted.py > /usr/local/bin/borg-restricted.py

You’re almost done!

Adding users/machines

Typically, you will want just one account per client machine. There is a handy script you can use for this (on the backup server) here.

touch /usr/local/bin/add-borguser.py
chmod 700 /usr/local/bin/add-borguser.py
curl https://git.square-r00t.net/BorgExtend/plain/tools/add-borguser.py > /usr/local/bin/add-borguser.py
add-borguser.py -h

To add a user, you simply do e.g.:

add-borguser.py someuser "ssh-ed25519 BASE64_KEY_STR_HERE"

And voila! The client machine is ready to go.

Performing a backup

It is now time to test creating a backup. Assuming you have configured your ~/.config/borgextend/backup.xml, etc. and backup.py (from BorgExtend) is in your $PATH, you can simply do:

backup.py init  # This only needs to be done when first adding a repository
backup.py backup

On the client machine.

To make sure that it backed up correctly,

backup.py list

And you should see a snapshot show up!

For bonus points…

Consider setting up a cron that does a chattr +i on the backup server for previous archives (/home/<USER>/<REPO_NAME>/data/) at the end of a backup run. This allows clients to not be able to delete their own backups. Great for DFIR/forensics and untrusted client purposes!

Author
Categories

Comments

There are currently no comments on this article.

Comment...

Enter your comment below. Fields marked * are required. You must preview your comment before submitting it.