Ubuntu Server btrfs Setup

I wanted to practice setting up btrfs on Ubuntu server. My requirement is nightly backups retained for 30 days. I started with a 10GB virtual disk on a VirtualBox VM. I partitioned it with the following table:

/boot      ext4    200MB
/          ext4   5000MB
/home      btrfs  5000MB
swap       swap    537MB

First boot, Sun Oct 22 19:36:46 MDT 2017

Filesystem      Size  Used Avail Use% Mounted on
udev            477M     0  477M   0% /dev
tmpfs           100M  3.2M   97M   4% /run
/dev/sda2       4.5G  1.4G  2.9G  33% /
tmpfs           497M     0  497M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           497M     0  497M   0% /sys/fs/cgroup
/dev/sda3       4.7G   17M  4.2G   1% /home
/dev/sda1       180M   57M  111M  34% /boot
tmpfs           100M     0  100M   0% /run/user/1000

I installed tools for doing automatic backups with btrfs https://github.com/digint/btrbk
sudo apt install btrbk

I created a configuration file using their 'local time-machine' example as a guide.

ryan@ubuntu:~$ cat /etc/btrbk/btrbk.conf
transaction_log            /var/log/btrbk.log
snapshot_dir               _btrbk_snap

timestamp_format           long
snapshot_preserve_daily    30
snapshot_preserve_weekly   0
snapshot_preserve_monthly  0

volume /mnt/btr_pool
  snapshot_dir btrbk_snapshots
  subvolume @home

I had to mount the btrfs main volume under /mnt/btr_pool to the fstab:
ryan@ubuntu:~$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
#              
# / was on /dev/sda2 during installation
UUID=6ae1fc17-428e-4446-941c-f478c71b9cfd /               ext4    errors=remount-ro 0       1
# /boot was on /dev/sda1 during installation
UUID=683a6648-1598-44d3-930e-62aba3b8a4a5 /boot           ext4    defaults        0       2
# /home was on /dev/sda3 during installation
UUID=50e15cf1-f4d8-4ac3-9116-dfea86eb7c33 /home           btrfs   defaults,subvol=@home 0       2
# swap was on /dev/sda4 during installation
UUID=ace0597d-3b9f-45ca-bab2-9a49ff6dbe51 none            swap    sw              0       0
# mount btrfs for backup
UUID=50e15cf1-f4d8-4ac3-9116-dfea86eb7c33 /mnt/btr_pool   btrfs   defaults,subvolid=0 0       0

/etc/cron.daily/btrbk:

#!/bin/sh
exec /usr/sbin/btrbk -q -c /etc/btrbk/btrbk.conf run

After creating some directories, the backups appear as:
ryan@ubuntu:~$ ls /mnt/btr_pool/btrbk_snapshots/
@home.20171022T2149  @home.20171022T2216    @home.20171022T2223
@home.20171022T2151  @home.20171022T2216_1  @home.20171022T2229
@home.20171022T2152  @home.20171022T2216_2  @home.20171022T2231

I had to change the VirtualBox network adapter type and install openssh in order to get my notes off of the VM http://wiredrevolution.com/virtualbox/setup-ssh-access-between-virtualbox-host-and-guest-vms

It might be a good idea to start using git in the /etc directory to keep track of changes. More info: https://unix.stackexchange.com/a/72764

I should try this again, but combining the / and /home partitions into one btrfs partition so that things like /etc and /var can also be tracked.

Data/Documents backup script

I used to always create directory under my home called data. I'd store files which I need to do work and my working files. Then, this would be the only directory I had to synchronize with Unison (http://www.cis.upenn.edu/~bcpierce/unison/)between machines. Since Fedora has included a directory called Documents, which has a similar purpose (and so has Mac OS X), I've just changed the name of this special directory from data to Documents.

That brings me to the topic of this article--how do I manage if I mess up a file in this directory, and I synchronize the mistake to other mirrors? That would mean none of the computers I work on has a copy anymore. The answer is rdiff-backup (http://www.nongnu.org/rdiff-backup/). I used to make just a user cron job to call rdiff-backup and update an incremental backup repo on my behalf, but I wanted this to be more mechanical (what if I added a new user, etc.). So I wrote the following script which replaces an old script I used to have which would only make a mirror with rsync.

Note that, on my configuration, the incremental backup repo is on the same device as the original files, but both are mirrored in an overnight backup job. This means I have (at least) four mirrors of the same files, which is sort of a waste but I don't really worry about it since it's about 200 MB / user. Ideally, you would keep just the local mirror on the disk, and the rdiff-backup repository on the backup device (or system over network). The downside here would be, if the backup device (or remote system) crashes you loose the ability to restore a previous increment.

Following is the script for maintaining incremental backup repositories with rdiff-backup for all users. Caution: the backup path should be as read-only as possible, but it is not if the user logs in to the system. For this reason, you'll have to count on them not to modify the files, or keep the files on a filesystem they cannot mount. For simple remote access over samba (the kind of access in my setup), you would have to configure a per-user share (so Joe can't see Susan's backup, for instance) and make each one read-only (so Joe cannot corrupt his repository by changing the files there).

#!/bin/sh
#
# Make incremental backups of 'Documents' directories for users
#
# Ryan Helinski

LOGFILE=/var/log/rdiff-backup.log

# Close stdout and stderr
1>&-; 2>&-;

# Direct stdout and stderr to logfile
exec 1>>$LOGFILE; exec 2>>$LOGFILE;

echo Begin $0, `date`;

# For debugging
#set -x

# Script-specific configuration
HOMESPATH=/home
BACKUPPATH=/opt/backup
RDIFF_OPTIONS="--print-statistics"
DOCSDIR=Documents

# Max increment life 
# The time interval is an integer followed by the character s, m, h, D, W, 
# M, or Y, indicating seconds,  minutes, hours,  days,  weeks,  months, or 
# years respectively, or a number of these con-catenated.
MAX_LIFE=4W

for USERHOME in $HOMESPATH/* ; do

# USERDIR=`basename $USERHOME`;
 USER=`stat -c "%U" $USERHOME`;

 if [ -d "$USERHOME/$DOCSDIR" ] ; then
  echo "Updating $BACKUPPATH/$USER/$DOCSDIR";

  # Create backup repo if it doesn't exist
  if [ ! -d "$BACKUPPATH/$USER/$DOCSDIR" ] ; then
   echo "Creating rdiff-backup repo";
   mkdir -p "$BACKUPPATH/$USER/$DOCSDIR";
   chown $USER "$BACKUPPATH/$USER";
   chown $USER "$BACKUPPATH/$USER/$DOCSDIR";
  fi

  # Update repo
  rdiff-backup $RDIFF_OPTIONS "$USERHOME/$DOCSDIR" \
   "$BACKUPPATH/$USER/$DOCSDIR" ;

  # Clean up repo
  #rdiff-backup --remove-older-than $MAX_LIFE --force \
  # "$BACKUPPATH/$USER/$DOCSDIR" ;

 fi

done

echo End $0, `date`;
echo

exit

To actually remove old backups, uncomment the code in the "Clean up repo" section.

Using the Seagate FreeAgent as a (periodic) Mirror Backup

I recently purchased a Seagate FreeAgent 100D USB drive. The drive is quite nice, small, quiet, and has advanced power management. I plugged the device in on a Fedora 7 installation, and it came right up.

The first problem I had was the NTFS partition that came on the device. This is easily fixed with fdisk and mkfs.ext3. Don't forget to use e2label, especially if you have more than one of the same device.

The next thing you want to do is determine the "id" of the device, so that you can uniquely identify it. The problem is that udev sets up devices in order of connection (/dev/sda, /dev/sdb, ...). To remedy this, use the symbolic links that udev creates under /dev/disk/ to identify the disk by one of the following means:

• /dev/disk/by-id/ - Uses connection, make, model, and serial number


• /dev/disk/by-uuid/ - Uses the UUID given to the partition when it was created


• /dev/disk/by-label/ - If you have used e2label, this is more meaningful than the UUID


• /dev/disk/by-path/ - This is the one you do not want to use!

I want to disable normal users (including myself) from modifying the contents of the backup, so the plan is to keep it unmounted and disable users from mounting it. Thefefor, this is the appropriate entry for me in /etc/fstab:

/dev/disk/by-id/usb-Seagate_FreeAgentDesktop_30DFK39D-0:0-part1 /media/FreeAgent        ext3    defaults        1 2

Note this will require the device to be present and consistent at boot-time. If you don't need this, change the last two numbers in the fstab entry to zero.

Before automating backup, I had a problem: The drive spins down automatically, so if you use only a command like rsync it may terminate due to an I/O Error because it simply timed out. This happened with me using ls, but it doesn't always happen -- so you should be more paranoid about the power mode.

The solution is to use a utility called sdparm, which is made to control SCSI (and SATA) disks. This is available for Fedora in a yum package called simply sdparm. This allows you to send a command to the drive to "start" (spin up) or "stop" (spin down).

Then, the final task is to create a backup procedure that is invoked by crond. Rather than writing something to /etc/crontab, I added the following file to /etc/cron.daily/. Make sure it has execute permissions, and call it (as root) a couple times to make sure it's working.

#!/bin/sh
#
# Backup to a USB disk 
#

# Env variables 
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/

# OK to change these
LOGFILE=/var/log/backup-usb.log
DISKDEVICE=/dev/disk/by-id/usb-Seagate_FreeAgentDesktop_30DFK39D-0\:0
DISKPARTITION=$DISKDEVICE-part1
MOUNTPOINT=/media/FreeAgent

# Leave these alone
DEVICEFILE=`readlink -f $DISKDEVICE`
PARTITIONFILE=`readlink -f $DISKPARTITION`

# Close stdout
1>&-;
# Close stderr
2>&-;

# Log needs to be rotated manually
#mv $LOGFILE $LOGFILE.1

# Direct stdout and stderr to logfile
exec 1>>$LOGFILE;
exec 2>>$LOGFILE;

echo Begin $0, `date`;

# Preliminary work (get backup device online)
if [ -e $DISKDEVICE ] ;
then
 echo "Device exists (is connected).";

# /usr/bin/sdparm --all $DISKDEVICE

 echo "Sending start (wake up) signal...";
 /usr/bin/sdparm --command=start $DISKDEVICE
 if [ "$?" -eq "0" ] ; 
 then 
  echo "Success";
 else 
  echo "Failed to start device";
  exit;
 fi

 # Get backup partition mounted
 if [ `grep $PARTITIONFILE /etc/mtab | wc -l` -le "0" ] ;
 then
  echo "Device $PARTITIONFILE is not mounted!";
  mount $PARTITIONFILE $MOUNTPOINT;
  if [ "$?" -eq "0" ] ;
  then
   echo "Mounted OK.";
  else
   echo "Failed to mount.";
   exit;
  fi
  
 else 
  echo "Partition $PARTITIONFILE is already mounted.";
 fi
else
 echo "Device doesn't exist, must not be connected!";
 exit;
fi

# Backup procedure 
# 
# For now I'm just using RSYNC because these files don't often change
# 
rsync --verbose --itemize-changes \
--archive --hard-links --partial --delete-before \
/opt/srv/ $MOUNTPOINT/srv/ && \
date && \
df -h $MOUNTPOINT && \
echo ;

# NOTE might also want to un-mount the device at this point 
# to prevent users from modifying it directly !
echo "Unmounting $PARTITIONFILE"
umount $PARTITIONFILE
echo "Stopping device $DISKDEVICE"
/usr/bin/sdparm --command=stop $DISKDEVICE
# /usr/bin/sdparm --all $DISKDEVICE && \

echo End $0, `date`;

The variables section and the rsync command have to be updated to fit your installation and your backup scheme.

Again, this worked well for me, you may have to make adjustments.

Nightly Backup Script

If you're working on a small project, you might want to do a simple nightly backup. The following Bash Shell Script, when copied into your /etc/cron.daily/ directory will do the trick. You might have to allow read access for 'other' users for everything you want to back up. It worked without changing that for me.

Obviously, you'll have to change the file and directories. This is just an example.

[ryan@a564n cmpe646]$ cat cmpe646_backup.sh
#!/bin/sh

date=$(date +%s)

tar -cf /home/cmpe646/cmpe646_project_backup_$date.tar /home/cmpe646/*/*.h* /home/cmpe646/*/*.c* /home/cmpe646/*/*.bench*
bzip2 /home/cmpe646/cmpe646_project_backup_$date.tar

exit 0