Encrypted FS with LUKS

Partitioning a Seagate FreeAgent

Outline

Check for bad blocks (not required)

badblocks -c 10240 -s -w -t random -v /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0

Partition drive. In this case, I create one 'Linux' partition which spans the whole drive.

cfdisk /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0

Create a LUKS device (partition) on the drive

cryptsetup --verbose --verify-passphrase luksFormat /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1

Open that device

cryptsetup luksOpen /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1 FreeAgent

Create an EXT partition on the device.

mkfs.ext3 -j -m 1 -O dir_index,filetype,sparse_super /dev/mapper/FreeAgent

These options aren't necessary. The following will also work fine:

mkfs.ext3 /dev/mapper/FreeAgent

Mount the encrypted partition

mkdir /media/FreeAgentLuks
mount /dev/mapper/FreeAgent /media/FreeAgentLuks

Transcript

This is something I did before, following instructions from somewhere.

[root@exciter ~]# badblocks -c 10240 -s -w -t random -v /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0
Checking for bad blocks in read-write mode
From block 0 to 488386583
Testing with random pattern: done                                
Reading and comparing: done                                
Pass completed, 0 bad blocks found.
[root@exciter ~]# cfdisk /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0
Disk has been changed.

WARNING: If you have created or modified any
DOS 6.x partitions, please see the cfdisk manual
page for additional information.

[root@exciter ~]# cryptsetup --verbose --verify-passphrase luksFormat /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1 

WARNING!
========
This will overwrite data on /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase: 
Verify passphrase: 
Command successful.
[root@exciter ~]# cryptsetup luksOpen /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1 FreeAgent
Enter LUKS passphrase for /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1: 
key slot 0 unlocked.
Command successful.
[root@exciter ~]# mkfs.ext3 -j -m 1 -O dir_index,filetype,sparse_super /dev/mapper/FreeAgent 
mke2fs 1.41.4 (27-Jan-2009)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
30531584 inodes, 122095871 blocks
1220958 blocks (1.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
3727 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
 102400000

Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 34 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Un-mounting

umount /mnt/FreeAgent
rmdir /mnt/FreeAgent
cryptsetup luksClose /dev/mapper/FreeAgent

(safe to remove)

Notes

• With LUKS, you can easily change the pass-phrase, and even have more than one at a time. See luksAddKey in the manual (man cryptsetup).
  

Zoom Lens Comparison

How useful is a high-power zoom lens? Very useful if it means you don't have to switch lenses and carry an extra lens around. The following table shows how much more powerful the Nikon 18-200mm 1:3.5-5.6 G is over the Nikon 28-80mm 1:3.3-5.6 G.

	D50 28-80mm	D90 18-200mm
Wide	28mm	18mm
Tele	80mm	200mm

Within each column, the only variable is the movement of the zoom ring on the lens. Within each row, I changed out the lens and the camera body, so exposure details may vary. Both cameras were on point-and-shoot mode, since this is merely an illustration of the field of view.

Need more swap space?

When I installed Fedora 10 from scratch, I thought that 2GB was probably enough swap space, given that disk access is slow. With only 4GB of RAM, which used to be a lot, as soon as I started using upwards of 6GB, the swap space I had allocated obviously became filled and the program would fail.

Therefore, I am reverting back to the rule of thumb,
"Your swap space should be about twice your RAM size"
which has never failed me.

However, even though I tried, it's a huge pain to make space when you have three hard drives all allocated into a boot partition, a swap partition, and a bunch of LVM partitions, and a logical volume that uses all the LVM space. Is freeing space for a new swap partition possible? Yes. Fast or convenient? No.

Luckily, I discovered that you can use a file in the file system for swap, just as our favourite operating system from Redmond does. I was successful in doing the following on two different machines:

Become the superuser.

$ sudo su

Create a large file of zeros.

# dd if=/dev/zero of=/swapfile1 bs=1024 count=6144k
6291456+0 records in
6291456+0 records out
6442450944 bytes (6.4 GB) copied, 106.946 s, 60.2 MB/s

Make this file a swap partition.

# mkswap -c /swapfile1
Setting up swapspace version 1, size = 6291452 KiB
no label, UUID=7f5ab06e-4315-4185-b5a2-b3151a5a608c

Add the partition to the swap space.

# swapon /swapfile1

Check that it worked; total swap should be 6GB larger now.

# free -m
          total       used       free     shared    buffers     cached
Mem:          3967       3936         30          0         75       3054
-/+ buffers/cache:        807       3159
Swap:         8144          0       8144

Add the swapon command somewhere in this file. This will add the file to the swap space the at boot.

# vi /etc/rc.local

You're done! However, a partition would be faster (and perhaps safer), so the next time you upgrade, consider re-partitioning.

Also note that Linux will use the partition swap first, as per its priority according to:

# cat /proc/swaps
Filename    Type  Size Used Priority
/dev/sdb1                               partition 2048248 96 -1
/swapfile1                              file  6291448 0 -2

And with that in mind, I can think of all sorts of other fixes for running out of swap space (USB drives, compact flash to SATA adapters, etc.).

Making a Time-lapse Video with the Nikon D50 and gphoto2

Who said you cannot teach a 2-year-old digital camera new tricks?

I was really excited when I found out I could control my Digital SLR camera via the USB cable. Nikon produces a software called Camera Control Pro. The software is $170 USD, and there is no non-pro version that I know of. They give you a 30-day trial, so I thought I would check it out.

At first, my camera wouldn't connect. I had to use the menu on the camera to take the USB mode out of "Mass Storage" and into "PTP". The camera stopped bringing up iPhoto and started working in Camera Control. Life was good.

I made a time-lapse of the sunset through my 10.5mm fisheye lens, taking one frame per minute. I used ImageMagick to convert the JPEGs into an MPEG, although this method lacks a lot of the control that I expected: frame rate, bitrate, etc.

As I was looking for a free way to do this, since I was not going to pay almost $200; I have to eat too you know. I (almost by accident) came across gphoto2. I think I had already heard of this software as I was wondering how I could capture images from a webcam, if I had one. The gphoto2 software (available through YUM in Fedora) provides a command-line method to do what I was doing with Camera Control; taking time-lapse pictures. However, it doesn't provide an interface similar to the one on the camera in a GUI form for changing camera settings. No matter, since you can use the controls on the camera directly in PTP mode.

So I made an AT job:

at 6:00am
cd /home/ryan/Pictures/Timelapse/Sunrise
gphoto2 --capture-image --interval=15
[CTRL+D]

I quickly found out that I need to purchase the Nikon EH-5 AC adapter since the camera spends a lot of power in PTP mode and the battery died overnight. Replaced with a spare just in the nick of time and everything went as planned. I should have increased the rate to four times per minute (every 15 seconds), but I was scared after the battery died that with nearly continuous long-shutter shots the battery would die again. As soon as I have the adapter, I'll try going to four frames per minute (or more).

Then, I used a script that had taken some time to develop to compile the frames into video. There are three major steps:

1. Down-sample the stills
2. Crop the stills to some standard video format (here I chose to move the crop region 131 pixels (all the way down) since the earth was more interesting than the sky
   
3. Use good old piping to massage the frames into an MPEG

The following script performs all this, and assumes that the captured images are named as gphoto2 names them, capt0000.jpg, capt0001.jpg, etc.

Here are the packages I needed:

• netpbm-progs
  
• mjpegtools
  

#!/bin/sh

echo "Resizing images to 720p (1280x720)..."
for JPG in capt????.jpg ;
do
TMPNAME=`echo $JPG | sed 's/capt/Mov/' | sed 's/\.jpg/\.ppm/'`;
if [ ! -f $TMPNAME ] ;
then
      echo "$JPG";
      CORRNAME=$JPG
      # Resizing gives an image that is 1280x851, so I need to also crop and shift by 65 pixels (to crop the top and bottom equally)
      convert -resize 1280x1280 -crop 1280x720+0+131 $CORRNAME $TMPNAME;
fi
done

echo "Compiling video...";
ls *.ppm | xargs cat | pamdepth 255 | ppmtoy4m -F 24:1 -S 420mpeg2 | yuvfps -s 4:1 -r 24:1 | mpeg2enc -f 12 -o timelapse.m1v


echo "Done!"

The last piping command

• makes a list of all the PPM's (this could be more elegant),
  
• makes them all arguments to cat (basically, sends them all to stdout),
  
• decreases the PPM color depth to one byte (255 values),
  
• translates the stream of PPM's to a Y4M stream using some standard MPEG settings (24 FPS, and 4:2:0 chroma subsampling mode),
• repeats frames to make four distinct exposures per second but 24 video frames per second,
• and finally compiles an MPEG using format 12 (ATSC 720p video).
  

And, finally, here is the fruit of my labor, which was originally 720p (1280x720 pixels), and 4 frames per second up-sampled to 24 frames per second. The framerate is a bit slow, so you might change the '-s' argument to 'yuvfps' to '8:1' to make it smoother, but with that I recommend to capture more frames per minute.


Yes, you can see my wall clock because a night-shift construction crew was lighting up my whole apartment.

For better results during daytime or night-time (not a transition between the two), I would recommend going to manual exposure control. This would make things a lot smoother.

So to recap, with gphoto2 you can:

• Take more frames per minute, such as 4
• Try making a scene using manual exposure control (like clouds or a tree)
• Create a 1080p (1920×1080) video since your native resolution is probably higher

Watch more on YouTube:

• http://www.youtube.com/watch?v=iWvCHlPRN-k
• http://www.youtube.com/watch?v=SbcrCUPQf-g

Nikon D-50 Cleaning Dust from Sensor

So, on some bright long-exposure shots, I noticed there were some consistent dust spots. I took some slow-shutter, out-of-focus test shots to see. I changed lenses thinking it could be one of the internal elements, but nothing---the dirt was on the sensor. I decided to clean out the inside of the body. Some dust was accumulating on the reflex mirror and the window for the viewfinder. However, if you move the mirror down and out of the way (ever so carefully!), there is a door in the rear which is called the shutter curtain. I thought I was out of luck and needed help, until (completely by chance) I came across a section in the User Manual about cleaning the low-pass filter.

You can see the dirt at about 4:30 in this test shot.

I knew that the low-pass filter was a fancy term for an IR filter that happens to provide an air-tight seal for the CCD sensor. There's a procedure in the manual that enables you to lock open the reflex mirror and the shutter curtain, exposing the low-pass filter and sensor. I highly recommend that you read these instructions; they're very easy to follow. They recommend not to contact the low-pass filter, and I totally agree.

I followed the instructions for locking the mirror and curtain open, and when I removed the body cap, I could clearly see the debris on the sensor. I simply blew some air in there, and it was freed. I recommend blowing air while holding the body upside-down. Make sure you have a lot of light.

Nikon also has directions for setting up test shots for seeing if you have a dust problem. It is geared for their dust removal software, but it also serves as a diagnostic.

Test shot following Nikon's directions after cleaning

You turn the camera off to exit the special lock mode. Presto! No more dirt on my photos!

MD5SUM reorganization revisited

I was running my shell script on a collection of over 10,000 files, and it was running for over three days. The shell interpreter is just way too slow when this amount of work is involved. So, I re-wrote the script in Perl and it is now lightning fast because I am able to take advantage of Perl hashes (associative arrays).

#!/usr/bin/perl

#

# This file takes a checksum file (output from md5sum utility)

# and attempts to reorganize the files in the directory to

# match the listing in the md5 file.

#

# Files not found in the md5 input are left alone.

# Files already in the right place are left alone.

# Other files have their checksums computed and, if they are found

# in the md5 input, they are moved to the appropriate location.

#

# WARNING: It confuses duplicate files!!!

#

use File::Basename;



if ( $#ARGV < reorgpath =" $ARGV[1];" reorgptah = "." lines =" ;



close(SUMSFILE);



foreach my $line (@lines) {

chomp($line);

$sums{substr($line,0,32)} = substr($line,34);

}



print "Read in ".($#lines+1)." checksums and paths.\n";



&reorg($reorgPath);



sub reorg {

my $dir = shift;

#print "Recurring for $dir\n";



opendir DIR, $dir or return;

my @contents =

 map "$dir/$_",

 sort grep !/^\.\.?$/,

 readdir DIR;

closedir DIR;

foreach my $file (@contents) {

 #print "Considering $file\n";

 #next unless !-1 && -d;

 if ( -d $file ) {

  &reorg($file);

 } else {

#   my @args = ("md5sum", $file);

#   system(@args) == 0 or die("System @args failed: $?\n");

  $tmpLine = `md5sum "$file"`;

  chomp($tmpLine);

  $tmpHash = substr($tmpLine,0,32);

  $tmpPath = substr($tmpLine,34);



  #print "$tmpHash -> $tmpPath\n";

  # Now lookup the hash, if the paths aren't the same, move the file

  if (defined $sums{$tmpHash}) {

   #print $sums{$tmpHash};

   if ($tmpPath ne $sums{$tmpHash}) {

    print "Moving ".$tmpPath." to ".$sums{$tmpHash}."\n";



    # Make directory for move if it doesn't exist

    @args = ("mkdir", "-p", dirname($sums{$tmpHash}) );

    system(@args) == 0 or print STDERR "Couldn't create directory @args: $!\n";

    # Move file to path found in checksum file

    @args = ("mv", $tmpPath, $sums{$tmpHash});

    system(@args) == 0 or print STDERR "Couldn't move: $!\n";

   } else {

    print "File $tmpPath is already in place.\n";

   }

  } else {

   print "No hash for $tmpPath found.\n";

  }

 }

}

return;

}



exit;

A few things still need to be fixed: Inserting a new hash from the file into the hash table should fail if it is already there (hash collision or duplicate files). Also, a reverse lookup in the hash table could save needing to compute the MD5SUM, but at what computational cost?

Data/Documents backup script

I used to always create directory under my home called data. I'd store files which I need to do work and my working files. Then, this would be the only directory I had to synchronize with Unison (http://www.cis.upenn.edu/~bcpierce/unison/)between machines. Since Fedora has included a directory called Documents, which has a similar purpose (and so has Mac OS X), I've just changed the name of this special directory from data to Documents.

That brings me to the topic of this article--how do I manage if I mess up a file in this directory, and I synchronize the mistake to other mirrors? That would mean none of the computers I work on has a copy anymore. The answer is rdiff-backup (http://www.nongnu.org/rdiff-backup/). I used to make just a user cron job to call rdiff-backup and update an incremental backup repo on my behalf, but I wanted this to be more mechanical (what if I added a new user, etc.). So I wrote the following script which replaces an old script I used to have which would only make a mirror with rsync.

Note that, on my configuration, the incremental backup repo is on the same device as the original files, but both are mirrored in an overnight backup job. This means I have (at least) four mirrors of the same files, which is sort of a waste but I don't really worry about it since it's about 200 MB / user. Ideally, you would keep just the local mirror on the disk, and the rdiff-backup repository on the backup device (or system over network). The downside here would be, if the backup device (or remote system) crashes you loose the ability to restore a previous increment.

Following is the script for maintaining incremental backup repositories with rdiff-backup for all users. Caution: the backup path should be as read-only as possible, but it is not if the user logs in to the system. For this reason, you'll have to count on them not to modify the files, or keep the files on a filesystem they cannot mount. For simple remote access over samba (the kind of access in my setup), you would have to configure a per-user share (so Joe can't see Susan's backup, for instance) and make each one read-only (so Joe cannot corrupt his repository by changing the files there).

#!/bin/sh
#
# Make incremental backups of 'Documents' directories for users
#
# Ryan Helinski

LOGFILE=/var/log/rdiff-backup.log

# Close stdout and stderr
1>&-; 2>&-;

# Direct stdout and stderr to logfile
exec 1>>$LOGFILE; exec 2>>$LOGFILE;

echo Begin $0, `date`;

# For debugging
#set -x

# Script-specific configuration
HOMESPATH=/home
BACKUPPATH=/opt/backup
RDIFF_OPTIONS="--print-statistics"
DOCSDIR=Documents

# Max increment life 
# The time interval is an integer followed by the character s, m, h, D, W, 
# M, or Y, indicating seconds,  minutes, hours,  days,  weeks,  months, or 
# years respectively, or a number of these con-catenated.
MAX_LIFE=4W

for USERHOME in $HOMESPATH/* ; do

# USERDIR=`basename $USERHOME`;
 USER=`stat -c "%U" $USERHOME`;

 if [ -d "$USERHOME/$DOCSDIR" ] ; then
  echo "Updating $BACKUPPATH/$USER/$DOCSDIR";

  # Create backup repo if it doesn't exist
  if [ ! -d "$BACKUPPATH/$USER/$DOCSDIR" ] ; then
   echo "Creating rdiff-backup repo";
   mkdir -p "$BACKUPPATH/$USER/$DOCSDIR";
   chown $USER "$BACKUPPATH/$USER";
   chown $USER "$BACKUPPATH/$USER/$DOCSDIR";
  fi

  # Update repo
  rdiff-backup $RDIFF_OPTIONS "$USERHOME/$DOCSDIR" \
   "$BACKUPPATH/$USER/$DOCSDIR" ;

  # Clean up repo
  #rdiff-backup --remove-older-than $MAX_LIFE --force \
  # "$BACKUPPATH/$USER/$DOCSDIR" ;

 fi

done

echo End $0, `date`;
echo

exit

To actually remove old backups, uncomment the code in the "Clean up repo" section.