Ryan's Bliggity Blog

Encrypted FS with LUKS

2010-01-09T11:17:00.007-05:00

Partitioning a Seagate FreeAgent

Outline

Check for bad blocks (not required)

badblocks -c 10240 -s -w -t random -v /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0

Partition drive. In this case, I create one 'Linux' partition which spans the whole drive.

cfdisk /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0

Create a LUKS device (partition) on the drive

cryptsetup --verbose --verify-passphrase luksFormat /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1

Open that device

cryptsetup luksOpen /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1 FreeAgent

Create an EXT partition on the device.

mkfs.ext3 -j -m 1 -O dir_index,filetype,sparse_super /dev/mapper/FreeAgent

These options aren't necessary. The following will also work fine:

mkfs.ext3 /dev/mapper/FreeAgent

Mount the encrypted partition

mkdir /media/FreeAgentLuks
mount /dev/mapper/FreeAgent /media/FreeAgentLuks

Transcript

This is something I did before, following instructions from somewhere.

[root@exciter ~]# badblocks -c 10240 -s -w -t random -v /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0
Checking for bad blocks in read-write mode
From block 0 to 488386583
Testing with random pattern: done                                
Reading and comparing: done                                
Pass completed, 0 bad blocks found.
[root@exciter ~]# cfdisk /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0
Disk has been changed.

WARNING: If you have created or modified any
DOS 6.x partitions, please see the cfdisk manual
page for additional information.

[root@exciter ~]# cryptsetup --verbose --verify-passphrase luksFormat /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1 

WARNING!
========
This will overwrite data on /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase: 
Verify passphrase: 
Command successful.
[root@exciter ~]# cryptsetup luksOpen /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1 FreeAgent
Enter LUKS passphrase for /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1: 
key slot 0 unlocked.
Command successful.
[root@exciter ~]# mkfs.ext3 -j -m 1 -O dir_index,filetype,sparse_super /dev/mapper/FreeAgent 
mke2fs 1.41.4 (27-Jan-2009)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
30531584 inodes, 122095871 blocks
1220958 blocks (1.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
3727 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
 102400000

Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 34 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Un-mounting

umount /mnt/FreeAgent
rmdir /mnt/FreeAgent
cryptsetup luksClose /dev/mapper/FreeAgent

(safe to remove)

Notes

With LUKS, you can easily change the pass-phrase, and even have more than one at a time. See luksAddKey in the manual (man cryptsetup).

Zoom Lens Comparison

2009-06-02T19:12:00.011-05:00

How useful is a high-power zoom lens? Very useful if it means you don't have to switch lenses and carry an extra lens around. The following table shows how much more powerful the Nikon 18-200mm 1:3.5-5.6 G is over the Nikon 28-80mm 1:3.3-5.6 G.

	D50 28-80mm	D90 18-200mm
Wide	28mm	18mm
Tele	80mm	200mm

Within each column, the only variable is the movement of the zoom ring on the lens. Within each row, I changed out the lens and the camera body, so exposure details may vary. Both cameras were on point-and-shoot mode, since this is merely an illustration of the field of view.

Need more swap space?

2009-05-14T13:36:00.004-05:00

When I installed Fedora 10 from scratch, I thought that 2GB was probably enough swap space, given that disk access is slow. With only 4GB of RAM, which used to be a lot, as soon as I started using upwards of 6GB, the swap space I had allocated obviously became filled and the program would fail.

Therefore, I am reverting back to the rule of thumb,
"Your swap space should be about twice your RAM size"
which has never failed me.

However, even though I tried, it's a huge pain to make space when you have three hard drives all allocated into a boot partition, a swap partition, and a bunch of LVM partitions, and a logical volume that uses all the LVM space. Is freeing space for a new swap partition possible? Yes. Fast or convenient? No.

Luckily, I discovered that you can use a file in the file system for swap, just as our favourite operating system from Redmond does. I was successful in doing the following on two different machines:

Become the superuser.

$ sudo su

Create a large file of zeros.

# dd if=/dev/zero of=/swapfile1 bs=1024 count=6144k
6291456+0 records in
6291456+0 records out
6442450944 bytes (6.4 GB) copied, 106.946 s, 60.2 MB/s

Make this file a swap partition.

# mkswap -c /swapfile1
Setting up swapspace version 1, size = 6291452 KiB
no label, UUID=7f5ab06e-4315-4185-b5a2-b3151a5a608c

Add the partition to the swap space.

# swapon /swapfile1

Check that it worked; total swap should be 6GB larger now.

# free -m
          total       used       free     shared    buffers     cached
Mem:          3967       3936         30          0         75       3054
-/+ buffers/cache:        807       3159
Swap:         8144          0       8144

Add the swapon command somewhere in this file. This will add the file to the swap space the at boot.

# vi /etc/rc.local

You're done! However, a partition would be faster (and perhaps safer), so the next time you upgrade, consider re-partitioning.

Also note that Linux will use the partition swap first, as per its priority according to:

# cat /proc/swaps
Filename    Type  Size Used Priority
/dev/sdb1                               partition 2048248 96 -1
/swapfile1                              file  6291448 0 -2

And with that in mind, I can think of all sorts of other fixes for running out of swap space (USB drives, compact flash to SATA adapters, etc.).

Making a Time-lapse Video with the Nikon D50 and gphoto2

2009-01-30T12:09:00.015-05:00

Who said you cannot teach a 2-year-old digital camera new tricks?

I was really excited when I found out I could control my Digital SLR camera via the USB cable. Nikon produces a software called Camera Control Pro. The software is $170 USD, and there is no non-pro version that I know of. They give you a 30-day trial, so I thought I would check it out.

At first, my camera wouldn't connect. I had to use the menu on the camera to take the USB mode out of "Mass Storage" and into "PTP". The camera stopped bringing up iPhoto and started working in Camera Control. Life was good.

I made a time-lapse of the sunset through my 10.5mm fisheye lens, taking one frame per minute. I used ImageMagick to convert the JPEGs into an MPEG, although this method lacks a lot of the control that I expected: frame rate, bitrate, etc.

As I was looking for a free way to do this, since I was not going to pay almost $200; I have to eat too you know. I (almost by accident) came across gphoto2. I think I had already heard of this software as I was wondering how I could capture images from a webcam, if I had one. The gphoto2 software (available through YUM in Fedora) provides a command-line method to do what I was doing with Camera Control; taking time-lapse pictures. However, it doesn't provide an interface similar to the one on the camera in a GUI form for changing camera settings. No matter, since you can use the controls on the camera directly in PTP mode.

So I made an AT job:


at 6:00am
cd /home/ryan/Pictures/Timelapse/Sunrise
gphoto2 --capture-image --interval=15
[CTRL+D]

I quickly found out that I need to purchase the Nikon EH-5 AC adapter since the camera spends a lot of power in PTP mode and the battery died overnight. Replaced with a spare just in the nick of time and everything went as planned. I should have increased the rate to four times per minute (every 15 seconds), but I was scared after the battery died that with nearly continuous long-shutter shots the battery would die again. As soon as I have the adapter, I'll try going to four frames per minute (or more).

Then, I used a script that had taken some time to develop to compile the frames into video. There are three major steps:

Down-sample the stills
Crop the stills to some standard video format (here I chose to move the crop region 131 pixels (all the way down) since the earth was more interesting than the sky
Use good old piping to massage the frames into an MPEG

The following script performs all this, and assumes that the captured images are named as gphoto2 names them, capt0000.jpg, capt0001.jpg, etc.

Here are the packages I needed:

netpbm-progs
mjpegtools


#!/bin/sh

echo "Resizing images to 720p (1280x720)..."
for JPG in capt????.jpg ;
do
TMPNAME=`echo $JPG | sed 's/capt/Mov/' | sed 's/\.jpg/\.ppm/'`;
if [ ! -f $TMPNAME ] ;
then
      echo "$JPG";
      CORRNAME=$JPG
      # Resizing gives an image that is 1280x851, so I need to also crop and shift by 65 pixels (to crop the top and bottom equally)
      convert -resize 1280x1280 -crop 1280x720+0+131 $CORRNAME $TMPNAME;
fi
done

echo "Compiling video...";
ls *.ppm | xargs cat | pamdepth 255 | ppmtoy4m -F 24:1 -S 420mpeg2 | yuvfps -s 4:1 -r 24:1 | mpeg2enc -f 12 -o timelapse.m1v


echo "Done!"

The last piping command

makes a list of all the PPM's (this could be more elegant),
makes them all arguments to cat (basically, sends them all to stdout),
decreases the PPM color depth to one byte (255 values),
translates the stream of PPM's to a Y4M stream using some standard MPEG settings (24 FPS, and 4:2:0 chroma subsampling mode),
repeats frames to make four distinct exposures per second but 24 video frames per second,
and finally compiles an MPEG using format 12 (ATSC 720p video).

And, finally, here is the fruit of my labor, which was originally 720p (1280x720 pixels), and 4 frames per second up-sampled to 24 frames per second. The framerate is a bit slow, so you might change the '-s' argument to 'yuvfps' to '8:1' to make it smoother, but with that I recommend to capture more frames per minute.

Yes, you can see my wall clock because a night-shift construction crew was lighting up my whole apartment.

For better results during daytime or night-time (not a transition between the two), I would recommend going to manual exposure control. This would make things a lot smoother.

So to recap, with gphoto2 you can:

Take more frames per minute, such as 4
Try making a scene using manual exposure control (like clouds or a tree)
Create a 1080p (1920×1080) video since your native resolution is probably higher

Watch more on YouTube:

Nikon D-50 Cleaning Dust from Sensor

2008-12-08T23:17:00.005-05:00

So, on some bright long-exposure shots, I noticed there were some consistent dust spots. I took some slow-shutter, out-of-focus test shots to see. I changed lenses thinking it could be one of the internal elements, but nothing---the dirt was on the sensor. I decided to clean out the inside of the body. Some dust was accumulating on the reflex mirror and the window for the viewfinder. However, if you move the mirror down and out of the way (ever so carefully!), there is a door in the rear which is called the shutter curtain. I thought I was out of luck and needed help, until (completely by chance) I came across a section in the User Manual about cleaning the low-pass filter.

You can see the dirt at about 4:30 in this test shot.

I knew that the low-pass filter was a fancy term for an IR filter that happens to provide an air-tight seal for the CCD sensor. There's a procedure in the manual that enables you to lock open the reflex mirror and the shutter curtain, exposing the low-pass filter and sensor. I highly recommend that you read these instructions; they're very easy to follow. They recommend not to contact the low-pass filter, and I totally agree.

I followed the instructions for locking the mirror and curtain open, and when I removed the body cap, I could clearly see the debris on the sensor. I simply blew some air in there, and it was freed. I recommend blowing air while holding the body upside-down. Make sure you have a lot of light.

Nikon also has directions for setting up test shots for seeing if you have a dust problem. It is geared for their dust removal software, but it also serves as a diagnostic.

Test shot following Nikon's directions after cleaning

You turn the camera off to exit the special lock mode. Presto! No more dirt on my photos!

MD5SUM reorganization revisited

2008-04-27T11:18:00.004-05:00

I was running my shell script on a collection of over 10,000 files, and it was running for over three days. The shell interpreter is just way too slow when this amount of work is involved. So, I re-wrote the script in Perl and it is now lightning fast because I am able to take advantage of Perl hashes (associative arrays).

#!/usr/bin/perl

#

# This file takes a checksum file (output from md5sum utility)

# and attempts to reorganize the files in the directory to

# match the listing in the md5 file.

#

# Files not found in the md5 input are left alone.

# Files already in the right place are left alone.

# Other files have their checksums computed and, if they are found

# in the md5 input, they are moved to the appropriate location.

#

# WARNING: It confuses duplicate files!!!

#

use File::Basename;



if ( $#ARGV < reorgpath =" $ARGV[1];" reorgptah = "." lines =" ;



close(SUMSFILE);



foreach my $line (@lines) {

chomp($line);

$sums{substr($line,0,32)} = substr($line,34);

}



print "Read in ".($#lines+1)." checksums and paths.\n";



&reorg($reorgPath);



sub reorg {

my $dir = shift;

#print "Recurring for $dir\n";



opendir DIR, $dir or return;

my @contents =

 map "$dir/$_",

 sort grep !/^\.\.?$/,

 readdir DIR;

closedir DIR;

foreach my $file (@contents) {

 #print "Considering $file\n";

 #next unless !-1 && -d;

 if ( -d $file ) {

  &reorg($file);

 } else {

#   my @args = ("md5sum", $file);

#   system(@args) == 0 or die("System @args failed: $?\n");

  $tmpLine = `md5sum "$file"`;

  chomp($tmpLine);

  $tmpHash = substr($tmpLine,0,32);

  $tmpPath = substr($tmpLine,34);



  #print "$tmpHash -> $tmpPath\n";

  # Now lookup the hash, if the paths aren't the same, move the file

  if (defined $sums{$tmpHash}) {

   #print $sums{$tmpHash};

   if ($tmpPath ne $sums{$tmpHash}) {

    print "Moving ".$tmpPath." to ".$sums{$tmpHash}."\n";



    # Make directory for move if it doesn't exist

    @args = ("mkdir", "-p", dirname($sums{$tmpHash}) );

    system(@args) == 0 or print STDERR "Couldn't create directory @args: $!\n";

    # Move file to path found in checksum file

    @args = ("mv", $tmpPath, $sums{$tmpHash});

    system(@args) == 0 or print STDERR "Couldn't move: $!\n";

   } else {

    print "File $tmpPath is already in place.\n";

   }

  } else {

   print "No hash for $tmpPath found.\n";

  }

 }

}

return;

}



exit;

A few things still need to be fixed: Inserting a new hash from the file into the hash table should fail if it is already there (hash collision or duplicate files). Also, a reverse lookup in the hash table could save needing to compute the MD5SUM, but at what computational cost?

Data/Documents backup script

2008-03-04T10:49:00.005-05:00

I used to always create directory under my home called data. I'd store files which I need to do work and my working files. Then, this would be the only directory I had to synchronize with Unison (http://www.cis.upenn.edu/~bcpierce/unison/)between machines. Since Fedora has included a directory called Documents, which has a similar purpose (and so has Mac OS X), I've just changed the name of this special directory from data to Documents.

That brings me to the topic of this article--how do I manage if I mess up a file in this directory, and I synchronize the mistake to other mirrors? That would mean none of the computers I work on has a copy anymore. The answer is rdiff-backup (http://www.nongnu.org/rdiff-backup/). I used to make just a user cron job to call rdiff-backup and update an incremental backup repo on my behalf, but I wanted this to be more mechanical (what if I added a new user, etc.). So I wrote the following script which replaces an old script I used to have which would only make a mirror with rsync.

Note that, on my configuration, the incremental backup repo is on the same device as the original files, but both are mirrored in an overnight backup job. This means I have (at least) four mirrors of the same files, which is sort of a waste but I don't really worry about it since it's about 200 MB / user. Ideally, you would keep just the local mirror on the disk, and the rdiff-backup repository on the backup device (or system over network). The downside here would be, if the backup device (or remote system) crashes you loose the ability to restore a previous increment.

Following is the script for maintaining incremental backup repositories with rdiff-backup for all users. Caution: the backup path should be as read-only as possible, but it is not if the user logs in to the system. For this reason, you'll have to count on them not to modify the files, or keep the files on a filesystem they cannot mount. For simple remote access over samba (the kind of access in my setup), you would have to configure a per-user share (so Joe can't see Susan's backup, for instance) and make each one read-only (so Joe cannot corrupt his repository by changing the files there).


#!/bin/sh
#
# Make incremental backups of 'Documents' directories for users
#
# Ryan Helinski

LOGFILE=/var/log/rdiff-backup.log

# Close stdout and stderr
1>&-; 2>&-;

# Direct stdout and stderr to logfile
exec 1>>$LOGFILE; exec 2>>$LOGFILE;

echo Begin $0, `date`;

# For debugging
#set -x

# Script-specific configuration
HOMESPATH=/home
BACKUPPATH=/opt/backup
RDIFF_OPTIONS="--print-statistics"
DOCSDIR=Documents

# Max increment life 
# The time interval is an integer followed by the character s, m, h, D, W, 
# M, or Y, indicating seconds,  minutes, hours,  days,  weeks,  months, or 
# years respectively, or a number of these con-catenated.
MAX_LIFE=4W

for USERHOME in $HOMESPATH/* ; do

# USERDIR=`basename $USERHOME`;
 USER=`stat -c "%U" $USERHOME`;

 if [ -d "$USERHOME/$DOCSDIR" ] ; then
  echo "Updating $BACKUPPATH/$USER/$DOCSDIR";

  # Create backup repo if it doesn't exist
  if [ ! -d "$BACKUPPATH/$USER/$DOCSDIR" ] ; then
   echo "Creating rdiff-backup repo";
   mkdir -p "$BACKUPPATH/$USER/$DOCSDIR";
   chown $USER "$BACKUPPATH/$USER";
   chown $USER "$BACKUPPATH/$USER/$DOCSDIR";
  fi

  # Update repo
  rdiff-backup $RDIFF_OPTIONS "$USERHOME/$DOCSDIR" \
   "$BACKUPPATH/$USER/$DOCSDIR" ;

  # Clean up repo
  #rdiff-backup --remove-older-than $MAX_LIFE --force \
  # "$BACKUPPATH/$USER/$DOCSDIR" ;

 fi

done

echo End $0, `date`;
echo

exit

To actually remove old backups, uncomment the code in the "Clean up repo" section.

Shell Script to Tunnel VNC Sessions over SSH (Improved)

2008-01-30T12:38:00.001-05:00

Following is a parameterized shell script which you can use with this syntax:

Usage: vnctunnel [ssh session] [screen] [vnc options]
Example: vnctunnel user@server :1

It's "improved" because it's completely parameterized--you can specify the SSH session, screen, and even vnc options.

(Assuming you copy it to your ~/bin directory and make it executable)

#!/bin/sh
# This is a parameterized script to SSH tunnel a VNC session
# Ryan Helinski

# Local settings
# Change this if you don't like `vncviewer'
VNCVIEWER=vncviewer
# Change this if you use ports near 25900 for something else locally
PORTOFFSET=20000
# Apply extra options to vncviewer
VNCOPTS="--AutoSelect=0 $3"

if [ $# -lt 2 ] ; then
        echo "Usage: $0 [ssh session] [screen]";
        echo "Example: $0 user@server :1";
fi

SSHPARM=$1;
SCREEN=`echo $2 | cut -d':' -f2`;
SSHPORT=$[$SCREEN+5900];

echo "Session: $SSHPARM, Screen: $SCREEN, Port: $SSHPORT"

ssh -f -L $[$PORTOFFSET+$SSHPORT]:localhost:$SSHPORT $SSHPARM sleep 10; \
        vncviewer localhost:$[$PORTOFFSET+$SSHPORT]:$SCREEN $VNCOPTS

exit

Using 7-zip for Archival

2008-01-30T12:02:00.000-05:00

I've been using 7-zip to archive large collections of files--mainly college and documents which have high potential to be compressed. 7-zip has a high compression ratio--higher than bzip2, but also higher than the evil RAR archive everyone seems to like--and it's GPL and widely available for GNU+Linux, UNIX and Mac via `p7zip'. However, I've been slightly concerned that it doesn't retain POSIX file modes. For files like text, graphics, etc. this doesn't really bother me. However, if I was going to archive something which contains scripts, or a shared directory where user ownership needs to be preserved, I'd need to retain this extra information.

I have seen it a couple times while browsing, and thought "oh...". When I tried to look for it, I really found a lack of examples. After careful reading of the manuals for tar and 7z, following is the solution:

To encode:

tar cf - data/ | 7za a data.tar.7z -si

To decode:

7za x ~/data.tar.7z -so | tar xf -

Trash Management Scripts

2008-01-28T10:17:00.000-05:00

Moving files to trash

Today's post is about some trash management scripts I have written for myself and would like to share. They should run on most GNU+Linux environments.

If you're like me, you prefer to mv file ~/.Trash/ rather than rm file, but this is both dangerous and hard to type. I created a (very small) script that will act as a command you can use to move files to your Trash bin without overwriting anything.

Uses 'numbered' backup scheme of 'mv', so the most recently trashed
file will have its name intact, and the backups will have the suffix
".~n~" appended, where the higher the n, the more recent the backup.


#!/bin/sh
#
# Moves files in the command arguments to the trash bin, while keeping 
# backups of any files already in that directory. 
#
# Uses 'numbered' backup scheme of 'mv', so the most recently trashed 
# file will have its name intact, and the backups will have the suffix 
# .~n~ appended where the higher the n, the more recent the backup. 
#
# It uses the basename of the file so that no (absolute or relative) 
# directory is preserved.
#
TRASH="$HOME/.Trash";

while [ $# -gt 0 ];
do
        mv --backup=numbered "$1" "$TRASH/`basename $1`";

        shift;
done

After you copy this file to your favorite bin directory and make it executable, you can use the following syntax

trash file1 file2 path/to/file3 path/to/file4

And, even if file1 and file3 have the same name, you'll still be able to find both in your trash bin.

Rounding up and deleting trash automatically

If your trash is anything like mine, the probability that a file is not trash increases exponentially every day. For this reason, I wanted to somehow tag when I threw something out so I could tell how old it was, and maybe delete everything after a specific number of days.

The solution I came up with was to create siblings of the .Trash directory, using UNIX time-stamps. In conjunction with a cron task, yesterday's trash will be in a directory at ~/.Trash-XXXXXXXXXX where the X's are the time-stamp for 4:00am today. In this manner, you'll have a bin for each day you throw something out. The script follows.


#!/bin/sh
#
# The first step is to move all files under ~/.Trash, other than 
# ., .., and .#bin-n into a new trash bin for yesterday.
# 
#

NEWBIN=`date +%s`;
NEWBINPATH="$HOME/.Trash-$NEWBIN";
OLDESTBIN="20"; # days

OLDESTBINTIME=`date -d "now - $OLDESTBIN days" +%s`;
OLDESTBINPATH="$HOME/.Trash-$OLDESTBINTIME";

echo "Moving current trash to $NEWBINPATH";
mkdir $NEWBINPATH;
mv $HOME/.Trash/* $HOME/.Trash/.[!.]* $NEWBINPATH/;

for BIN in $HOME/.Trash-*; do 

        STAMP=`echo "$BIN" | cut -d'-' -f2`;
#       echo $BIN $STAMP $OLDESTBINTIME;
        if [ $STAMP -lt $OLDESTBINTIME ] ;
        then
                echo "Deleting $BIN";
#               rm -Rf $BIN;
        fi

done

So I copied this file into my ~/bin/ directory and used crontab -e to add the following line to my crontab:

00 4 * * * /home/ryan/bin/trash-roundup.sh

Right now this script will send errors to crond, which should be delivered in your local mail (accessed using mail). Also, deleting old bins is disabled since I haven't had a chance to thoroughly test it.

To actually delete old trash, choose a value for the OLDESTBIN variable in the script, this is the longest time that a bin will hang around. Then, you have to un-comment the line with rm in the script.

Using the Seagate FreeAgent as a (periodic) Mirror Backup

2008-01-25T11:34:00.001-05:00

I recently purchased a Seagate FreeAgent 100D USB drive. The drive is quite nice, small, quiet, and has advanced power management. I plugged the device in on a Fedora 7 installation, and it came right up.

The first problem I had was the NTFS partition that came on the device. This is easily fixed with fdisk and mkfs.ext3. Don't forget to use e2label, especially if you have more than one of the same device.

The next thing you want to do is determine the "id" of the device, so that you can uniquely identify it. The problem is that udev sets up devices in order of connection (/dev/sda, /dev/sdb, ...). To remedy this, use the symbolic links that udev creates under /dev/disk/ to identify the disk by one of the following means:

/dev/disk/by-id/ - Uses connection, make, model, and serial number

/dev/disk/by-uuid/ - Uses the UUID given to the partition when it was created

/dev/disk/by-label/ - If you have used e2label, this is more meaningful than the UUID

/dev/disk/by-path/ - This is the one you do not want to use!

I want to disable normal users (including myself) from modifying the contents of the backup, so the plan is to keep it unmounted and disable users from mounting it. Thefefor, this is the appropriate entry for me in /etc/fstab:

/dev/disk/by-id/usb-Seagate_FreeAgentDesktop_30DFK39D-0:0-part1 /media/FreeAgent        ext3    defaults        1 2

Note this will require the device to be present and consistent at boot-time. If you don't need this, change the last two numbers in the fstab entry to zero.

Before automating backup, I had a problem: The drive spins down automatically, so if you use only a command like rsync it may terminate due to an I/O Error because it simply timed out. This happened with me using ls, but it doesn't always happen -- so you should be more paranoid about the power mode.

The solution is to use a utility called sdparm, which is made to control SCSI (and SATA) disks. This is available for Fedora in a yum package called simply sdparm. This allows you to send a command to the drive to "start" (spin up) or "stop" (spin down).

Then, the final task is to create a backup procedure that is invoked by crond. Rather than writing something to /etc/crontab, I added the following file to /etc/cron.daily/. Make sure it has execute permissions, and call it (as root) a couple times to make sure it's working.

#!/bin/sh
#
# Backup to a USB disk 
#

# Env variables 
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/

# OK to change these
LOGFILE=/var/log/backup-usb.log
DISKDEVICE=/dev/disk/by-id/usb-Seagate_FreeAgentDesktop_30DFK39D-0\:0
DISKPARTITION=$DISKDEVICE-part1
MOUNTPOINT=/media/FreeAgent

# Leave these alone
DEVICEFILE=`readlink -f $DISKDEVICE`
PARTITIONFILE=`readlink -f $DISKPARTITION`

# Close stdout
1>&-;
# Close stderr
2>&-;

# Log needs to be rotated manually
#mv $LOGFILE $LOGFILE.1

# Direct stdout and stderr to logfile
exec 1>>$LOGFILE;
exec 2>>$LOGFILE;

echo Begin $0, `date`;

# Preliminary work (get backup device online)
if [ -e $DISKDEVICE ] ;
then
 echo "Device exists (is connected).";

# /usr/bin/sdparm --all $DISKDEVICE

 echo "Sending start (wake up) signal...";
 /usr/bin/sdparm --command=start $DISKDEVICE
 if [ "$?" -eq "0" ] ; 
 then 
  echo "Success";
 else 
  echo "Failed to start device";
  exit;
 fi

 # Get backup partition mounted
 if [ `grep $PARTITIONFILE /etc/mtab | wc -l` -le "0" ] ;
 then
  echo "Device $PARTITIONFILE is not mounted!";
  mount $PARTITIONFILE $MOUNTPOINT;
  if [ "$?" -eq "0" ] ;
  then
   echo "Mounted OK.";
  else
   echo "Failed to mount.";
   exit;
  fi
  
 else 
  echo "Partition $PARTITIONFILE is already mounted.";
 fi
else
 echo "Device doesn't exist, must not be connected!";
 exit;
fi

# Backup procedure 
# 
# For now I'm just using RSYNC because these files don't often change
# 
rsync --verbose --itemize-changes \
--archive --hard-links --partial --delete-before \
/opt/srv/ $MOUNTPOINT/srv/ && \
date && \
df -h $MOUNTPOINT && \
echo ;

# NOTE might also want to un-mount the device at this point 
# to prevent users from modifying it directly !
echo "Unmounting $PARTITIONFILE"
umount $PARTITIONFILE
echo "Stopping device $DISKDEVICE"
/usr/bin/sdparm --command=stop $DISKDEVICE
# /usr/bin/sdparm --all $DISKDEVICE && \

echo End $0, `date`;

The variables section and the rsync command have to be updated to fit your installation and your backup scheme.

Again, this worked well for me, you may have to make adjustments.

Shell Script to Reorganize a Mirror to Match a Newer Mirror

2008-01-18T14:55:00.000-05:00

This script has a very specific purpose, but you've probably had a chance where it could have come in handy. The idea is, if there are two mirrors of a bunch of files, and you reorganize (more around and/or rename) files on one mirror, then when you go to RSYNC the other mirror, it thinks the files that were only moved were added and deleted.

To use this script, you would first move the files on one mirror, and generate an MD5 sum file, as in:

$ find ./ -type f -exec md5sum {} \; | tee checksums.md5

Then, copy this small file to the root of the other mirror, and invoke the script with the name of the checksums file:

$ md5sum-reorg.sh checksums.md5

The script will process every file under the working directory and attempt to correct its path so that it matches a file in the new 'checksums.md5'. Files not found in the checksums file are left alone. Files already in place are left alone (obviously), without even calculating their md5sum.

Script follows:


#!/bin/bash
#
# This file takes a checksum file (output from md5sum utility)
# and attempts to reorganize the files in the directory to 
# match the listing in the md5 file. 
#
# Files not found in the md5 input are left alone.
# Files already in the right place are left alone.
# Other files have their checksums computed and, if they are found 
# in the md5 input, they are moved to the appropriate location.
#
# WARNING: It confuses duplicate files!!!
#

if [ $# -lt 1 ];
then
 echo "Usage: $0 [checksums file]";
 exit;
fi

declare -a SUMS
declare -a NEWPATHS

exec 10<$1
let count=0

echo "Parsing checksums and paths from input file...";

while read LINE <&10; do
 SUM=`echo "$LINE" | cut -d' ' -f1`;
 NEWPATH=`echo "$LINE" | cut -d' ' -f1 --complement | cut -d' ' -f1 --complement`;
 SUMS[$count]=$SUM;
 NEWPATHS[$count]=$NEWPATH;
 ((count++));
done
# close file 
exec 10>&-

echo "Compiling list of files that need to be checked...";

TMPFILE=`mktemp`;
find ./ -type f -printf "%p\n" > $TMPFILE;

exec 10<$TMPFILE
while read OLDFILE <&10; do
 echo "Trying to find new path for $OLDFILE";

 SKIP=0;
 let count=0;
 while [ $count -lt ${#NEWPATHS[@]} ] ; do
  if [ "${NEWPATHS[$count]}" == "$OLDFILE" ]; then
   echo "File already exists at '${NEWPATHS[$count]}'";
   SKIP=1;
   break;
  fi
  ((count++));
 done

 if [ $SKIP -eq 1 ]; then
  continue; #skip the rest of this iteration
 fi


 echo "Computing checksum of $OLDFILE";
 OLDSUM=`md5sum "$OLDFILE" | cut -d' ' -f1`;

 # iterate over the pair of arrays until we might find a matching sum
 let count=0;
 while [ "$count" -lt ${#SUMS[@]} ]; do 
  SUM=${SUMS[$count]};
  NEWPATH=${NEWPATHS[$count]};

  if [ "$SUM" == "$OLDSUM" ];
  then
   if [ "$OLDFILE" != "$NEWPATH" ] ;
   then
    NEWPARENT=`dirname "$NEWPATH"`;
    if [ ! -d "$NEWPARENT" -a "$NEWPARENT" != "." ];
    then
     echo "Making directory $NEWPARENT";
     mkdir "-p" "$NEWPARENT";
    fi
    echo "Moving $OLDFILE to $NEWPATH";
    mv "$OLDFILE" "$NEWPATH";
   else
    echo "Path hasn't changed.";
   fi
   break;
  fi
  ((count++));
 done

done

exec 10>&-

exit

In case it's not clear, this is offered without any warranty or guarantee whatsoever.

Scripts for Moving Large Files to DVD (GNU+Linux environment)

2008-01-11T23:55:00.000-05:00

Often, we have tons of files (videos, ISO's, software packages, etc.) that we're done with but don't want to throw away. I have written a few scripts to pack a directory of these kinds of files up into volumes (without any kind of compression) so that they can be written to DVD. Each volume should have an MD5 sum file generated which provides both a directory for that volume (since you can keep a copy locally and search using grep) and a means to verify its integrity.

The first script attempts to "pack" files at the current directory level (it's non-recursive) into volumes of a specified size. This is really the only part of this post that warrants a real script since it's a nested loop. Note that really small files will always end up in the extra space on the first volume, so you may want to move them first.


#!/bin/sh
#
# Script takes files (or directories) in the current working directory 
# and moves them to subdirectories for writing out to discs
#
# This allows collections of relatively same-sized files or directories
# of files to be packed into volumes for storage on optical media.
#
# Modified to output a shell script instead of making any changes
# Disk capacity can now be entered

# Defaults
discSize="0";
discDefaultSize="4380000";
discInitNumDef="0";
discInitNum="0";
scriptPathDef="pack.sh";
diskPath="disks";

echo -n "Enter the volume number at which to start [$discInitNumDef]: ";
read discNumOffset;

if [ "$discNumOffset" == "" ] ;
then
 discNumOffset=$discInitNumDef;
fi
echo $discNumOffset;

echo -n "Enter the maximum capacity of the media [$discDefaultSize]: ";
read discMaxSize;

if [ "$discMaxSize" == "" ] ;
then 
 discMaxSize=$discDefaultSize;
fi
echo $discMaxSize;

echo -n "A shell script will be output, move files now? [y/N]";
read moveFiles;

if [ "$moveFiles" == "" ];
then
 moveFiles="N";
 echo "Not going to move files.";
fi

echo -n "Enter the path to save the shell script [$scriptPathDef]: ";
read $scriptPath;

if [ "$scriptPath" == "" ] ;
then
 scriptPath=$scriptPathDef;
fi

echo "Going to write shell script to '$scriptPath'.";

# Declare disk size array
diskSizes[0]=0;
arraySize=1;

echo "#!/bin/sh" > $scriptPath;

if [ ! -d "$diskPath" ];
then
 echo "mkdir \"$diskPath\";" >> $scriptPath;
fi

if [ ! -d "$diskPath/`expr $discNum + $discNumOffSet`" ] ;
then
 echo "mkdir \"$diskPath/`expr $discInitNum + $discNumOffset`\";" >> $scriptPath;
fi

for file in * ;
do

if [ "$file" != "$diskPath" -a "$file" != "$scriptPath" ] ;
then 
 echo "$file";

 discNum=$discInitNum;

 newSize=`du -s "$file" | cut -f1`;
 #discSize=`du -s todisk/$discNum/ | cut -f1`;
 #discSize=`expr $diskSizes[$discNum] + $newSize`;
 discSize=${diskSizes[$discNum]};

 echo "newSize = $newSize, discSize = $discSize";

 if [ $newSize -gt $discMaxSize ] ;
 then
  echo "$file is larger than the disc size, skipping it.";
 else
  while [ `expr $discSize + $newSize` -gt $discMaxSize ] 
  do
   echo "Won't fit in $discNum + $discNumOffset: $discSize + $newSize > $discMaxSize";

   discNum=`expr $discNum + 1`;
   
   if [ $discNum -ge $arraySize ] ;
   then
    #diskSizes[$diskNum]=0;
    diskSizes=( ${diskSizes[@]} 0 );
    arraySize=`expr $arraySize + 1`;

    if [ ! -d "$diskPath/`expr $discNum + $discNumOffset`" ];
    then
            echo "mkdir \"$diskPath/`expr $discNum + $discNumOffset`\";" >> $scriptPath;
    fi


   fi
   
   discSize=${diskSizes[$discNum]};
  done
  
  echo "Going to move $file into $discNum to make $discSize kb `expr $discSize + $newSize`";
 
  echo "mv \"$file\" \"$diskPath/`expr $discNum + $discNumOffset`/\";" >> $scriptPath;

  # Update disc size entry
  diskSizes[$discNum]=`expr $discSize + $newSize`;
 
 fi

fi

done

echo "Disk sizes:";

for DISC in ${diskSizes[@]}
do
 echo "$DISC kb";
done

exit;

The next, albeit more simple script creates checksum files (for later use with md5sum -c ...) in each volume and provides the option to save a copy of the checksum file to another location.


#!/bin/sh
#
# Create checksum files for disk volumes generated by 'disk-pack'.
# These files allow the fidelity of the optical media to be 
# evaluated, and allow the contents of the disk to be catalogued.
#
# This file should not change any files; only add new files. 
#

CATDIRDEF="`pwd`";
echo -n "Path to save a duplicate of the MD5 checksums [$CATDIRDEF]: ";
read CATDIR;

if [ "$CATDIR" == "" ];
then
 CATDIR=$CATDIRDEF;
fi

echo "Saving duplicate checksums in '$CATDIR'";

if [ ! -d "$CATDIR" ];
then 
 echo "Directory doesn't exist.";
 exit;
fi

PREFIXDEF="disk";
echo -n "Prefix to use in checksum file names [$PREFIXDEF]: ";
read PREFIX;

if [ "$PREFIX" == "" ];
then
        PREFIX=PREFIXDEF;
fi
echo "Using prefix '$PREFIX'.";

for DISK in [0-9]* ;
do

 if [ "$DISK" != "." -a "$DISK" != ".." ]
 then
  echo "Processing volume $DISK";
  cd $DISK;
  find . -type f -exec md5sum {} \; | tee ../tempsums.md5;
  cd ..;

  if [ -e "$CATDIR/$PREFIX$DISK.md5" ];
  then 
   echo "WARNING: Catalog file already exists, using alternate name.";
   NUM="0";
   while [ -e ""$CATDIR/$PREFIX$DISK-$NUM.md5 ]; do
    NUM=`expr $NUM + 1`;
   done
   cp tempsums.md5 $CATDIR/$PREFIX$DISK-$NUM.md5;
  else
   cp tempsums.md5 $CATDIR/$PREFIX$DISK.md5;
  fi

  if [ -e "$DISK/$PREFIX$DISK.md5" ];
  then
   echo "WARNING: File $DISK/$PREFIX$DISK.md5 already exists, using alternate name.";
   NUM="0";
   while [ -e $DISK/$PREFIX$DISK-$NUM.md5 ]; do
    NUM=`expr $NUM + 1`;
   done
   mv tempsums.md5 $DISK/$PREFIX$DISK-$NUM.md5;
  else
   mv tempsums.md5 $DISK/$PREFIX$DISK.md5;
  fi
 fi
done

Finally, you're ready to put these volumes out to optical media (since you've minimized internal fragmentation, captured a catalog of the files, and took an extra step to preserve integrity). You can do so using your favorite method, but when there are many volumes (like more than three) I prefer to take the following steps.

The following command, if you have genisoimage will create a .iso file for the directory '40' and it will have the volume name "Volume40" when you mount it.

genisoimage -o volume40.iso -J -r -V Volume40 40/

After you have a .iso file, you're almost ready to burn. Always, always, always mount the ISO image (mount -o loop -t iso9660 volume40.iso isotest/), enter it and check some of the MD5 sums to make sure you have a good .iso file! You'll have to check the man page for genisoimage and make sure you're providing the command-line options correctly if the files in the ISO seem corrupted.

If you're familiar with cdrecord, it is now provided by wodim. You need to be root. The command looks like:

wodim -v -eject speed=8 dev='/dev/scd0' volume40.iso

Then, before I delete anything, I always insert the CD, preferably into another optical drive, and run md5sum -c volume40.md5. Now you know you have an exact copy, you can put it in a case and delete the original. Note I'm assuming that if the disc fidelity decays that the files can be found again from the Internet--make sure you have even more redundancy if these are your personal files!

New Split Directory Script

2008-01-11T23:49:00.000-05:00

Follow-up to: "Making directory traversals more efficient"

Back at the beginning, I posted a lengthy script that would split up a congested directory alphabetically. Recently, I needed it again, but needed it to be smarter, so I re-wrote it. Also, I figured out how to insert code into Blogger. Enjoy.


#!/bin/sh
#
# by Ryan Helinski, January 2008
#
# This is the second revision of a script that should be used 
# when there are too many files or directories at a single level
# on the file system. 
#
# It now recognizes the word, "the", and that the name should 
# be alphabetized by the words following.
#
# A script is output so the changes can be reviewed before
# any are made. 
#
# The script should add to existing bin directories if they 
# already exist.
#
# A further improvement would be to allow the split to be 
# multi-level.

BINS=(0-9 a b c d e f g h i j k l m n o p q r s t u v w x y z);
BIN_EXPS=(0-9 Aa Bb Cc Dd Ee Ff Gg Hh Ii Jj Kk Ll Mm Nn Oo Pp Qq Rr Ss Tt Uu Vv Ww Xx Yy Zz);

SCRIPT_FILE=".script.sh";

echo "#!/bin/sh" > $SCRIPT_FILE;

for BIN in ${BINS[*]};
do
 if [ -d $BIN ];
 then 
  echo "mv $BIN .$BIN" >> $SCRIPT_FILE;
 else
  echo "mkdir .$BIN" >> $SCRIPT_FILE;
 fi
done

INDEX="0";
while [ "$INDEX" -lt "${#BINS[*]}" ];
do
 echo "mv [Tt][Hh][Ee]\ [${BIN_EXPS[$INDEX]}]* .${BINS[$INDEX]}/" >> $SCRIPT_FILE;
 INDEX=`expr $INDEX + 1`;
done

INDEX="0";
while [ "$INDEX" -lt "${#BINS[*]}" ];
do
 echo "mv [${BIN_EXPS[$INDEX]}]* .${BINS[$INDEX]}/" >> $SCRIPT_FILE;
 INDEX=`expr $INDEX + 1`;
done

for BIN in ${BINS[*]};
do
 echo "mv .$BIN $BIN" >> $SCRIPT_FILE;
done

ANSWER="";
while [ "$ANSWER" != "yes" -a "$ANSWER" != "no" ];
do
 echo "Script written to \"$SCRIPT_FILE\", execute now? (yes, no)";
 read ANSWER;
done

if [ "$ANSWER" == "yes" ];
then
 sh $SCRIPT_FILE;
 
 ANSWER="";
 while [ "$ANSWER" != "yes" -a "$ANSWER" != "no" ];
 do
  echo "Delete script file? (yes, no)";
  read ANSWER;
 done

 if [ "$ANSWER" == "yes" ];
 then
  rm "$SCRIPT_FILE";
 fi
fi

exit;

Make OGG not MP3

2007-12-04T09:39:00.000-05:00

In reference to recent news, MP3 is still gaining momentum, as Wal-Mart is replacing their Windows Media Audio (WMA) files with MP3. In my humble opinion, MP3 is still better than WMA because it is so well understood (coders, decoders, support, household name, etc.) and WMA is really just a way for Microsoft to get a piece of the Fraunhofer pie. However, as long as MP3 is really "gaining momentum", then we need to collectively do something about it...

Personally, I've made the following pledges and hope they inspire someone:

I will not distribute MP3
I deal with a lot of original content, and after mix-down there's a choice--how to encode it to let someone hear it over the Internet? I will always choose FLAC or Ogg Vorbis, and refuse to send an MP3.
I will maintain my personal collection in Ogg Vorbis
When I buy a new CD, the first thing I do (usually) is run it through GNOME Sound Juicer (which supports OGG and FLAC by default) and create Ogg Vorbis files so I can load it onto music player (software) or portable media player (hardware), and to protect the data from being damaged by wear on the CD. This is the cheif method that OGG could become ubiquitous. Transcoding MP3 to OGG is lossy (it's much better to start from the original sampling), and any audiophile would tell you it's not worth it. Therefor, the most important source of OGG files is at the source (audio CD, FLAC, etc.), and we can become less dependent on MP3 by replacing MP3 files with newly-encoded OGG files.
I will not buy a portable media player that does not support free codecs like Ogg Vorbis
Okay, I still want to be able to listen to MP3, but in order to support pledge #2, I need my portable media device to support OGG out of the box. I personally have a COWON (iAudio) T2, but there are many other options from abroad.
I prefer Ogg Vorbis Internet radio over other formats
Hey, if OGG has the same fidelity at a lower bandwidth (you'll note the rates on Internet streams have a lower average), then we're saving the host bandwidth by preferring an Ogg Vorbis stream. Also, since Internet radio is struggling with royalties, why don't they save themselves money by dropping their MP3 streams and providing only Ogg Vorbis streams? I feel that most Internet radio listeners would not mind having to download an additional codec (if they even need to) to make the switch.

These are reasonable commitments to make, and the second big corporations catch on, we'll start to see MP3 dwindle away. Ogg Vorbis will have to make a statistical win (what is the probability of an audio file being an OGG?), and a compatibility win (what is the probability I can play this OGG?) in order to acheive this.

Note that DRM is not even in my vocabulary--it seems like the only "solution" to DRM is to avoid it at all costs. This business of limiting where, when and who can listen to a file (that someone has paid for, mind you) is nonsense. This is exactly the reason we dropped vinyl albums and magnetic tapes for compact discs--so we could hear the same thing again whenever and wherever we want.

Finally, Gmail offers IMAP

2007-11-06T09:47:00.000-05:00

Previously Google claimed that IMAP would be too difficult to implement given their unconventional e-mail organization scheme (all e-mails basically in one folder). Now, they've finally opened up IMAP servers which allows much greater control than POP 3 previously did. For instance, one should be able to now import old e-mails into Gmail so that now all your mail is searcheable from one place!

GNU Screen -- The best thing since VNC?

2007-09-20T23:16:00.000-05:00

I have recently found myself in a bit of a blunder. I found myself opening a VNC server (with vncserver) and then connecting to it just to open a bunch of xterm's, and this seemed wasteful to download pictures of a text-mode terminal. I also SSH in and run long-term tasks and want to disconnect--but don't want to use nohup because I might want to re-gain control of the program or just see in real-time what the output is.

Then I remembered I read the description of a software some time ago in pirut, and thought it was a great idea. This software, as I remember it, was much like what I've just found here, but it seems like this project is defunct. I know there was something which was its own daemon and allowed persistent sessions, but...

After a lot of Googling, I finally came accross GNU Screen. I find this software awesome! Basically, it's like a window manager for your Secure Shell, allowing multiple new shells to be opened and switched between, "screen splitting" so you can see two screens at once, persistence of your tasks--which was exactly what I was looking for, and other features like sharing the session between users. Now I can have my multiple terminals running on the remote machine and use them effectively and more efficiently than via VNC.

The Wikipedia article links to many good tutorials in its references section. I recommend you try a few of them, especially this one, but if you're like me--you need to know right away:

install: yum install screen

run: screen

disconnect: CTRL+a d

reconnect: screen -r -d

Disclaimer: Yes, VNC is newer than Screen. No, Screen is not convenient when you need to use GUI's since even if you get it to work, it's forwarding who X windows, which can be much too slow if you're connecting over WAN. VNC is definitely the way to go if you have GUI programs you want to use in conjunction with text-mode.

Happy screening!

Jamendo

2007-09-03T19:32:00.000-05:00

Jamendo is an excellent program which allows artists to release musical works into the public domain under various formats. With plugins for programs like Rhythmbox, you can use the service as an Internet Radio, and if you like something you can visit the artist's site on Jamendo or get the whole album on BitTorrent. Recordings are uploaded in WAV for FLAC, and are distributed in OGG and MP3.

If you are an artist with content that you feel would should be in the public domain, sharing your original recordings is easy. More importantly, you can get feedback on your recordings from the Jamendo community, which can be exciting because all the opinions (although sometimes misguided) are impartial--which is more than you can expect of your friends :) .

SourceForge.net User Network

2007-09-01T13:22:00.000-05:00

Anyone can join SourceForge.net, even if you don't have a project to share. Many projects need testers, developers, documentation writers, and translators (and more). There's a long list of projects which have advertised they need assistance (http://sourceforge.net/people/). It's a great way to help out the SF.net community and gain coding experience both by examining code and by extending/adding new code.

Configure OpenSSH to automatically authenticate

2007-08-30T07:21:00.001-05:00

Sometimes entering your pass-phrase to gain SSH access can get redundant, or you may want an automated script to be able to authenticate as you (as described at http://sial.org/howto/rsync/).

I've been trying to do this for some time now--apparently if you read the man pages carefully enough you can figure this out. I thought I'd need a more advanced SSH client, but OpenSSH already has everything you need--the ssh-keygen program is key. Following is a brief walk-through of how to configure connection from client to server for SSH. Replace the words client and server as appropriate.

client$ ssh-keygen -t rsa

It will ask where to save, you want the default since this is where ssh will look. It will ask for a pass-phrase, and since we're trying to get around having to enter a pass-phrase, we don't want to protect our authentication, so just hit enter twice. Then it will give you a fingerprint and put files in your ~/.ssh directory.

Copy the ~/.ssh/id_rsa.pub to the server machine taking care not to overwrite the same file there:

client$ scp ~/.ssh/id_rsa.pub user@server.address:.ssh/client.pub

Now SSH to the server and add the public key to the authorized keys file like so:

server$ cd ~/.ssh
server$ cat client.pub >> authorized_keys
server$ rm client.pub

The SSH server won't let you use the key unless the file is secured. That is, the keys should only be readable by you.

server$ ls -l authorized_keys
-rw-rw-r-- 1 user users authorized_keys
server$ chmod og-rw authorized_keys

Now we should be able to open an SSH session simply by:

client$ ssh user@server.address

where you can omit the 'user@' part if the user names are the same on the local and remote machines.

Finally, note that doing all this means if you leave your machine open, an intruder has more doors they can open! This is why you should always lock and/or time-out your screen. If the key has no passphrase, then if someone copies it, they can gain access to the machine you opened with this procedure. Therefore, you should restrict access to your machine by configuring /etc/hosts.allow and /etc/hosts.deny. If you're concerned, don't let the passphrase for the key be null (use a passphrase). Taking these steps can help protect your key.

Cost Review for Luxeon LED Headlight

2007-08-16T23:53:00.000-05:00

To justify building on myself, I had to do a cost analysis.

Purchased:

4x D-cell Batteries ($5)
1x LED Dynamics BoostPuck ($35)
4x Lumileds Luxeon I Star/O ($25)

Subtotal: $65

Spare parts:
1x 270uF capacitor
1x 24-gauge (AWG) lead wire
1x Pentium3 Heat Sink
1x Mounting bracket
1x Rocker switch
2x 2 D-cell Battery Holder
1x 18-gauge (AWG) lead wire

Note that comparable systems are water-resistant and include a rechargeable battery system, but can cost from $120 up to $300.

Rebuild of Luxeon I Bicycle Headlight

2007-08-16T16:20:00.000-05:00

This is a practical comparison between two light sources: an old 2-C cell halogen headlight and a single Luxeon I Star (with optics), both driven at the proper power. This is an image with no special adjustments and the sources about 5 feet away. It is easy to see that the Luxeon I has both more flux (total amount of light) and better throw (maximum intensity of beam). Note that halogen bulbs are known to be more efficient than traditional vacuum bulbs, see Lighting efficacy for more. According to my calculations, the power delivered to the halogen bulb is about 1.5 Watts, and the power to the Luxeon I (obviously) is 1 Watt. Note the spectrum of light output, and how this would make riding safer. This comparison allows the efficiency of the LED to really "shine".

Note that my power supply (limited to 1 amp) may not have been able to provide enough current to drive the Halogen bulb to the same power as the 2 C-cells, so the halogen light may get brighter with actual batteries. However, the power measurements quoted are absolutely correct and (together with the comparison picture) still illustrate the point of efficiency. Also note that only one Luxeon I (1-watt) is switched on in the comparison picture, whereas four are used in my headlight assembly.

The LED Dynamics BoostPuck 350mA current regulator came today. The model with lead wires attached was not available, so I had to solder on wires. It really is small as they say; even smaller than my rocker switch. I left the extra pins (reference voltage, and dim) not connected.

Using my power supply, I verified all the parts and connections were OK. The meter pictured is showing volts across the input power pins of the BoostPuck (out of frame). The system is was actually pulling down the voltage on my 1-amp-maximum power supply set at 6.0V by 200mV. You can see that the LED's are fully illuminated, which requires at least 12 Volts in their wiring configuration...

I soldered everything together as I mentioned I would in my last post (in series). I included a 270uF capacitor. The only problem with the BoostPuck is that it's a sealed unit and has no way to mount it to anything, unless you use the pins to anchor it (which I don't). It's not heavy, so I didn't think it's all that important. Using industrial hot glue, I fixed it to the rocker switch, and the capacitor to it as well. I expect the hot glue not to hold for long, especially if things heat up. I'll use some epoxy if all goes well--the hot glue comes off easily, epoxy won't.

The light output looks about right. It might be a little dimmer than the unregulated circuit, but it's worth the gains in battery efficiency/power delivery. The BoostPuck is producing over 12VDC across the four LED's in series, which is about what I expected. I haven't actually checked, but I'm assuming that the diode current is 350mA. I didn't make measurements of the current drawn from the batteries because of the AC components.

The light beam itself is continuous (as opposed to strobed); waving hands in front of the beam doesn't reveal individual light pulses as with other charge-pumping lights I've encountered. Also, the intensity differences between the new and old LED's is gone as I expected. The newer LED's needed a higher voltage than the older LED's to produce the same amount of light, and fixing the current at 350mA equalized the light intensity.

For clarity, below is a crude circuit diagram including all the parts.

Luxeon LED headlight assembly

2007-08-14T20:28:00.000-05:00

Have an old processor heat sink laying around? This Pentium 3 heatsink is the perfect size for four Luxeon I Star/O assemblies (LumiLEDs Product# LXHL-NWE8). Luxeon light sources need very good heat dissipation, and this might be over-doing it, but why not prolong the life of the light sources?

I wanted to get around paying up to $300 for a battery-powered bicycle headlamp, so I decided to meet or exceed the expectations with available parts.

I initially bought only two of these Luxeon Stars (with optics) over a year ago. I didn't have a good way to drive them other than my Elenco Precision (TM) power supply and alkaline batteries, both of which are voltage sources. I tried using two D cell alkaline batteries, this provided 3.0V which is less than the typical 3.4V, and not didn't force enough current to for the LED's to be all the way on. I also didn't have a good mounting bracket, or a good way to mount the batteries to the bicycle frame.

Recently, using the mounting bracket from and old mechanical speedometer and zip ties, I was able to make an excellent mount. This stops the lights from changing pitch (very important for this application) or breaking free. I connected them all in parallel to try to give them all 3.0V from the two D cells.

Lesson learned: Don't solder when the device is a metal core PCB and it's sitting on a heat sink with silicon paste in between! Leave these PCB's on an insulator while soldering.

I noted that they were only drawing around 800mA, nowhere near 1.4A (4x350mA). I decided to increase my power supply voltage potential, but this meant there needs to be another component in series with the light sources to allow the excess voltage to be dissipated. In the application notes from LumiLEDs, they recommend using a 6V power supply and a 10 Ohm resistor for a Luxeon I. I added another battery pack to the frame to get 6.3V, but I decided to use 0.9V rectifying diodes (3A max) rather than four resistors (one in series with each supply) which would require desoldering. This achieved 1.6A with one diode in series and 1.1A with two diodes in series. I stuck with the two diodes in series to stay on the safe side. The diodes get rather hot very quickly since they're dissipating so much power. This is a major problem, by my measurements -- 45% of the power was being dissipated in the diodes themselves, delivering only 55% of the power from the batteries to the light sources. Another reason that a passive approach (linear mode voltage regulation) is bad for alkaline battery applications is that the higher the power delivery, the more the batteries self heat and increase internal resistance. In other words, Alkaline's are less efficient when discharged quickly. It is a common trend that batteries are more efficient with intermittent loading for reasons related to battery theory. For much more, see the IEEE paper BatteryDriven System Design: A New Frontier in Low Power Design, by Kanishka Lahiri, Anand Raghunathan, Sujit Dey, and Debashis Panigrahi.

Lesson learned: The voltage drop across an LED is related to the energy released in each photon--relating to the wavelength of the light emitted. (Note that this is the P-N junction voltage drop, and differs from the external voltage drop and light spectrum is also highly dependent upon construction materials). The current through the LED corresponds to the number of photons emitted--relating to the intensity of light emitted. These brief rules of thumb really help in understanding how to apply high-power LED's.

Lesson learned: Luxeon LED's are constant-current devices, so to get them to the proper luminous intensity and power consumption, they need to be driven at the proper current amperage. For Luxeon I's this is 350mA, for Luxeon III's this is 700mA, etc. This point remained subtle to me because I figured I could always just find a voltage potential which would result in the proper current being drawn. The problem is that LumiLEDs has a tighter manufacturer tolerance of 350mA operating conditions than the forward voltage (Vf) which is loosely maintained. This means each light source will require a slightly different Vf to draw 350mA and use run at the full 1 Watt. The most robust solution is a constant current source which can be used to drive multiple light source in series. This also allows the supply voltage to fluctuate (eg: a draining battery) and not affect the luminous intensity as it would without a regulator.

Lesson learned: A common optimization for alkaline battery-powered applications is to use a charge-pumping power regulator. These are capable of producing voltages higher than their input voltage (12VDC from 6VDC for instance) by intermittently presenting a larger load and driving an oscillating circuit. The typical result is luminous intensity remaining mostly flat until the battery is mostly depleted. Just connecting a 12VDC battery supply would result in an exponentially decaying luminous intensity.

These conclusions lead me to replace my diode-dissipating circuit with a switch-mode power supply including a boost circuit. Because of the scarce availability of through-hole mounting switch-mode power supplies and the necessity for a PCB, I was very happy to find products from LED Dynamics including the BoostPuck. Their Application Figure 3 in their datasheet is my exact application. The regulators are unfortunately around $34, but it's worth the efficiency of a switch-mode power supply, the optimized battery utilization, and proper driving (constant current of 350mA) of the light sources to get the most light without risking burning out the LED.

When the unit arrives, I'll removed the hot glue (to keep the parts clean) and connect the light sources in series and connect the BoostPuck to the ligth sources after the rocker switch. As per the application notes, I'll need to include a capacitor since my batteries are a few feet away. More as soon as the part comes in.

1979 Free Spirit Ten Speed

2007-08-03T12:15:00.000-05:00

Once I started riding this old Free Spirit ten-speed, I was hooked. Most of my riding is on a paved recreational trail, so the thin tires aren't a hazard and the bike is a lot more efficient than a Mountain Bike on those conditions.

The bicycle came from Sears & Roebuck circa 1979. I decided that even if I buy a new road bike, that I wanted to keep the Free Spirit around since it was in such good condition compared to a few much younger bikes I have seen. The parts are old (Shimano) parts, but most of the following procedures still apply to most bicycles.

The first problem was that the wheels needed to be "trued" very badly, the crank was loose, it had lots of grime around the chain hardware, the brakes needed to be tensioned, and it had some out of order accessories. The accessories it had were a luggage wrack in the back which had a broken support (which is laying on the ground behind the bike in the picture), and a mechanical speedometer which had a loose cable and just made chirping noises. It also had "soft" plastic handle grips on the end of the dropped handle bars which were petrified and provided no cushioning.

I started by disassembling the whole thing! I cleaned all the parts with CLOROX(tm) disinfecting wipes because they have a cleaning agent which does no harm and are great for absorbing grime. I put the luggage wrack aside because it just got in the way of mounting/dismounting (and was broken). As for the speedometer, modern bicycle computers (speedometer, odometer, tripometer, etc.) are now inexpensive, under $12 at your local "superstore", and allow you to program your wheel size (unlike the mechanical one), so I replaced the huge speedometer with the little computer.

I took the wheels to the local bike shop and had them trued for $5/wheel. They still have a little jitter, but it's nothing compared to the thumping and wobbling that they had before. They had both lateral (off rotation axis) wobble and radial (tending to not form a circle) wobble. I can ride over 20 mph now and not leave the ground.

While the wheels were being worked on, I was able to remove the plastic grips a lot easier than I expected (these things have tendency to become attached). Rubber grips can be a real pain, but I figured the plastic grips would have been fused to the handlebars. I just inserted a flat-head screw driver (on the bottom side to avoid any visible scratching) in between the grip and the handle bar. I didn't pry with the screwdriver, rather just inserted and removed it several times, spraying WD-40 down the length of the screwdriver each time. After about three times, I was able to break each grip free by trying to twist the grip around the handlebar. Once it came free, I took out the screwdriver and worked it off. I cleaned the lubricant off the bars. I bought some handlebar gel tape from Bartrager, and installed them according to these directions since there were none in the box. I guess I have long handlebars because I didn't have to cut the length of the tape, I just made a cut to taper the distal end of the tape so it was flush with the handlebar cap.

The bicycle has a disk brake in the back which needed to be manually tightened because it didn't lock the wheel. The brakes in the front were also too loose. Tip: If brake pads don't lock the wheel, they need more tension. Take the wheel off of the frame, and ask an assistant to hold the pads together. If your caliper makes it easy to disengage the recoiling springs, you won't need an assistant. Taking the wheel off will provide enough slack to make the adjustment--I found it was impossible otherwise. Adjust where the steel cable is fastened on the caliper side about a centimeter and tighten everything back up.

The bike was back in order but the crank still felt bad because it moved inside the frame. It's a one-piece crank and is a lot simpler than I thought before. Take the chain off first. It has a big locking nut which you'll need a big wrench for. Note that the threads are in the opposite direction from most threads, so you take the nut off by turning it in a clockwise direction. Under that, there were some washers which act as dust covers. There is a big nut that the ball bearings ride on which needs to be finger-tightened, it was loose and this is where the crank wobble came from. I de-greased the ball bearings and the journals inside the frame. It was essentially dry inside just a lot of black deposits on everything. There was some fatigue on the right-hand (gear) side where the ball bearings either crashed down or the bearing retainer was hitting. With one-piece cranks, look inside where the crank fits and look for the obstructions from the other tubes. Find a place to fit the corner of the crank and angle the crank through as to take the gear away from the frame. This will give you more space and will let the crank rest. After everything was clean (I had to use an abrasive brush in some places), I greased everything up as much as I could and put everything back together. I used a screwdriver to make the final torque on the hand-tightened nut, I stopped as soon as I felt no wobbling of the crank. When you put everything back together, you should be able to spin the crank and it should keep spinning for a while. If it doesn't make it more than a few turns after you let go, it's too tight.

After a few other adjustments, like the kickstand position, everything was ready to go. The bike feels a lot tighter like a new bike would. All the rattling, chirping, and wobbling is gone.

The bike will need new tires soon (the thin tires don't last very long), I have about 300 miles on mine and the tread in the back is disappearing. When I replace these, I'll try to disassemble the bearings on the wheels and try to revive them like I did the crank bearings. I'll also replace the chain at that time.

Mount USB drives on boot

2007-07-17T11:54:00.000-05:00

On Fedora Core 7, leaving a USB drive (a simple flash disk) connected doesn't seem to be mounting after reboot even though it's properly listed in the /etc/fstab. I went through a few solutions to this, like this one, but the only one that worked for me was modifying the /etc/rc.sysinit script as I explain below.

I seem to remember creating an entry in the fstab, and allowing the auto option (which is part of the default specification), would make your USB partition mount on boot. The issue with FC7 seems to be that the usb-storage kernel module isn't being loaded when the fstab is first loaded, and that the USB devices aren't initialized yet by /etc/rc.sysinit. This seems to be common with all newer distros, in accordance with the move to udev, hal and other components, which facilitate plug & play user mounting and they don't need them to be available before entering a GUI. This is great for workstations, but I have a "headless" server configuration, therefor I'm not using GNOME to mount anything on-demand, but rather they should be available statically.

After a few days of trial & error, and inspiration from this post, I added the following lines to the end of my /etc/rc.sysinit. Note that changes to this particular file may be removed by system updates (Using yum) as I explain below.

# Try to mount all USB partitions
# This should be moved to `rc.local' and be prepended with a wait
modprobe usb-storage
mount -a

I believe this is the most most robust method, since simply adding the entry to the fstab will allow it to be mounted after the USB devices are initialized. It is preferred to use the /etc/rc.local script to issue this nature of commands (local/custom configuration), but this script seems to be run before the /etc/rc.sysinit script exits, meaning the USB partitions may not yet be available. Commanding the /etc/rc.local script to wait for the /etc/rc.sysinit script to exit may be a solution.

Finally, note that you can guarantee consistency of the partition before mounting it by carefully editing the corresponding fstab line in the normal way, see man 5 fstab for more.

Label your EXT3 partitions!

2007-07-14T14:16:00.000-05:00

If you have removable drives used for backup or portable media, and you're using EXT3 rather than FAT16 (the default format of most USB flash drives), mounting these drives by their Linux device name is ambiguous. For example, I have a hard drive in a Venus DS-3 with a single EXT3 partition on it. The problem is, making the entry in the /etc/fstab file. The /dev/sd[a-z][1-9] designation is not good enough because it changes with the order of the connections during boot-time or plug-in time.

Making labels for each partition to be mounted fixes the problem entirely. To create a label for an ext2/ext3 partition, use the e2label command:

e2label [device] [label]

My exact command was:

e2label /dev/sde1 "Venus1"

Note that I had to find the right drive using 'fdisk' before I knew it was 'sde1'.

The change takes place right away, and then you can put a line in your /etc/fstab like:

LABEL="Venus1"          /media/venus1           ext3    noauto          0 0

And it will run through the drives to find the right label. We've associated the mount location with the partition on the physical device. This is especially important if you're mounting the drives on a server and are not using any automatic mounting facility.

Some recommend to also label the static partitions if you're using only EXT3. If you're using LVM, the groups and volumes are already labeled.

Alternatively, you may use a UUID, obtained from

udevinfo -q env -n /dev/sdd1

Where sdd1 is your USB device, which you can find from

fdisk -l

There is a lot more info you can get with udevinfo but for now, just grab the ID_FS_UUID and you can use it in your /etc/fstab as follows:

UUID=492bc203-de77-40b6-be4b-49e84fee430e /data ext3 auto,defaults 0 0

See http://www.patrickmin.com/linux/tip.php?name=usbkey_mount and
http://floatingsun.net/articles/howtos/howto-usb-automount.html for more.

Narrow-angle Panormas with Hugin

2007-07-14T13:16:00.000-05:00

I have a telephoto lens and I wanted to stitch together several pictures of the Baltimore-Washington International airfield. Taking the pictures was just fine, although I hadn't learned how to set the exposure settings yet (see my post on this), and need to do it again on a clear day. What I didn't realize was that there's essentially no 'barrel distortion' with narrow-angle shots.

I found that it's hard for Hugin to figure out the angle of your shots when there's no barrel distortion. It misbehaves if you try to use the same procedure as you would for wide angle panoramas, and gives you completely insane average errors and if you try a preview--it won't be pretty. There are two things you need to do: tell it your horizontal field of view angle, and don't allow barrel distortion optimization. Only use "Positions and View (y,p,r,v)", and check that the barrel distortion stays close to 0.000 for now. This means that your panorama is essentially flat.

I was taking my panorama shots in portrait mode, which meant that my horizontal angle was 75% (the inverse of the aspect ratio 4/3) of the angle in landscape mode. That angle was about 8.10 degrees, and so the angle for each shot was 6.08 degrees. I got my focal length from the EXIF data, 224 mm, but the angle is most important.

Before final optimiztion and stitching, you may want to change the barrel distortion to something between 0.0034 and 0.006. This will add perspective to the panorama. Make sure you still don't run the optimizer for barrel distortion.

Panoramas with the Nikon D-50

2007-07-14T12:45:00.000-05:00

This is a panorama I made while learning the following procedure for the Nikon D-50. These same instructions should apply to most Nikon Digital SLR cameras, and can be translated to most other digital cameras. In fact, I find that the huge resolution of my camera doesn't really impact the final product because you want most of the image to fit on your computer monitor, and you're literally multiplying megapixel area each time you take a shot (your canvas gets big quick)--so for close range panoramas, cheap digital cameras are fine as long as you can apply the following instructions to them.

The Nikon CoolPix series, unlike the Nikon D-50, has a Panorama Assist mode. This mode allowed you to take the first shot, which would set up the shutter, aperture and white balance for the rest of the shots. Then the right third of the last image would appear on the left-hand side and be semi-transparent. This allowed you to align landmarks and get good anchors for your stiching. The fixing of the shooting settings was also nice because it meant that the seems wouldn't be visible if you use a good panorama tool (see my post on Hugin).

However, the Nikon D-50 is more professional and didn't include such a mode. No matter! After reading over the manual, which is available on the internet (it was hard to find at Nikon's site, www.nikonusa.com), it was easy to piece together the same process as the 'Panorama Assist' mode.

How to do it:

Get a tripod and make sure it has a level. Set up the tripod and check the level before and after mounting the camera. Loosen the yaw all the way, and tighten the pitch and roll. Without a Tripod, you'll find that your panorama will actually be askew, and picking a level rectangle for cropping will leave you with a lot of wasted image area.

If you're shooting a large up and down angle and have the standard 300mm lens, roll the camera and shoot portrait shots. This means if you don't rotate each shot, you might be stichting tops and bottoms of images in Hugin, which is counterinuitive, but you can rotate the anchor in Hugin later and everything will be fine.

Since we're post-processing the images, you should use 'Fine' mode. You don't have to use 'Raw', and in fact I noticed no real difference between 'Raw' and 'Fine' modes for this application.

The the 'M' mode on the mode dial means 'Manual Mode'. So, you just put the camera into this mode, and the shooting settings (shutter speed and aperture) are fixed.

Note that being in Manual Mode means the camera is not doing anything for you! You'll have to find some angle in your panorama to set the exposure settings until the exposure meter says zero. I favor the angle closest to the sun, but don't let the sun enter the frame. This will leave the dark portions of the panorama underexposed. The exposure meter is inside the viewfinder along the bottom. Rotate the winder on the right-hand side as you normally would to change the shutter speed and find the fastest speed that gets the reticle inside the meter. Then hold down the button on top, next to the power switch, shaped like an iris and rotate the same winder to adjust the aperture. Repeat the process until the reticle in the exposure meter is at zero.

On other thing you'll want to do is find a target at nominal distance to focus (or perform an auto focus) on. Then switch the camera into manual focus mode, since we don't want it to change. Don't accidentally change the focus.

Now you can start snapping away. Pick the most left angle and shoot, then change the yaw of the camera keeping a landmark that was on the right-hand side in the frame. Don't share too much of the yaw angle with the previous frame or you'll take too many pictures. Don't share too little or you won't have enough landmarks (common features) to teach the panorama tool how to stitch them together. You'll get the hang of it.

Finally, load the pictures onto your GNU+Linux workstation (or Windows, bleh) and import the series of shots into Hugin. http://hugin.sourceforge.net/

Hugin Deserves a Hug!

2007-07-14T12:35:00.000-05:00

Yes, I know other bloggers have already written in depth about Hugin, but I'd like to share my experience.

First of all, Hugin is just a graphical front-end for already established panorama libraries (which I didn't know existed) called panotools. Before, I would piece together a panorama by hand with The GIMP, but that never made the edges seamless, and never accounted for the lens barrel distortion.

The official page:
http://hugin.sourceforge.net/

They have plenty of tutorials to get you started. The program is not overcomplicated and you can get the hang of it rather quickly (make sure you are following the tutorials). It's just a matter of teaching the panorama tool how the images fit together and how the camera distorts the landscape.

The panorma shown was taken with a Nikon Coolpix 5400 in Panorama Assist mode some time ago. I hadn't been able to see the pictures stitched together because of the intensity of the barrel distortion, but with Hugin, it was a snap!

More process-specific posts on Hugin to come.

Announcing New Bliggity Blog

2007-07-14T12:32:00.000-05:00

Assuming that anyone might read this, I just want to make this post to announce that my blog will no longer be strictly GNU+Linux and computer tips. It will still be related to these topics, but posts on 'how to do this' either with a GUI or from the command line are getting too scarce. Note that I will start to use 'labels' (aka 'tags') so that you can still filter out all but a certain topic.

Pesky Windows "Thumbs.db" Files

2007-05-16T12:35:00.000-05:00

Windows creates a Thumbs.db file whenever it finds an image in a directory. If you are trying to reduce the number of files on your file system, or trying (really hard) to save space, you can issue the following commands as administrator.

locate -r ".*/Thumbs.db" > ~/delme-thumbs.txt

Which queries the mlocate database for the thumbnail files. If you're not running mlocate, you would use the find command here.

cat ~/delme-thumbs.txt | tr \\n \\0 | xargs -0 du -hsc

This will show you how much each file is taking up and include a sum of the total size. Finally, you might want to free these files with the following command:

cat delme-thumbs.txt | tr \\n \\0 | xargs -0 rm

To Beep or not to Beep

2007-03-22T09:28:00.000-05:00

Seems like Linux has done everything to prevent anyone from controlling the I/O devices under /dev, which includes you. I had a simple task, I wanted my "mail-notification" program to beep when I had mail. Under "When new mail arrives:", there was the command "play /path/to/some/sound.aiff" or something... I don't want any process interrupting my sound card because I'm running Fedora Linux, and I don't have JACK or another sound mixer which will mix sounds from multiple processes (maybe one day ALSA will work like this).

So anyway, I tried a million different commands...

echo -ne "\a"

which prints just a bell character, but it's interpretted by mail-notification (or nothing) and is lost--not sounded on the system beeper.

I tried a C program by Johnathan Nightingale, beep, which I had lying around, but it fails because the kernel doesn't like it touching the devices.

Finally, I thought "hmm, beep.c is just writing to /dev/console, why can't I?" and I tried

echo -ne "\a" > /dev/console

And it worked! It's not the same beep as Thunderbird/xterm/emacs, but it's fine with me!

Executing a command on multiple files with FIND

2007-03-18T14:45:00.000-05:00

Okay, so apparently, the FIND command is so powerful that you can even tell it what to do with the files when it finds them. I always used to find multiple files, translate newline characters to null characters, then pass each argument to XARGS as in:
find ./ -regex ".*\.mp3" | tr \\n \\0 | xargs -0 -n 1 md5sum > mp3checksums.md5

But, you can also say:
find ./ -regex ".*\.mp3" -exec md5sum {} \; > mp3checksums.md5

Where, the {} implies to insert an individual argument into that part of the exec string, and the semicolon (escaped so it's not picked up by BASH) ends the -exec agrument.

Computer Internal Dust Solutions

2007-03-13T07:40:00.000-05:00

It seems like opening up my computer (or multiple in my case) and blowing it out every month is releasing too much dust back into the air, and all those particles that were stuck in there can wreak havoc upon your moving parts (fans). Assuming you operate electronic devices which have ventilation fans indoors, try the following.

Get a HEPA filter and run it, at least on its lowest setting, all the time.
The electricity it takes to run such filters is completely eclipsed by how much dust they remove from the indoor air and prevent those particles from being stirred up again. It's better for human breathing, and better for devices with moving parts. Also, if the filter circulates the air, your components might run cooler.

Install low-impedance dust filters in front of your computer's ventilation intakes.
Many computer cases are already shipping with single-layer plastic mesh filters which will stop most huge dust particles (the ones you can see anyway). I decided to give it a shot and installed some extra dust filters on the intakes of two computers. One computer had side ports which were completely open (no, fan, just big holes). NOTE: If you have filters installed, air will follow the path of least impedance, so either block or put filters over all air passageways that flow into the case. Don't put filters on exhaust ports (ports which blow out).
I decided to use WEB Vent Filters, but these have a few drawbacks: they have fibers which come loose, and are higher impedance than the coarse foam. Coarse foam like that in SHOP*VAC Foam Sleeves is preferred for low impedance of air flow, but doesn't offer good filtering of smaller particles. Both are readily available at the local hardware store.

I put WEB Vent Filters on the outside of the case (so they wouldn't get sucked into the fan) for one, and in between the case and the air duct for the other. On the other computer, I took the front panel off, and installed the filters over the intake fans, but under the outer cover. The only concern I have now is cleaning these. I'm hoping that after the one-to-two month interval, I can vacuum the filters externally to clear large debris back out through the intake, and then the next interval I can remove them and wash them under water.

WARNING: Remember, the foam should be low impedance! Any dust filter you install over a ventilation port or fan will impede the flow of air through your device. Fans will spin a little faster, but air flow will be slower. Make sure this doesn't cause any heat issues, and either remove the filter or install more fans if necessary. Inevitably, over time the filters will become clogged and become further impeding, and will require replacement or washing.

Vacuum and dust often.
I know, this is common sense, but I've seen people who just have a million Duster cans and they blow out their computer whenever it starts to make noise. Grinding noises are a very bad sign--this means that something is on or around the fan's journal (which usually has ball bearings), and the fan will most likely fail if not cleaned. Oiling the fan's journals using three-in-one motor oil can help, but the best bet is to keep the fans free from particles beforehand, and replace them when things start to go awry. Rule of thumb: if you have to clean it out more than once every one to two months, you have a dust problem. Also, blowing the dust around can cause a fan assembly to become dirty.

Finally, keeping these computer systems away from sources of dust is a good idea. Major sources of dust include laundry, bedding, and animals and humans.

Shell Script to Parse Command Line Arguments

2007-02-04T11:11:00.000-05:00

Problem: I needed a script that took just file names and translated large JPEG's into smaller or thumbnail images. The problem was the convert command, part of ImageMagick, needs the specific output file name.

Solution: The following script.

#!/bin/bash

if [ ! -d Thumbs ] ; then
mkdir Thumbs;
fi

while [ $# -ge 1 ]; do
TARGET=`echo -n "$1" | sed "s/.jpg/t.png/" | sed "s/.JPG/T.PNG/" | sed "s/.jpeg/t.png/" | sed "s/.JPEG/T.PNG/"`;
if [ -e $TARGET ] ; then
echo "Refusing to overwrite $TARGET. ";
else
echo -n "Converting $1 to Thumbs/$TARGET: ";
convert $1 -resize 128x128 Thumbs/$TARGET;

if [ $? -eq 0 ]
then
echo "OK."
else
echo "ERROR: $?"
fi

fi

shift

done
exit 0

Bragging Rights

2006-12-27T10:46:00.000-05:00

I put a new computer together, so I could have better performance than my previous rig. I wanted a powerful but quiet workstation. I just didn't feel like installing Windows--I've never felt so dissuaded from installing Windows. It just seemed like an unnecessary hassle. I just wanted to have just Fedora Linux on the machine.

In sort of an aimless manner, I ended up having a software RAID partition with an 899GB capacity. I don't know if I'll keep this configuration, but it's very fast and I can't imagine running out of space. I just hope it's not drawing too much power. I was pretty floored by the following responses from Linux:

/proc/cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz
stepping : 6
cpu MHz : 1596.000
cache size : 4096 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr lahf_lm
bogomips : 4811.97

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz
stepping : 6
cpu MHz : 1596.000
cache size : 4096 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr lahf_lm
bogomips : 4805.23

Now, my Pentium 4 3.2GHz processor gave me 6,400 bogomips for each CPU, but we all know that only approximately 6,400 operations per second can occur. In other words, the second CPU is virtual and only gets work done when the 31-stage Pentium 4 pipeline would otherwise be empty. These two 4,800 bogomips readings can occur simultaneously, allowing approximately 9,600 bogomips for the whole system.

/proc/meminfo:
MemTotal: 2074924 kB
MemFree: 1084092 kB
Buffers: 32624 kB
Cached: 673980 kB
SwapCached: 0 kB
Active: 481828 kB
Inactive: 454244 kB
HighTotal: 1179264 kB
HighFree: 264560 kB
LowTotal: 895660 kB
LowFree: 819532 kB
SwapTotal: 4095992 kB
SwapFree: 4095992 kB
Dirty: 42252 kB
Writeback: 0 kB
AnonPages: 229628 kB
Mapped: 55352 kB
Slab: 35120 kB
PageTables: 4552 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 5133452 kB
Committed_AS: 602780 kB
VmallocTotal: 114680 kB
VmallocUsed: 7012 kB
VmallocChunk: 107160 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 4096 kB
Filesystem Size Used Avail Use% Mounted on
/dev/md2 899G 3.2G 849G 1% /
/dev/md0 99M 10M 84M 11% /boot
tmpfs 1014M 0 1014M 0% /dev/shm
192.168.0.5:/home 360G 249G 94G 73% /home
/dev/hde 3.3G 3.3G 0 100% /media/FC_6 i386 DVD

There are two 500GB SATA 3.0GB/s Seagates!

[root@localhost ryan]# /sbin/hdparm -t /dev/md2

/dev/md2:
Timing buffered disk reads: 426 MB in 3.01 seconds = 141.61 MB/sec
[root@localhost ryan]# /sbin/hdparm -t /dev/md2 ; /sbin/hdparm -t
/dev/md2; /sbin/hdparm -t /dev/md2

/dev/md2:
Timing buffered disk reads: 434 MB in 3.01 seconds = 144.34 MB/sec

/dev/md2:
Timing buffered disk reads: 434 MB in 3.00 seconds = 144.60 MB/sec

/dev/md2:
Timing buffered disk reads: 434 MB in 3.01 seconds = 144.26 MB/sec
[root@localhost ryan]# /sbin/hdparm -T /dev/md2 ; /sbin/hdparm -T
/dev/md2; /sbin/hdparm -T /dev/md2

/dev/md2:
Timing cached reads: 5120 MB in 2.00 seconds = 2565.42 MB/sec

/dev/md2:
Timing cached reads: 5152 MB in 2.00 seconds = 2580.52 MB/sec

/dev/md2:
Timing cached reads: 5060 MB in 2.00 seconds = 2533.59 MB/sec

Nightly Backup Script

2006-12-12T13:28:00.000-05:00

If you're working on a small project, you might want to do a simple nightly backup. The following Bash Shell Script, when copied into your /etc/cron.daily/ directory will do the trick. You might have to allow read access for 'other' users for everything you want to back up. It worked without changing that for me.

Obviously, you'll have to change the file and directories. This is just an example.

[ryan@a564n cmpe646]$ cat cmpe646_backup.sh
#!/bin/sh

date=$(date +%s)

tar -cf /home/cmpe646/cmpe646_project_backup_$date.tar /home/cmpe646/*/*.h* /home/cmpe646/*/*.c* /home/cmpe646/*/*.bench*
bzip2 /home/cmpe646/cmpe646_project_backup_$date.tar

exit 0

Blowing out Fans

2006-12-12T10:05:00.000-05:00

I took some highly compressed air and blew out a small processor fan (VIA EPIA) because it seemed to be making noise. It started to make more noise after I blew it out. So at first, I thought maybe I was using too much air pressure, but then I remembered that tiny fans usually make the most noise when compared to much larger fans. Later, I went and noticed that the fan was spinning a lot faster after I blew it out. Something I didn't expect to see. Then, I noticed a downward trend over the last half of a year since the fan was put into commission.

What I get from these pictures is that the fan is now spinning faster like it was half a year ago and the processor also started running cooler after I blew out the fan.

The conclusion: The failure mode of computer case fans is to develop (physical and electrical) resistance and spin slower and slower. They should be kept clean to prolong the life of the fan and the components they cool, but don't use too much pressure--you don't want to break the assembly.

If you're using compressed air -- you can use about 80-100 PSI, but don't use anything bigger than a Sports Ball Inflator or you'll have too much air velocity. Also, if you don't have a filter or anything on your compressor, you have to be careful about debris! Don't hold the air canister upside down or anything that might stir up dust or debris inside the tank. For most people, it's probably best to go to the store and get a can of "air" for your exact purpose.

Comparison of 20.1" LCD Monitors

2006-11-19T19:52:00.000-05:00

I wanted to know which display would be better for desktop computing: a normal 1600x1200 pixel display or a widescreen 1680x1050 pixel (HDTV) display. The widescreen, would be more efficient for movies. I remember doing this once before, so I decided to write it out this time. To simplify my analysis, I chose to consider only 20.1" diagonal viewing areas, with the aforementioned resolutions.

For the widescreen, I calculate the display is 17" by 10.6", giving a viewing area of 181.6 square inches. For the 4:3 ratio display, I got 16" by 12" inches, and a viewing area of 193.9 square inches. This is a 6.7% bigger display area for the 4:3 ratio display. More generally, if the diagonal length is fixed, a square maximizes the area of a rectangle.

To compare the resolutions, the widescreen gives 1.76 megapixels, and the 4:3 ratio gives 1.92 megapixels. This is a 9.1% increase in pixel count for the 4:3 ratio display.

In conclusion, you win with the 1600x1200 20.1" non-widescreen display in terms of specifications, but they are generally more expensive (partly due to TFT manufacturing realities) but more compatible with video games (until recently). The wide screen, however, might be a more natural design because of our rectangular viewing perspectives.

Fedora Core 5 + IMAP Server + SquirrelMail

2006-11-03T09:33:00.000-05:00

The next best thing, because sharing the Mozilla Thunderbird Storage folder is giving me problems when I change networks, is to have an IMAP server that Thunderbird can store/retrieve messages with.

The mail from the Internet accounts could be copied to the IMAP server using Thunderbird, even if a delivery system (POP/SMTP) is not in place (for mail to get there otherwise). I installed SquirrelMail and tested this capability with Thunderbird already. I closed down sendmail, installed and started postfix. I set up the Cyrus IMAP, which was the hardest part of the IMAP/SquirrelMail setup.

The tutorials I found to help do this on Fedora Core 5 are:

http://cyrusimap.web.cmu.edu/imapd/install.html
http://nakedape.cc/info/Cyrus-IMAP-HOWTO/quickstart-fedora.html
http://www.howtoforge.com/perfect_setup_fedora_core_5

The last one shows how to set up SMTP, as does the one from Cyrus itself. I'm still trying to figure out if I can have my own SMTP and use my IP Address (or its alias) as a domain for sending mail to.

Even if I can't, I think that having the IMAP server in place, just for storing mail, would be a big help as the mail would be accessible from anywhere (even using SquirrelMail).

PHP Gas Mileage Application

2006-11-02T18:40:00.000-05:00

Here's a little something I wrote out of necessity and decided not to stop before it was completed. I'm pretty proud of it and am surprised how easy PHP was to learn. It basically helps me keep track of my gas mileage over the web so I can throw the receipt out faster, and don't have to carry a spreadsheet around. The following is from the program itself:

This program is a gas mileage database and analyzer written in PHP. It is self-contained (has a single source code file). It stores data in a simple text file in HTTP-query format. It allows users to add new records, and add new data files for different vehicles. It creates a backup before a new record is added. It also implements password protection before users can add refueling records.

Source Code:
Source Code
See it in action

Simple Web File Upload Form

2006-10-28T20:45:00.001-05:00

I put the following form and post processing files together from a manual on PHP. The HTML could really be embedded in the PHP file, but this might be better if you only want one copy of the PHP code on the system. Modify both to fit your environment and make sure the PHP file is executable.

Example .HEADER.html

<br /><h1>Index of /srv/pub</h1>
<br />
<br /><a href="/">Back to Welcome Page</a><br>
<br /><br>
<br />This form allows you to upload a file of up to 10 MiB:
<br />
<br /><!-- The data encoding type, enctype, MUST be specified as below -->
<br /><form enctype="multipart/form-data" action="/cgi-bin/getfile.php" 
<br />method="POST">
<br />    <input type="hidden" name="dest_dir" value="/srv/pub/" />
<br />    <!-- MAX_FILE_SIZE must precede the file input field -->
<br />    <input type="hidden" name="MAX_FILE_SIZE" value="1000000000" />
<br />    <!-- Name of input element determines name in $_FILES array -->
<br />    Send this file: <input name="userfile" type="file" />
<br />    <input type="submit" value="Send File" />
<br /></form>
<br />
<br />

Example getfile.php

<br /><html>
<br /><head>
<br /><title>File Upload</title>
<br /></head>
<br /><body>
<br />
<br /><h1>File Upload</h1>
<br />
<br /><?php
<br />// In PHP versions earlier than 4.1.0, $HTTP_POST_FILES should be used instead
<br />// of $_FILES.
<br />
<br />//$uploaddir = '/var/www/srv/Public/';
<br />$uploaddir = $_POST['dest_dir'];
<br />$uploadfile = $uploaddir . basename($_FILES['userfile']['name']);
<br />
<br />echo '<pre>';
<br />echo "Trying to save to: ".$uploadfile."\n";
<br />
<br />if (move_uploaded_file($_FILES['userfile']['tmp_name'], $uploadfile)) {
<br />   chmod($uploadfile, 0644);
<br />   echo "File is valid, and was successfully uploaded.\n\n";
<br />   echo "Saved in: ".$uploaddir."\n";
<br />   echo "Note that this program sets the file permissions to 0644.\n";
<br />
<br />   echo "<a href=\"javascript:history.go(-1)\">Click Here to Continue.</a>\n";
<br />
<br />} else {
<br />   echo "File upload failed.\n";
<br />     switch ($_FILES['uploadFile']['error'])
<br />     { case 1:
<br />         print '<p> The file is bigger than this PHP installation allows</p>';
<br />         break;
<br />       case 2:
<br />         print '<p> The file is bigger than this form allows</p>';
<br />         break;
<br />       case 3:
<br />         print '<p> Only part of the file was uploaded</p>';
<br />         break;
<br />       case 4:
<br />         print '<p> No file was uploaded</p>';
<br />         break;
<br />     }
<br />}
<br />
<br />echo "<br><br>Here is some more debugging info:\n<br>\n";
<br />print_r($_FILES);
<br />
<br />print "</pre>";
<br />
<br />?>
<br />

This code is derived from a few examples.

Web File Search CGI Program for Linux

2006-10-28T20:23:00.000-05:00

So you can install Fedora, and choose the Web Server package. If you are running any kind of file server, you will probably want to find a file over the web. This script allows users to search the mlocate database which is usually compiled by your Linux installation every night. It's a fast database which circumvents searching the filesystem manually. I had to write it, so I extend my source code to everyone in hopes that it will help someone.

It has some thought put into security. The results are displayed as links to facilitate viewing/downloading the located file. It should be used to search one part of the file system. In my case, the public area is /srv/. This part of the filesystem should be linked into the Apache document root via

ln -s /srv /var/www/html/srv

to allow the users access to the files with your HTTP server.

Following is the Perl-CGI code. Name it locate.cgi and it should go in your cgi-bin directory. Don't forget to chmod 755 the file so it's executable. Because of technical limitations of Blogger, I can't paste the Perl-CGI code here--please follow the following link:

http://www.cs.umbc.edu/~rhelins1/getfile.php?id=26

Using the Same Thunderbird Storage Folder on Windows and Linux

2006-10-26T13:01:00.000-05:00

Today I finally solved the problem of having multiple e-mail clients and multiple storage folders. First I used an IMAP account I have access to in order to move all the old e-mail I have on multiple computers onto the Thunderbird "Local Folders" in Linux on my server. This is as easy as copying all the messages in one Inbox to the Inbox on the IMAP account, and then on the other computer moving the messages out of the Inbox on the IMAP account to the local Inbox account. It's not the fastest, but it's the most compatible as Thunderbird doesn't support much importing from other e-mail storage folders.

So once I had everything on the "Local Folders" account in Thunderbird on Linux, I changed the Local Message Storage directory on Windows (not on the Server) to the storage directory in my home folder on my Linux server via a drive-mapped Samba share:
Y:\.thunderbird\ayrxordj.default\Mail\Local Folders
This can be found by going to Tools->Account Settings->Local Folders. Click Browse. You'll have to have "Show Hidden Files" ON to locate your Local Folders path with the GUI. You also need to make sure that all your e-mail accounts you have are set up to store the mail in the "Local Folders" rather than an individual folder for that account.
After some testing with no networking on the Windows computer, if the client doesn't have accesss to the network drive, you just get nothing under "Local Folders". In other words, it doesn't crash or anything.

So what does this mean? If I download a message on my Windows Thunderbird client, it is exactly like downloading it on the Linux Thunderbird client. It is the same for the Linux client (vice-versa). It also means that now I can have "Delete message from POP after download" set to "Yes" on both clients and avoid downloading e-mail that I've already seen.

Issues I can imagine running into include having both clients open simultaneously and possible Operating System dissimilarities in the implementation of Thunderbird. Also, it took a long time to open the gigantic storage folder the first time over the network, but cleaning up the 2-3 thousand messages and perhaps "compacting" the folders should help. The other issue is, since I merged the two clients, there are several duplicate e-mails, but there is an extension to find & delete duplicate messages which has already solved this problem for me. I removed all the duplicates (leaving me with only roughly two thirds of the messages, because this merge was so large) and compacting the folders did solve the long-to-load problem.

Now I can load up Thunderbird on Windows or Linux using the same "Local Folders" storage directory. Finally, no more getting on different computers/clients to find different mail.

There are still issues with using this technique. If you have a laptop or are losing connectivity, Windows will let you know that "Delayed Writing" failed. Also, a solution like having IMAP storage would be much more network-friendly in terms of bandwidth. See my post on setting up Cyrus IMAPD

Shell Script for Tunneling VNC over SSH

2006-10-24T11:54:00.000-05:00

If you use VNC over the Internet rather than just over your LAN, it is not recommended to allow the 2590x port to be forwarded to the Internet. Instead, if you already have port 22 for SSH forwarded to the machine you want to get VNC from, you're all set to connect securely via an SSH tunnel. Note that you still have to have VNC set up and working on the LAN and know which screen number you are using (if you've used the GUI to allow your desktop to be connected to remotely, then you are using screen 1). This can also be accomplished using PuTTy with its GUI.

Replace {user} with your username on the remote machine, and the {WAN Address} with your public IP address on the remote machine. Replace the screen with the (single) number of the screen you use (1,2,3,...). Due to technical limitations, I can't use backslashes, so please make the command a one-liner. The ssh with the sleep argument needs to have a command after it that uses the forwarded port or it will close immediately.

#!/bin/sh

ssh -f -L 2590{screen}:127.0.0.1:590{screen} 
 {user}@{WAN Address} sleep 10; 
 vncviewer 127.0.0.1:2590{screen}:{screen};

exit

So an example script would look like:

#!/bin/sh

ssh -f -L 25901:127.0.0.1:5901 user@mysubnet.domain.com sleep 10; 
 vncviewer 127.0.0.1:25901:1;

exit

Shell Scripts for making links unique or sole copies

2006-10-24T11:41:00.000-05:00

Here are scripts to convert soft links to unique copies, and to sole copied (which moves the target of the link to the location of the link).

#!/bin/sh

# Check that link exists
ls "$1" > /dev/null
if test $? -eq 0
then
echo -n "File found. "
else
echo "Error: File not found."
exit 1
fi

linktarget=`find "$1" -printf "%l\0"`
linktargettype=`stat --format=%F "$linktarget"`

# Check that link target exists
if test -z "$linktarget"
then
  echo "Error: Null link target, not a valid soft link."
  exit 1
else
  echo -n "Found soft link. "
fi

# Check that link target is NOT a directory
#if test "$linktargettype" = "directory"
#  then
#    echo "Link to directory, skipping."
#    exit 0
#  else
#    echo "Not a directory."
#fi

# Remove soft link
echo -n "Unlinking $1... "
unlink "$1"
if test $? -eq 0
then
  echo "Done."
else
  echo "Error!"
  exit 1
fi

# Replace soft link with unique copy
echo "Creating copy from $1"
echo -n " to $linktarget... "
cp -a "$linktarget" "$1"
if test $? -eq 0
then
  echo "Done."
else
  echo "Error!"
  exit 1
fi


exit 0

Here's the same script for changing the link to the unique copy:

# Check that link exists
ls "$1" > /dev/null
if test $? -eq 0
then
echo -n "File found. "
else
echo "Error: File not found."
exit 1
fi

linktarget=`find "$1" -printf "%l\0"`
linktargettype=`stat --format=%F "$linktarget"`

# Check that link target exists
if test -z "$linktarget"
 then
   echo "Error: Null link target, not a valid soft link."
   exit 1
 else
   echo -n "Found soft link. "
fi

# Check that link target is NOT a directory
#if test "$linktargettype" = "directory"
#  then
#    echo "Link to directory, skipping."
#    exit 0
#  else
#    echo "Not a directory."
#fi

# Remove soft link
echo -n "Unlinking $1... "
unlink "$1"
if test $? -eq 0
 then
   echo "Done."
 else
   echo "Error!"
   exit 1
fi

# Replace soft link with only copy
echo "Moving $1"
echo -n " to $linktarget... "
mv "$linktarget" "$1"
if test $? -eq 0
 then
   echo "Done."
 else
   echo "Error!"
   exit 1
fi


exit 0

Convert Soft Links to Hard Links

2006-10-24T11:38:00.001-05:00

Here's a Shell script I wrote to convert large numbers of soft links to hard links on Linux.
It now handles soft links on different file systems properly.

#!/bin/sh
# soft2hard.sh by Ryan Helinski
# Replace a soft (symbolic) link with a hard one.
#
# $1 is name of soft link
# Returns 0 on success, 1 otherwise
#
# Example: To replace all the soft links in a particular directory:
# find ./ -type l | tr \\n \\0 | xargs -0 -n 1 soft2hard.sh
#
# finds all files under ./ of type link (l), replaces (tr) the newline
# characters with null characters and then pipes each filename one-by-one
# to soft2hard.sh

# Check that link exists
ls "$1" > /dev/null
if test $? -eq 0
then
echo -n "File found. "
else
echo "Error: File not found."
exit 1
fi

linktarget=`find "$1" -printf "%l\0"`
linktargettype=`stat --format=%F "$linktarget"`

# Check that link target exists
if test -z "$linktarget"
 then
   echo "Error: Null link target, not a soft link."
   exit 1
 else
   echo -n "Found soft link. "
fi


# Check that link target is NOT a directory
if test "$linktargettype" = "directory"
 then
   echo "Link to directory, skipping."
   exit 0
 else
   echo "Not a directory."
fi

# Remove soft link
echo -n "Unlinking $1... "
unlink "$1"
if test $? -eq 0
 then
   echo "Done."
 else
   echo "Error!"
   exit 1
fi

# Replace with hard link
echo "Creating hard link from $1"
echo -n " to $linktarget... "
ln "$linktarget" "$1"
if test $? -eq 0
 then
   echo "Done."
 else
   echo "Error creating hard link, replacing soft link"
   ln -s "$linktarget" "$1"
   exit 1
fi


exit 0

GRUB Read Error

2006-10-10T09:20:00.000-05:00

If you've ever gotten the following error:

Loading GRUB... Read error

It looks like the GRUB read error occurs not because I shut the system down abrubtly or modified the GRUB configuration incorrectly, but because I'm trying to boot the system up without a keyboard. So, for now, I'll just keep the keyboard and mouse plugged in, but I'd like to be able to boot without these eventually. Some settings in the BIOS (like enabling USB keyboard support) might let me get around this error.

Optimizing Multimedia and Backup Storage with Hard or Soft Links

2006-10-10T08:04:00.000-05:00

Here are the notes on this subject in my Server Log with some annotations. Basically, it's (hard links are) a quick fix for duplicate files to free up space.

I had md5 files generated for everything that was copied onto the server, so using the shell commands sort, grep, and uniq, I was able to clean up a lot of space from files that had been copied over twice as a result of using WinMerge to prepare for deleting the backups that were on one of the hard drives that went into the RAID.

I was about to start playing with FSlint, but by chance came across a perl program called dupseek. I was looking for a script which replaced one of two duplicate files with a link.

As far as I can tell, this is an excellent program with a well thought-out algorithm for finding duplicate files on a Unix system (based on personal experience, and what it says on their page), but more importantly to me, it has a function for creating Unix soft links in place of the duplicate files. Although it's text-mode, this is the best program I've used for dealing with duplicate files. And text-mode is just fine! This is a real life-saver because I don't want duplicates sitting on the file system, and I'd like to keep some files cross-referenced in directories. Also, removing duplicates in directories which are already backed up will make the copy on hard disk seem like it has fewer files than that on the CD.

Hard links would really be better (for me), because soft (symbolic) links in Unix probably raise some compatibility issues when, for instance, trying to but the directory on a CD. Unless interpreted correctly, these links are just files. With hard links, however, the same file system inode is just referenced by two different directories. Since the directories in question won't be changing, this wouldn't raise any issues (deletion, separation, etc.).

In fact, gnomebaker currently has a bug where soft links are dereferenced for computer the size of the CD image, but the link file itself if put onto the CD.

I was able to save less than 10 GB by realizing there were duplicate media files as a result of combining directories before the big move to the server RAID, and more than 2 GB by using dupseek.

http://www.pixelbeat.org/fslint/
http://www.beautylabs.net/software/dupseek.html

Notes:

You can use a text file, generated yourself or by filtering the report generated the “-b report” function of dupseek, which contains filenames to be removed. You can pipe these names to xargs which calls rm. This is useful in case files have been copied to more specific directories and many duplicates lie in a general directory (e.g. “downloads” vs. “singles”).

cat [name of report file] | grep “/Downloads/” | tr \\n \\0 | xargs -0 rm

Which accomplishes the task of filtering the report to only files which are in a path which includes
“/Downloads/”, replacing the new line characters with null characters, and xargs passes each of these lines to rm. This removes the duplicate files which are in the common directory (which you want to clean out, and not preserve). Also, this must be executed from the same directory the report file was created in reference to (to make the relative filenames match up). For safety, try replacing rm with ls before you do anything to make sure you're about to remove the right files:

cat [name of report file] | tr \\n \\0 | xargs -0 ls | less

Only after checking over this output, hit q and then run:

cat [name of report file] | tr \\n \\0 | xargs -0 rm

Note: A much safer plan is to use the interactive mode of dupseek, or use the FSlint GUI (takes into account hard links). The interactive mode of dupseek got too repetitive and so I just used it to identify duplicates in one case, but in general the interactive or batch mode is fine (be careful of the batch mode, unless you're running my version of dupseek).

In the end, dupseek is better for batch jobs (but I feel that only with my hard link modification) and FSLint is better for compatibility and running on directories like home folders where it is the case that you want to leave some unique files alone. Of course, a compressed file system would be a step better, but who has the (CPU) time for that?

Making directory traversals more efficient

2006-09-23T11:00:00.000-05:00

I wrote a script which takes less than a second to run, and which splits bulky folders up into 28 different subfolders folders. If there's a well-organized folder with more than 30-50 subfolders under it, then they're all the same type of subdirectory and they only differ in their name. More than thirty or fifty subfolders is too much for your users to digest at once, and it slows down traversing the file system. It takes longer for Linux to get all the files necessary to list all the subfolders, and if you're browsing with Apache, it might take a while to download the whole listing.

A very popular solution for this type of directory is to subdivide the collection of similar subfolders by their name. I wrote a script which does this based on the first character of the folder name. This way, you can divide a huge directory into chunks that make traversals more efficient for the computer and more responsive to your users (even over CIFS, FTP, NFS, etc.).

Following is the script code:

#!/bin/sh

mkdir /tmp/myindex

mkdir /tmp/myindex/0-9
mkdir /tmp/myindex/misc
mkdir /tmp/myindex/a
mkdir /tmp/myindex/b
mkdir /tmp/myindex/c
mkdir /tmp/myindex/d
mkdir /tmp/myindex/e
mkdir /tmp/myindex/f
mkdir /tmp/myindex/g
mkdir /tmp/myindex/h
mkdir /tmp/myindex/i
mkdir /tmp/myindex/j
mkdir /tmp/myindex/k
mkdir /tmp/myindex/l
mkdir /tmp/myindex/m
mkdir /tmp/myindex/n
mkdir /tmp/myindex/o
mkdir /tmp/myindex/p
mkdir /tmp/myindex/q
mkdir /tmp/myindex/r
mkdir /tmp/myindex/s
mkdir /tmp/myindex/t
mkdir /tmp/myindex/u
mkdir /tmp/myindex/v
mkdir /tmp/myindex/w
mkdir /tmp/myindex/x
mkdir /tmp/myindex/y
mkdir /tmp/myindex/z

mv ./[0-9]* /tmp/myindex/0-9/
mv ./[!0-9a-zA-Z]* /tmp/myindex/misc/
mv ./[Aa]* /tmp/myindex/a/
mv ./[Bb]* /tmp/myindex/b/
mv ./[Cc]* /tmp/myindex/c/
mv ./[Dd]* /tmp/myindex/d/
mv ./[Ee]* /tmp/myindex/e/
mv ./[Ff]* /tmp/myindex/f/
mv ./[Gg]* /tmp/myindex/g/
mv ./[Hh]* /tmp/myindex/h/
mv ./[Ii]* /tmp/myindex/i/
mv ./[Jj]* /tmp/myindex/j/
mv ./[Kk]* /tmp/myindex/k/
mv ./[Ll]* /tmp/myindex/l/
mv ./[Mm]* /tmp/myindex/m/
mv ./[Nn]* /tmp/myindex/n/
mv ./[Oo]* /tmp/myindex/o/
mv ./[Pp]* /tmp/myindex/p/
mv ./[Qq]* /tmp/myindex/q/
mv ./[Rr]* /tmp/myindex/r/
mv ./[Ss]* /tmp/myindex/s/
mv ./[Tt]* /tmp/myindex/t/
mv ./[Uu]* /tmp/myindex/u/
mv ./[Vv]* /tmp/myindex/v/
mv ./[Ww]* /tmp/myindex/w/
mv ./[Xx]* /tmp/myindex/x/
mv ./[Yy]* /tmp/myindex/y/
mv ./[Zz]* /tmp/myindex/z/

mv /tmp/myindex/* ./
rmdir /tmp/myindex

exit 0

Some 'for' loops would make this code substantially smaller, and if someone makes a suggestion I'll try it.

Compiling the vt1211.ko Kernel Module

2006-09-23T10:47:00.000-05:00

I basically followed the directions from VIA, except I used the latest patch file for kernel 2.6.17 available at Lars Ekman's website: http://hem.bredband.net/ekmlar/vt1211.html instead of the one VIA provides in their VIA FC5 Hardware Monitor Application Notes (www.viaarena.com). The patch file is designed to patch the whole kernel tree (only a small part, but you need the whole thing). If you have 2.6.17-1.2174_FC5, the module I compiled is already available on Lars Ekman's website.

Below is a copy & paste of the instructions from VIA, available on VIA Arena, with annotations made where I remember their instructions didn't work. Fill in the mentioned kernel name with your kernel. If I ever have to do this again, I'll make my own instructions.

I got the corresponding kernel source package from YUM instead of downloading the RPM. The package is called kernel-devel. This puts files in /usr/src/redhat. Note that when you're done, you can use YUM to again remove this package. I had to do a few things to get rpmbuild installed and working (I remember it needed a few software packages from YUM), but then I could make it through the first few lines.

#rpm –ivh kernel-2.6.15-1.2054_FC5.src.rpm
#cd /usr/src/redhat/SPECS
#rpmbuild –bp –-target=i686 kernel-2.6.spec
#cd /usr/src/redhat/BUILD

Then they want you to move the kernel source tree to /usr/src/{name of kernel} and then apply the patch.

You can find the kernel source directory in path
/usr/src/redhat/BUILD/kernel-2.6.15/linux-2.6.15.i686. Move the
kernel source directory to path /usr/src/kernels and patch
the os default kernel.
#cd /usr/src/redhat/BUILD/kernel-2.6.15
#mv linux-2.6.15.i686 /usr/src/linux-2.6.15-1.2054_FC5
#cp vt1211_FC5.patch /usr/src
#cd /usr/src
#patch –p0

Now you have the new kernel tree, and you need to compile the kernel with the vt1211 source as a module to get the vt1211.ko.

Select needed item and rebuild the patched kernel
Edit the Makefile in directory linux-2.6.15-1.2054_FC5.Find the string
“EXTRAVERSION = -prep” and modify it to “EXTRAVERSION = -1.2054_FC5”.
#cd linux-2.6.15-1.2054_FC5
#cp /boot/config-2.6.15-1.2054_FC5 .config
#make menuconfig
Device Drivers ---> Hardware Monitoring support --->
[M] Hardware Monitoring support
[M] VT1211
After set the kernel item completely and save it, we can rebuild the kernel
source. When the compiling module procedure is completed, you can find
the “vt1211.ko” module to path
/lib/modules/2.6.15-1.2054_FC5/kernel/drivers/hwmon
#make
#cp linux-2.6.15-1.2054_FC5/drivers/hwmon/vt1211.ko
/lib/modules/2.6.15-1.2054_FC5/kernel/drivers/hwmon
#depmod -a

The GUI menuconfig didn't work for me. If you run make config, you can do the same thing in text mode. If you just run make, I think it will ask you if you want Kernel/Module whenever not specified by the .config file you copied from your boot folder. Make sure you select Module (type the letter 'm') for the vt1211 objects. Then, about six hours later, it'll finally make it through all the source and you'll get your .ko file. Copy it out of there and into your /lib/modules tree and then run depmod as in the instructions above.

When you're done, you can go back to the /usr/src/{linux kernel name} and run make clean, or you can remove that whole directory if you no longer need it. You can also remove the kernel-devel package, rpmbuild, etc.

Update

Linux kernel 2.6.21 now includes an even better version of the vt1211 module! This kernel is the default on RedHat Fedora 7 Linux. Finally time to upgrade my server installation.

Create md5 sum files in Linux

2006-09-18T09:06:00.000-05:00

Learned that the Linux shell can be quickly used to generate md5 checksums of an entire directory to an md5 file, the command is of the form:

cd [directory to generate check sums for]
find ./ -printf “%p\0” | xargs -0 -n 1 md5sum > ./checksums.md5

Then, to check the files in a new location (e.g. on a CD-ROM), use the command:

md5sum -c ./checksums.md5

Or there are other programs for validating the files with the checksums, like wxChecksums which is available for Windows.

I wanted this because the graphical wxChecksums was too difficult to compile, and didn't seem worth the effort. Although I'm sure a couple backwards-compatibility packages for gcc would fix this problem, the command-line method makes it straightforward for doing multiple check sums in a single command in text mode.

Python BitTorrent Daemon

2006-09-17T13:24:00.000-05:00

If you're running a server and leeching/seeding torrents, and it's a no-screen machine, you might not want to use a BitTorrent client with an interface. The main reason is, the interface requires a lot of CPU cycles to constantly update and if you're not even looking, why bother? Azureus is the most powerful BitTorrent client, but it's also the most CPU-intensive.

On my Mini-ITX server, Azureus running on a VNC Server (xvnc) left the kernel 60% or less idle at times when the GUI was open or when traffic was heavy (30-40% cycles given to Azureus), and Azureus is also known for hogging memory (weakness of Java). After I switched to just letting btseed run, the total CPU usage is about 8% (92% idle). This means that the kernel is more responsive to new tasks.

A Python BitTorrent service (Daemon) either comes with Fedora or is part of the packages installed as dependencies of the YUM bittorrent-gui package. There is a Daemon set up called btseed, which is capable of monitoring .torrent files in a specified directory and downloading them automatically. There's essentially no documentation I can find about how to use this correctly, but I got it working OK.

The /etc/init.d/btseed script should be used to start/stop/restart the Daemon. This script launches btseed, which is an incarnation of the launchmany-console Python program. The file /etc/sysconfig/bittorrent should be configured to your preferences.

The Daemon is capable of UPNP, which makes it pretty nice. Here is my /etc/sysconfig/bittorrent configuration file:

SEEDDIR=/srv/bittorrent/data
SEEDOPTS="--max_upload_rate 50 --display_interval 30 --minport 49900 --maxport 52000"
SEEDLOG=/var/log/bittorrent/btseed.log
TRACKPORT=6969
TRACKDIR=/srv/bittorrent/data
TRACKSTATEFILE=/srv/bittorrent/state/bttrack
TRACKLOG=/var/log/bittorrent/bttrack.log
TRACKOPTS="--min_time_between_log_flushes 4.0 --show_names 1 --hupmonitor 1"

Note that the TRACK* settings are not important unless you're running a tracker.

Get a free domain name (IP alias) for your subnet

2006-09-17T12:47:00.000-05:00

DynDNS.com provides a free IP Address Alias for everyone. The only problem is, I have a dynamic IP address, so I had to also get ddclient (for Linux) working. It was pretty simple—a package was available on YUM, and it installs itself by default as a daemon (service). All you need to do is set up the /etc/ddclient.conf file to work correctly. You need to do two things with the ddclient: discover your IP address, and check with DynDNS. I basically just had to uncomment some lines and fill in the details. DynDNS has a facility for giving you your WAN IP address. Your router does too, but it may be too hard to find.