This blog allows me to disseminate knowledge I think should be readily available. If I find myself ranting, I try to make sure it's informative, and not completely subjective. I try to offer solutions--not problems--but in the case I do present a problem, please feel free to post a comment.
Monday, December 08, 2008
Nikon D-50 Cleaning Dust from Sensor
I knew that the low-pass filter was a fancy term for an IR filter that happens to provide an air-tight seal for the CCD sensor. There's a procedure in the manual that enables you to lock open the reflex mirror and the shutter curtain, exposing the low-pass filter and sensor. I highly recommend that you read these instructions; they're very easy to follow. They recommend not to contact the low-pass filter, and I totally agree.
I followed the instructions for locking the mirror and curtain open, and when I removed the body cap, I could clearly see the debris on the sensor. I simply blew some air in there, and it was freed. I recommend blowing air while holding the body upside-down. Make sure you have a lot of light.
Nikon also has directions for setting up test shots for seeing if you have a dust problem. It is geared for their dust removal software, but it also serves as a diagnostic.
You turn the camera off to exit the special lock mode. Presto! No more dirt on my photos!
Sunday, April 27, 2008
MD5SUM reorganization revisited
I was running my shell script on a collection of over 10,000 files, and it was running for over three days. The shell interpreter is just way too slow when this amount of work is involved. So, I re-wrote the script in Perl and it is now lightning fast because I am able to take advantage of Perl hashes (associative arrays).
#!/usr/bin/perl
#
# This file takes a checksum file (output from md5sum utility)
# and attempts to reorganize the files in the directory to
# match the listing in the md5 file.
#
# Files not found in the md5 input are left alone.
# Files already in the right place are left alone.
# Other files have their checksums computed and, if they are found
# in the md5 input, they are moved to the appropriate location.
#
# WARNING: It confuses duplicate files!!!
#
use File::Basename;
if ( $#ARGV < reorgpath =" $ARGV[1];" reorgptah = "." lines =" ;
close(SUMSFILE);
foreach my $line (@lines) {
chomp($line);
$sums{substr($line,0,32)} = substr($line,34);
}
print "Read in ".($#lines+1)." checksums and paths.\n";
&reorg($reorgPath);
sub reorg {
my $dir = shift;
#print "Recurring for $dir\n";
opendir DIR, $dir or return;
my @contents =
map "$dir/$_",
sort grep !/^\.\.?$/,
readdir DIR;
closedir DIR;
foreach my $file (@contents) {
#print "Considering $file\n";
#next unless !-1 && -d;
if ( -d $file ) {
&reorg($file);
} else {
# my @args = ("md5sum", $file);
# system(@args) == 0 or die("System @args failed: $?\n");
$tmpLine = `md5sum "$file"`;
chomp($tmpLine);
$tmpHash = substr($tmpLine,0,32);
$tmpPath = substr($tmpLine,34);
#print "$tmpHash -> $tmpPath\n";
# Now lookup the hash, if the paths aren't the same, move the file
if (defined $sums{$tmpHash}) {
#print $sums{$tmpHash};
if ($tmpPath ne $sums{$tmpHash}) {
print "Moving ".$tmpPath." to ".$sums{$tmpHash}."\n";
# Make directory for move if it doesn't exist
@args = ("mkdir", "-p", dirname($sums{$tmpHash}) );
system(@args) == 0 or print STDERR "Couldn't create directory @args: $!\n";
# Move file to path found in checksum file
@args = ("mv", $tmpPath, $sums{$tmpHash});
system(@args) == 0 or print STDERR "Couldn't move: $!\n";
} else {
print "File $tmpPath is already in place.\n";
}
} else {
print "No hash for $tmpPath found.\n";
}
}
}
return;
}
exit;
A few things still need to be fixed: Inserting a new hash from the file into the hash table should fail if it is already there (hash collision or duplicate files). Also, a reverse lookup in the hash table could save needing to compute the MD5SUM, but at what computational cost?
Tuesday, March 04, 2008
Data/Documents backup script
I used to always create directory under my home called data. I'd store files which I need to do work and my working files. Then, this would be the only directory I had to synchronize with Unison (http://www.cis.upenn.edu/~bcpierce/unison/)between machines. Since Fedora has included a directory called Documents, which has a similar purpose (and so has Mac OS X), I've just changed the name of this special directory from data to Documents.
That brings me to the topic of this article--how do I manage if I mess up a file in this directory, and I synchronize the mistake to other mirrors? That would mean none of the computers I work on has a copy anymore. The answer is rdiff-backup (http://www.nongnu.org/rdiff-backup/). I used to make just a user cron job to call rdiff-backup and update an incremental backup repo on my behalf, but I wanted this to be more mechanical (what if I added a new user, etc.). So I wrote the following script which replaces an old script I used to have which would only make a mirror with rsync.
Note that, on my configuration, the incremental backup repo is on the same device as the original files, but both are mirrored in an overnight backup job. This means I have (at least) four mirrors of the same files, which is sort of a waste but I don't really worry about it since it's about 200 MB / user. Ideally, you would keep just the local mirror on the disk, and the rdiff-backup repository on the backup device (or system over network). The downside here would be, if the backup device (or remote system) crashes you loose the ability to restore a previous increment.
Following is the script for maintaining incremental backup repositories with rdiff-backup for all users. Caution: the backup path should be as read-only as possible, but it is not if the user logs in to the system. For this reason, you'll have to count on them not to modify the files, or keep the files on a filesystem they cannot mount. For simple remote access over samba (the kind of access in my setup), you would have to configure a per-user share (so Joe can't see Susan's backup, for instance) and make each one read-only (so Joe cannot corrupt his repository by changing the files there).
#!/bin/sh
#
# Make incremental backups of 'Documents' directories for users
#
# Ryan Helinski
LOGFILE=/var/log/rdiff-backup.log
# Close stdout and stderr
1>&-; 2>&-;
# Direct stdout and stderr to logfile
exec 1>>$LOGFILE; exec 2>>$LOGFILE;
echo Begin $0, `date`;
# For debugging
#set -x
# Script-specific configuration
HOMESPATH=/home
BACKUPPATH=/opt/backup
RDIFF_OPTIONS="--print-statistics"
DOCSDIR=Documents
# Max increment life
# The time interval is an integer followed by the character s, m, h, D, W,
# M, or Y, indicating seconds, minutes, hours, days, weeks, months, or
# years respectively, or a number of these con-catenated.
MAX_LIFE=4W
for USERHOME in $HOMESPATH/* ; do
# USERDIR=`basename $USERHOME`;
USER=`stat -c "%U" $USERHOME`;
if [ -d "$USERHOME/$DOCSDIR" ] ; then
echo "Updating $BACKUPPATH/$USER/$DOCSDIR";
# Create backup repo if it doesn't exist
if [ ! -d "$BACKUPPATH/$USER/$DOCSDIR" ] ; then
echo "Creating rdiff-backup repo";
mkdir -p "$BACKUPPATH/$USER/$DOCSDIR";
chown $USER "$BACKUPPATH/$USER";
chown $USER "$BACKUPPATH/$USER/$DOCSDIR";
fi
# Update repo
rdiff-backup $RDIFF_OPTIONS "$USERHOME/$DOCSDIR" \
"$BACKUPPATH/$USER/$DOCSDIR" ;
# Clean up repo
#rdiff-backup --remove-older-than $MAX_LIFE --force \
# "$BACKUPPATH/$USER/$DOCSDIR" ;
fi
done
echo End $0, `date`;
echo
exit
To actually remove old backups, uncomment the code in the "Clean up repo" section.
Wednesday, January 30, 2008
Shell Script to Tunnel VNC Sessions over SSH (Improved)
Usage: vnctunnel [ssh session] [screen] [vnc options]
Example: vnctunnel user@server :1It's "improved" because it's completely parameterized--you can specify the SSH session, screen, and even vnc options.
(Assuming you copy it to your ~/bin directory and make it executable)
#!/bin/sh
# This is a parameterized script to SSH tunnel a VNC session
# Ryan Helinski
# Local settings
# Change this if you don't like `vncviewer'
VNCVIEWER=vncviewer
# Change this if you use ports near 25900 for something else locally
PORTOFFSET=20000
# Apply extra options to vncviewer
VNCOPTS="--AutoSelect=0 $3"
if [ $# -lt 2 ] ; then
echo "Usage: $0 [ssh session] [screen]";
echo "Example: $0 user@server :1";
fi
SSHPARM=$1;
SCREEN=`echo $2 | cut -d':' -f2`;
SSHPORT=$[$SCREEN+5900];
echo "Session: $SSHPARM, Screen: $SCREEN, Port: $SSHPORT"
ssh -f -L $[$PORTOFFSET+$SSHPORT]:localhost:$SSHPORT $SSHPARM sleep 10; \
vncviewer localhost:$[$PORTOFFSET+$SSHPORT]:$SCREEN $VNCOPTS
exit
Using 7-zip for Archival
I have seen it a couple times while browsing, and thought "oh...". When I tried to look for it, I really found a lack of examples. After careful reading of the manuals for tar and 7z, following is the solution:
To encode:
tar cf - data/ | 7za a data.tar.7z -siTo decode:
7za x ~/data.tar.7z -so | tar xf -
Monday, January 28, 2008
Trash Management Scripts
Moving files to trash
file will have its name intact, and the backups will have the suffix
".~n~" appended, where the higher the n, the more recent the backup.
#!/bin/sh
#
# Moves files in the command arguments to the trash bin, while keeping
# backups of any files already in that directory.
#
# Uses 'numbered' backup scheme of 'mv', so the most recently trashed
# file will have its name intact, and the backups will have the suffix
# .~n~ appended where the higher the n, the more recent the backup.
#
# It uses the basename of the file so that no (absolute or relative)
# directory is preserved.
#
TRASH="$HOME/.Trash";
while [ $# -gt 0 ];
do
mv --backup=numbered "$1" "$TRASH/`basename $1`";
shift;
done
trash file1 file2 path/to/file3 path/to/file4Rounding up and deleting trash automatically
#!/bin/sh
#
# The first step is to move all files under ~/.Trash, other than
# ., .., and .#bin-n into a new trash bin for yesterday.
#
#
NEWBIN=`date +%s`;
NEWBINPATH="$HOME/.Trash-$NEWBIN";
OLDESTBIN="20"; # days
OLDESTBINTIME=`date -d "now - $OLDESTBIN days" +%s`;
OLDESTBINPATH="$HOME/.Trash-$OLDESTBINTIME";
echo "Moving current trash to $NEWBINPATH";
mkdir $NEWBINPATH;
mv $HOME/.Trash/* $HOME/.Trash/.[!.]* $NEWBINPATH/;
for BIN in $HOME/.Trash-*; do
STAMP=`echo "$BIN" | cut -d'-' -f2`;
# echo $BIN $STAMP $OLDESTBINTIME;
if [ $STAMP -lt $OLDESTBINTIME ] ;
then
echo "Deleting $BIN";
# rm -Rf $BIN;
fi
done
00 4 * * * /home/ryan/bin/trash-roundup.shFriday, January 25, 2008
Using the Seagate FreeAgent as a (periodic) Mirror Backup
I recently purchased a Seagate FreeAgent 100D USB drive. The drive is quite nice, small, quiet, and has advanced power management. I plugged the device in on a Fedora 7 installation, and it came right up.
The first problem I had was the NTFS partition that came on the device. This is easily fixed with fdisk and mkfs.ext3. Don't forget to use e2label, especially if you have more than one of the same device.
The next thing you want to do is determine the "id" of the device, so that you can uniquely identify it. The problem is that udev sets up devices in order of connection (/dev/sda, /dev/sdb, ...). To remedy this, use the symbolic links that udev creates under /dev/disk/ to identify the disk by one of the following means:
- /dev/disk/by-id/ - Uses connection, make, model, and serial number
- /dev/disk/by-uuid/ - Uses the UUID given to the partition when it was created
- /dev/disk/by-label/ - If you have used e2label, this is more meaningful than the UUID
- /dev/disk/by-path/ - This is the one you do not want to use!
I want to disable normal users (including myself) from modifying the contents of the backup, so the plan is to keep it unmounted and disable users from mounting it. Thefefor, this is the appropriate entry for me in /etc/fstab:
/dev/disk/by-id/usb-Seagate_FreeAgentDesktop_30DFK39D-0:0-part1 /media/FreeAgent ext3 defaults 1 2Note this will require the device to be present and consistent at boot-time. If you don't need this, change the last two numbers in the fstab entry to zero.
Before automating backup, I had a problem: The drive spins down automatically, so if you use only a command like rsync it may terminate due to an I/O Error because it simply timed out. This happened with me using ls, but it doesn't always happen -- so you should be more paranoid about the power mode.
The solution is to use a utility called sdparm, which is made to control SCSI (and SATA) disks. This is available for Fedora in a yum package called simply sdparm. This allows you to send a command to the drive to "start" (spin up) or "stop" (spin down).
Then, the final task is to create a backup procedure that is invoked by crond. Rather than writing something to /etc/crontab, I added the following file to /etc/cron.daily/. Make sure it has execute permissions, and call it (as root) a couple times to make sure it's working.
#!/bin/sh
#
# Backup to a USB disk
#
# Env variables
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/
# OK to change these
LOGFILE=/var/log/backup-usb.log
DISKDEVICE=/dev/disk/by-id/usb-Seagate_FreeAgentDesktop_30DFK39D-0\:0
DISKPARTITION=$DISKDEVICE-part1
MOUNTPOINT=/media/FreeAgent
# Leave these alone
DEVICEFILE=`readlink -f $DISKDEVICE`
PARTITIONFILE=`readlink -f $DISKPARTITION`
# Close stdout
1>&-;
# Close stderr
2>&-;
# Log needs to be rotated manually
#mv $LOGFILE $LOGFILE.1
# Direct stdout and stderr to logfile
exec 1>>$LOGFILE;
exec 2>>$LOGFILE;
echo Begin $0, `date`;
# Preliminary work (get backup device online)
if [ -e $DISKDEVICE ] ;
then
echo "Device exists (is connected).";
# /usr/bin/sdparm --all $DISKDEVICE
echo "Sending start (wake up) signal...";
/usr/bin/sdparm --command=start $DISKDEVICE
if [ "$?" -eq "0" ] ;
then
echo "Success";
else
echo "Failed to start device";
exit;
fi
# Get backup partition mounted
if [ `grep $PARTITIONFILE /etc/mtab | wc -l` -le "0" ] ;
then
echo "Device $PARTITIONFILE is not mounted!";
mount $PARTITIONFILE $MOUNTPOINT;
if [ "$?" -eq "0" ] ;
then
echo "Mounted OK.";
else
echo "Failed to mount.";
exit;
fi
else
echo "Partition $PARTITIONFILE is already mounted.";
fi
else
echo "Device doesn't exist, must not be connected!";
exit;
fi
# Backup procedure
#
# For now I'm just using RSYNC because these files don't often change
#
rsync --verbose --itemize-changes \
--archive --hard-links --partial --delete-before \
/opt/srv/ $MOUNTPOINT/srv/ && \
date && \
df -h $MOUNTPOINT && \
echo ;
# NOTE might also want to un-mount the device at this point
# to prevent users from modifying it directly !
echo "Unmounting $PARTITIONFILE"
umount $PARTITIONFILE
echo "Stopping device $DISKDEVICE"
/usr/bin/sdparm --command=stop $DISKDEVICE
# /usr/bin/sdparm --all $DISKDEVICE && \
echo End $0, `date`;The variables section and the rsync command have to be updated to fit your installation and your backup scheme.
Again, this worked well for me, you may have to make adjustments.
Friday, January 18, 2008
Shell Script to Reorganize a Mirror to Match a Newer Mirror
#!/bin/bash
#
# This file takes a checksum file (output from md5sum utility)
# and attempts to reorganize the files in the directory to
# match the listing in the md5 file.
#
# Files not found in the md5 input are left alone.
# Files already in the right place are left alone.
# Other files have their checksums computed and, if they are found
# in the md5 input, they are moved to the appropriate location.
#
# WARNING: It confuses duplicate files!!!
#
if [ $# -lt 1 ];
then
echo "Usage: $0 [checksums file]";
exit;
fi
declare -a SUMS
declare -a NEWPATHS
exec 10<$1
let count=0
echo "Parsing checksums and paths from input file...";
while read LINE <&10; do
SUM=`echo "$LINE" | cut -d' ' -f1`;
NEWPATH=`echo "$LINE" | cut -d' ' -f1 --complement | cut -d' ' -f1 --complement`;
SUMS[$count]=$SUM;
NEWPATHS[$count]=$NEWPATH;
((count++));
done
# close file
exec 10>&-
echo "Compiling list of files that need to be checked...";
TMPFILE=`mktemp`;
find ./ -type f -printf "%p\n" > $TMPFILE;
exec 10<$TMPFILE
while read OLDFILE <&10; do
echo "Trying to find new path for $OLDFILE";
SKIP=0;
let count=0;
while [ $count -lt ${#NEWPATHS[@]} ] ; do
if [ "${NEWPATHS[$count]}" == "$OLDFILE" ]; then
echo "File already exists at '${NEWPATHS[$count]}'";
SKIP=1;
break;
fi
((count++));
done
if [ $SKIP -eq 1 ]; then
continue; #skip the rest of this iteration
fi
echo "Computing checksum of $OLDFILE";
OLDSUM=`md5sum "$OLDFILE" | cut -d' ' -f1`;
# iterate over the pair of arrays until we might find a matching sum
let count=0;
while [ "$count" -lt ${#SUMS[@]} ]; do
SUM=${SUMS[$count]};
NEWPATH=${NEWPATHS[$count]};
if [ "$SUM" == "$OLDSUM" ];
then
if [ "$OLDFILE" != "$NEWPATH" ] ;
then
NEWPARENT=`dirname "$NEWPATH"`;
if [ ! -d "$NEWPARENT" -a "$NEWPARENT" != "." ];
then
echo "Making directory $NEWPARENT";
mkdir "-p" "$NEWPARENT";
fi
echo "Moving $OLDFILE to $NEWPATH";
mv "$OLDFILE" "$NEWPATH";
else
echo "Path hasn't changed.";
fi
break;
fi
((count++));
done
done
exec 10>&-
exit
Friday, January 11, 2008
Scripts for Moving Large Files to DVD (GNU+Linux environment)
Often, we have tons of files (videos, ISO's, software packages, etc.) that we're done with but don't want to throw away. I have written a few scripts to pack a directory of these kinds of files up into volumes (without any kind of compression) so that they can be written to DVD. Each volume should have an MD5 sum file generated which provides both a directory for that volume (since you can keep a copy locally and search using grep) and a means to verify its integrity.
The first script attempts to "pack" files at the current directory level (it's non-recursive) into volumes of a specified size. This is really the only part of this post that warrants a real script since it's a nested loop. Note that really small files will always end up in the extra space on the first volume, so you may want to move them first.
#!/bin/sh
#
# Script takes files (or directories) in the current working directory
# and moves them to subdirectories for writing out to discs
#
# This allows collections of relatively same-sized files or directories
# of files to be packed into volumes for storage on optical media.
#
# Modified to output a shell script instead of making any changes
# Disk capacity can now be entered
# Defaults
discSize="0";
discDefaultSize="4380000";
discInitNumDef="0";
discInitNum="0";
scriptPathDef="pack.sh";
diskPath="disks";
echo -n "Enter the volume number at which to start [$discInitNumDef]: ";
read discNumOffset;
if [ "$discNumOffset" == "" ] ;
then
discNumOffset=$discInitNumDef;
fi
echo $discNumOffset;
echo -n "Enter the maximum capacity of the media [$discDefaultSize]: ";
read discMaxSize;
if [ "$discMaxSize" == "" ] ;
then
discMaxSize=$discDefaultSize;
fi
echo $discMaxSize;
echo -n "A shell script will be output, move files now? [y/N]";
read moveFiles;
if [ "$moveFiles" == "" ];
then
moveFiles="N";
echo "Not going to move files.";
fi
echo -n "Enter the path to save the shell script [$scriptPathDef]: ";
read $scriptPath;
if [ "$scriptPath" == "" ] ;
then
scriptPath=$scriptPathDef;
fi
echo "Going to write shell script to '$scriptPath'.";
# Declare disk size array
diskSizes[0]=0;
arraySize=1;
echo "#!/bin/sh" > $scriptPath;
if [ ! -d "$diskPath" ];
then
echo "mkdir \"$diskPath\";" >> $scriptPath;
fi
if [ ! -d "$diskPath/`expr $discNum + $discNumOffSet`" ] ;
then
echo "mkdir \"$diskPath/`expr $discInitNum + $discNumOffset`\";" >> $scriptPath;
fi
for file in * ;
do
if [ "$file" != "$diskPath" -a "$file" != "$scriptPath" ] ;
then
echo "$file";
discNum=$discInitNum;
newSize=`du -s "$file" | cut -f1`;
#discSize=`du -s todisk/$discNum/ | cut -f1`;
#discSize=`expr $diskSizes[$discNum] + $newSize`;
discSize=${diskSizes[$discNum]};
echo "newSize = $newSize, discSize = $discSize";
if [ $newSize -gt $discMaxSize ] ;
then
echo "$file is larger than the disc size, skipping it.";
else
while [ `expr $discSize + $newSize` -gt $discMaxSize ]
do
echo "Won't fit in $discNum + $discNumOffset: $discSize + $newSize > $discMaxSize";
discNum=`expr $discNum + 1`;
if [ $discNum -ge $arraySize ] ;
then
#diskSizes[$diskNum]=0;
diskSizes=( ${diskSizes[@]} 0 );
arraySize=`expr $arraySize + 1`;
if [ ! -d "$diskPath/`expr $discNum + $discNumOffset`" ];
then
echo "mkdir \"$diskPath/`expr $discNum + $discNumOffset`\";" >> $scriptPath;
fi
fi
discSize=${diskSizes[$discNum]};
done
echo "Going to move $file into $discNum to make $discSize kb `expr $discSize + $newSize`";
echo "mv \"$file\" \"$diskPath/`expr $discNum + $discNumOffset`/\";" >> $scriptPath;
# Update disc size entry
diskSizes[$discNum]=`expr $discSize + $newSize`;
fi
fi
done
echo "Disk sizes:";
for DISC in ${diskSizes[@]}
do
echo "$DISC kb";
done
exit;
The next, albeit more simple script creates checksum files (for later use with md5sum -c ...) in each volume and provides the option to save a copy of the checksum file to another location.
#!/bin/sh
#
# Create checksum files for disk volumes generated by 'disk-pack'.
# These files allow the fidelity of the optical media to be
# evaluated, and allow the contents of the disk to be catalogued.
#
# This file should not change any files; only add new files.
#
CATDIRDEF="`pwd`";
echo -n "Path to save a duplicate of the MD5 checksums [$CATDIRDEF]: ";
read CATDIR;
if [ "$CATDIR" == "" ];
then
CATDIR=$CATDIRDEF;
fi
echo "Saving duplicate checksums in '$CATDIR'";
if [ ! -d "$CATDIR" ];
then
echo "Directory doesn't exist.";
exit;
fi
PREFIXDEF="disk";
echo -n "Prefix to use in checksum file names [$PREFIXDEF]: ";
read PREFIX;
if [ "$PREFIX" == "" ];
then
PREFIX=PREFIXDEF;
fi
echo "Using prefix '$PREFIX'.";
for DISK in [0-9]* ;
do
if [ "$DISK" != "." -a "$DISK" != ".." ]
then
echo "Processing volume $DISK";
cd $DISK;
find . -type f -exec md5sum {} \; | tee ../tempsums.md5;
cd ..;
if [ -e "$CATDIR/$PREFIX$DISK.md5" ];
then
echo "WARNING: Catalog file already exists, using alternate name.";
NUM="0";
while [ -e ""$CATDIR/$PREFIX$DISK-$NUM.md5 ]; do
NUM=`expr $NUM + 1`;
done
cp tempsums.md5 $CATDIR/$PREFIX$DISK-$NUM.md5;
else
cp tempsums.md5 $CATDIR/$PREFIX$DISK.md5;
fi
if [ -e "$DISK/$PREFIX$DISK.md5" ];
then
echo "WARNING: File $DISK/$PREFIX$DISK.md5 already exists, using alternate name.";
NUM="0";
while [ -e $DISK/$PREFIX$DISK-$NUM.md5 ]; do
NUM=`expr $NUM + 1`;
done
mv tempsums.md5 $DISK/$PREFIX$DISK-$NUM.md5;
else
mv tempsums.md5 $DISK/$PREFIX$DISK.md5;
fi
fi
done
Finally, you're ready to put these volumes out to optical media (since you've minimized internal fragmentation, captured a catalog of the files, and took an extra step to preserve integrity). You can do so using your favorite method, but when there are many volumes (like more than three) I prefer to take the following steps.
The following command, if you have genisoimage will create a .iso file for the directory '40' and it will have the volume name "Volume40" when you mount it.
genisoimage -o volume40.iso -J -r -V Volume40 40/
After you have a .iso file, you're almost ready to burn. Always, always, always mount the ISO image (mount -o loop -t iso9660 volume40.iso isotest/), enter it and check some of the MD5 sums to make sure you have a good .iso file! You'll have to check the man page for genisoimage and make sure you're providing the command-line options correctly if the files in the ISO seem corrupted.
If you're familiar with cdrecord, it is now provided by wodim. You need to be root. The command looks like:
wodim -v -eject speed=8 dev='/dev/scd0' volume40.iso
Then, before I delete anything, I always insert the CD, preferably into another optical drive, and run md5sum -c volume40.md5. Now you know you have an exact copy, you can put it in a case and delete the original. Note I'm assuming that if the disc fidelity decays that the files can be found again from the Internet--make sure you have even more redundancy if these are your personal files!
New Split Directory Script
Back at the beginning, I posted a lengthy script that would split up a congested directory alphabetically. Recently, I needed it again, but needed it to be smarter, so I re-wrote it. Also, I figured out how to insert code into Blogger. Enjoy.
#!/bin/sh
#
# by Ryan Helinski, January 2008
#
# This is the second revision of a script that should be used
# when there are too many files or directories at a single level
# on the file system.
#
# It now recognizes the word, "the", and that the name should
# be alphabetized by the words following.
#
# A script is output so the changes can be reviewed before
# any are made.
#
# The script should add to existing bin directories if they
# already exist.
#
# A further improvement would be to allow the split to be
# multi-level.
BINS=(0-9 a b c d e f g h i j k l m n o p q r s t u v w x y z);
BIN_EXPS=(0-9 Aa Bb Cc Dd Ee Ff Gg Hh Ii Jj Kk Ll Mm Nn Oo Pp Qq Rr Ss Tt Uu Vv Ww Xx Yy Zz);
SCRIPT_FILE=".script.sh";
echo "#!/bin/sh" > $SCRIPT_FILE;
for BIN in ${BINS[*]};
do
if [ -d $BIN ];
then
echo "mv $BIN .$BIN" >> $SCRIPT_FILE;
else
echo "mkdir .$BIN" >> $SCRIPT_FILE;
fi
done
INDEX="0";
while [ "$INDEX" -lt "${#BINS[*]}" ];
do
echo "mv [Tt][Hh][Ee]\ [${BIN_EXPS[$INDEX]}]* .${BINS[$INDEX]}/" >> $SCRIPT_FILE;
INDEX=`expr $INDEX + 1`;
done
INDEX="0";
while [ "$INDEX" -lt "${#BINS[*]}" ];
do
echo "mv [${BIN_EXPS[$INDEX]}]* .${BINS[$INDEX]}/" >> $SCRIPT_FILE;
INDEX=`expr $INDEX + 1`;
done
for BIN in ${BINS[*]};
do
echo "mv .$BIN $BIN" >> $SCRIPT_FILE;
done
ANSWER="";
while [ "$ANSWER" != "yes" -a "$ANSWER" != "no" ];
do
echo "Script written to \"$SCRIPT_FILE\", execute now? (yes, no)";
read ANSWER;
done
if [ "$ANSWER" == "yes" ];
then
sh $SCRIPT_FILE;
ANSWER="";
while [ "$ANSWER" != "yes" -a "$ANSWER" != "no" ];
do
echo "Delete script file? (yes, no)";
read ANSWER;
done
if [ "$ANSWER" == "yes" ];
then
rm "$SCRIPT_FILE";
fi
fi
exit;
Tuesday, December 04, 2007
Make OGG not MP3
Personally, I've made the following pledges and hope they inspire someone:
- I will not distribute MP3
I deal with a lot of original content, and after mix-down there's a choice--how to encode it to let someone hear it over the Internet? I will always choose FLAC or Ogg Vorbis, and refuse to send an MP3. - I will maintain my personal collection in Ogg Vorbis
When I buy a new CD, the first thing I do (usually) is run it through GNOME Sound Juicer (which supports OGG and FLAC by default) and create Ogg Vorbis files so I can load it onto music player (software) or portable media player (hardware), and to protect the data from being damaged by wear on the CD. This is the cheif method that OGG could become ubiquitous. Transcoding MP3 to OGG is lossy (it's much better to start from the original sampling), and any audiophile would tell you it's not worth it. Therefor, the most important source of OGG files is at the source (audio CD, FLAC, etc.), and we can become less dependent on MP3 by replacing MP3 files with newly-encoded OGG files. - I will not buy a portable media player that does not support free codecs like Ogg Vorbis
Okay, I still want to be able to listen to MP3, but in order to support pledge #2, I need my portable media device to support OGG out of the box. I personally have a COWON (iAudio) T2, but there are many other options from abroad. - I prefer Ogg Vorbis Internet radio over other formats
Hey, if OGG has the same fidelity at a lower bandwidth (you'll note the rates on Internet streams have a lower average), then we're saving the host bandwidth by preferring an Ogg Vorbis stream. Also, since Internet radio is struggling with royalties, why don't they save themselves money by dropping their MP3 streams and providing only Ogg Vorbis streams? I feel that most Internet radio listeners would not mind having to download an additional codec (if they even need to) to make the switch.
Note that DRM is not even in my vocabulary--it seems like the only "solution" to DRM is to avoid it at all costs. This business of limiting where, when and who can listen to a file (that someone has paid for, mind you) is nonsense. This is exactly the reason we dropped vinyl albums and magnetic tapes for compact discs--so we could hear the same thing again whenever and wherever we want.
Tuesday, November 06, 2007
Finally, Gmail offers IMAP
Thursday, September 20, 2007
GNU Screen -- The best thing since VNC?
Then I remembered I read the description of a software some time ago in pirut, and thought it was a great idea. This software, as I remember it, was much like what I've just found here, but it seems like this project is defunct. I know there was something which was its own daemon and allowed persistent sessions, but...
After a lot of Googling, I finally came accross GNU Screen. I find this software awesome! Basically, it's like a window manager for your Secure Shell, allowing multiple new shells to be opened and switched between, "screen splitting" so you can see two screens at once, persistence of your tasks--which was exactly what I was looking for, and other features like sharing the session between users. Now I can have my multiple terminals running on the remote machine and use them effectively and more efficiently than via VNC.
The Wikipedia article links to many good tutorials in its references section. I recommend you try a few of them, especially this one, but if you're like me--you need to know right away:
- install: yum install screen
- run: screen
- disconnect: CTRL+a d
- reconnect: screen -r -d
Disclaimer: Yes, VNC is newer than Screen. No, Screen is not convenient when you need to use GUI's since even if you get it to work, it's forwarding who X windows, which can be much too slow if you're connecting over WAN. VNC is definitely the way to go if you have GUI programs you want to use in conjunction with text-mode.
Happy screening!
Monday, September 03, 2007
Jamendo
If you are an artist with content that you feel would should be in the public domain, sharing your original recordings is easy. More importantly, you can get feedback on your recordings from the Jamendo community, which can be exciting because all the opinions (although sometimes misguided) are impartial--which is more than you can expect of your friends :) .
Saturday, September 01, 2007
SourceForge.net User Network
Thursday, August 30, 2007
Configure OpenSSH to automatically authenticate
I've been trying to do this for some time now--apparently if you read the man pages carefully enough you can figure this out. I thought I'd need a more advanced SSH client, but OpenSSH already has everything you need--the ssh-keygen program is key. Following is a brief walk-through of how to configure connection from client to server for SSH. Replace the words client and server as appropriate.
client$ ssh-keygen -t rsa
It will ask where to save, you want the default since this is where ssh will look. It will ask for a pass-phrase, and since we're trying to get around having to enter a pass-phrase, we don't want to protect our authentication, so just hit enter twice. Then it will give you a fingerprint and put files in your ~/.ssh directory.
Copy the ~/.ssh/id_rsa.pub to the server machine taking care not to overwrite the same file there:
client$ scp ~/.ssh/id_rsa.pub user@server.address:.ssh/client.pub
Now SSH to the server and add the public key to the authorized keys file like so:
server$ cd ~/.ssh
server$ cat client.pub >> authorized_keys
server$ rm client.pub
The SSH server won't let you use the key unless the file is secured. That is, the keys should only be readable by you.
server$ ls -l authorized_keys
-rw-rw-r-- 1 user users authorized_keys
server$ chmod og-rw authorized_keys
Now we should be able to open an SSH session simply by:
client$ ssh user@server.address
where you can omit the 'user@' part if the user names are the same on the local and remote machines.
Finally, note that doing all this means if you leave your machine open, an intruder has more doors they can open! This is why you should always lock and/or time-out your screen. If the key has no passphrase, then if someone copies it, they can gain access to the machine you opened with this procedure. Therefore, you should restrict access to your machine by configuring /etc/hosts.allow and /etc/hosts.deny. If you're concerned, don't let the passphrase for the key be null (use a passphrase). Taking these steps can help protect your key.
Thursday, August 16, 2007
Cost Review for Luxeon LED Headlight
Purchased:
4x D-cell Batteries ($5)
1x LED Dynamics BoostPuck ($35)
4x Lumileds Luxeon I Star/O ($25)
Subtotal: $65
Spare parts:
1x 270uF capacitor
1x 24-gauge (AWG) lead wire
1x Pentium3 Heat Sink
1x Mounting bracket
1x Rocker switch
2x 2 D-cell Battery Holder
1x 18-gauge (AWG) lead wire
Note that comparable systems are water-resistant and include a rechargeable battery system, but can cost from $120 up to $300.
Rebuild of Luxeon I Bicycle Headlight
This is a practical comparison between two light sources: an old 2-C cell halogen headlight and a single Luxeon I Star (with optics), both driven at the proper power. This is an image with no special adjustments and the sources about 5 feet away. It is easy to see that the Luxeon I has both more flux (total amount of light) and better throw (maximum intensity of beam). Note that halogen bulbs are known to be more efficient than traditional vacuum bulbs, see Lighting efficacy for more. According to my calculations, the power delivered to the halogen bulb is about 1.5 Watts, and the power to the Luxeon I (obviously) is 1 Watt. Note the spectrum of light output, and how this would make riding safer. This comparison allows the efficiency of the LED to really "shine".Note that my power supply (limited to 1 amp) may not have been able to provide enough current to drive the Halogen bulb to the same power as the 2 C-cells, so the halogen light may get brighter with actual batteries. However, the power measurements quoted are absolutely correct and (together with the comparison picture) still illustrate the point of efficiency. Also note that only one Luxeon I (1-watt) is switched on in the comparison picture, whereas four are used in my headlight assembly.
The LED Dynamics BoostPuck 350mA current regulator came today. The model with lead wires attached was not available, so I had to solder on wires. It really is small as they say; even smaller than my rocker switch. I left the extra pins (reference voltage, and dim) not connected.

Using my power supply, I verified all the parts and connections were OK. The meter pictured is showing volts across the input power pins of the BoostPuck (out of frame). The system is was actually pulling down the voltage on my 1-amp-maximum power supply set at 6.0V by 200mV. You can see that the LED's are fully illuminated, which requires at least 12 Volts in their wiring configuration...

I soldered everything together as I mentioned I would in my last post (in series). I included a 270uF capacitor. The only problem with the BoostPuck is that it's a sealed unit and has no way to mount it to anything, unless you use the pins to anchor it (which I don't). It's not heavy, so I didn't think it's all that important. Using industrial hot glue, I fixed it to the rocker switch, and the capacitor to it as well. I expect the hot glue not to hold for long, especially if things heat up. I'll use some epoxy if all goes well--the hot glue comes off easily, epoxy won't.
The light output looks about right. It might be a little dimmer than the unregulated circuit, but it's worth the gains in battery efficiency/power delivery. The BoostPuck is producing over 12VDC across the four LED's in series, which is about what I expected. I haven't actually checked, but I'm assuming that the diode current is 350mA. I didn't make measurements of the current drawn from the batteries because of the AC components.

The light beam itself is continuous (as opposed to strobed); waving hands in front of the beam doesn't reveal individual light pulses as with other charge-pumping lights I've encountered. Also, the intensity differences between the new and old LED's is gone as I expected. The newer LED's needed a higher voltage than the older LED's to produce the same amount of light, and fixing the current at 350mA equalized the light intensity.
For clarity, below is a crude circuit diagram including all the parts.
Tuesday, August 14, 2007
Luxeon LED headlight assembly

Have an old processor heat sink laying around? This Pentium 3 heatsink is the perfect size for four Luxeon I Star/O assemblies (LumiLEDs Product# LXHL-NWE8). Luxeon light sources need very good heat dissipation, and this might be over-doing it, but why not prolong the life of the light sources?
I wanted to get around paying up to $300 for a battery-powered bicycle headlamp, so I decided to meet or exceed the expectations with available parts.
I initially bought only two of these Luxeon Stars (with optics) over a year ago. I didn't have a good way to drive them other than my Elenco Precision (TM) power supply and alkaline batteries, both of which are voltage sources. I tried using two D cell alkaline batteries, this provided 3.0V which is less than the typical 3.4V, and not didn't force enough current to for the LED's to be all the way on. I also didn't have a good mounting bracket, or a good way to mount the batteries to the bicycle frame.
Recently, using the mounting bracket from and old mechanical speedometer and zip ties, I was able to make an excellent mount. This stops the lights from changing pitch (very important for this application) or breaking free. I connected them all in parallel to try to give them all 3.0V from the two D cells.
Lesson learned: Don't solder when the device is a metal core PCB and it's sitting on a heat sink with silicon paste in between! Leave these PCB's on an insulator while soldering.

I noted that they were only drawing around 800mA, nowhere near 1.4A (4x350mA). I decided to increase my power supply voltage potential, but this meant there needs to be another component in series with the light sources to allow the excess voltage to be dissipated. In the application notes from LumiLEDs, they recommend using a 6V power supply and a 10 Ohm resistor for a Luxeon I. I added another battery pack to the frame to get 6.3V, but I decided to use 0.9V rectifying diodes (3A max) rather than four resistors (one in series with each supply) which would require desoldering. This achieved 1.6A with one diode in series and 1.1A with two diodes in series. I stuck with the two diodes in series to stay on the safe side. The diodes get rather hot very quickly since they're dissipating so much power. This is a major problem, by my measurements -- 45% of the power was being dissipated in the diodes themselves, delivering only 55% of the power from the batteries to the light sources. Another reason that a passive approach (linear mode voltage regulation) is bad for alkaline battery applications is that the higher the power delivery, the more the batteries self heat and increase internal resistance. In other words, Alkaline's are less efficient when discharged quickly. It is a common trend that batteries are more efficient with intermittent loading for reasons related to battery theory. For much more, see the IEEE paper BatteryDriven System Design: A New Frontier in Low Power Design, by Kanishka Lahiri, Anand Raghunathan, Sujit Dey, and Debashis Panigrahi.
Lesson learned: The voltage drop across an LED is related to the energy released in each photon--relating to the wavelength of the light emitted. (Note that this is the P-N junction voltage drop, and differs from the external voltage drop and light spectrum is also highly dependent upon construction materials). The current through the LED corresponds to the number of photons emitted--relating to the intensity of light emitted. These brief rules of thumb really help in understanding how to apply high-power LED's.
Lesson learned: Luxeon LED's are constant-current devices, so to get them to the proper luminous intensity and power consumption, they need to be driven at the proper current amperage. For Luxeon I's this is 350mA, for Luxeon III's this is 700mA, etc. This point remained subtle to me because I figured I could always just find a voltage potential which would result in the proper current being drawn. The problem is that LumiLEDs has a tighter manufacturer tolerance of 350mA operating conditions than the forward voltage (Vf) which is loosely maintained. This means each light source will require a slightly different Vf to draw 350mA and use run at the full 1 Watt. The most robust solution is a constant current source which can be used to drive multiple light source in series. This also allows the supply voltage to fluctuate (eg: a draining battery) and not affect the luminous intensity as it would without a regulator.
Lesson learned: A common optimization for alkaline battery-powered applications is to use a charge-pumping power regulator. These are capable of producing voltages higher than their input voltage (12VDC from 6VDC for instance) by intermittently presenting a larger load and driving an oscillating circuit. The typical result is luminous intensity remaining mostly flat until the battery is mostly depleted. Just connecting a 12VDC battery supply would result in an exponentially decaying luminous intensity.
These conclusions lead me to replace my diode-dissipating circuit with a switch-mode power supply including a boost circuit. Because of the scarce availability of through-hole mounting switch-mode power supplies and the necessity for a PCB, I was very happy to find products from LED Dynamics including the BoostPuck. Their Application Figure 3 in their datasheet is my exact application. The regulators are unfortunately around $34, but it's worth the efficiency of a switch-mode power supply, the optimized battery utilization, and proper driving (constant current of 350mA) of the light sources to get the most light without risking burning out the LED.
When the unit arrives, I'll removed the hot glue (to keep the parts clean) and connect the light sources in series and connect the BoostPuck to the ligth sources after the rocker switch. As per the application notes, I'll need to include a capacitor since my batteries are a few feet away. More as soon as the part comes in.
Friday, August 03, 2007
1979 Free Spirit Ten Speed

Once I started riding this old Free Spirit ten-speed, I was hooked. Most of my riding is on a paved recreational trail, so the thin tires aren't a hazard and the bike is a lot more efficient than a Mountain Bike on those conditions.
The bicycle came from Sears & Roebuck circa 1979. I decided that even if I buy a new road bike, that I wanted to keep the Free Spirit around since it was in such good condition compared to a few much younger bikes I have seen. The parts are old (Shimano) parts, but most of the following procedures still apply to most bicycles.
The first problem was that the wheels needed to be "trued" very badly, the crank was loose, it had lots of grime around the chain hardware, the brakes needed to be tensioned, and it had some out of order accessories. The accessories it had were a luggage wrack in the back which had a broken support (which is laying on the ground behind the bike in the picture), and a mechanical speedometer which had a loose cable and just made chirping noises. It also had "soft" plastic handle grips on the end of the dropped handle bars which were petrified and provided no cushioning.
I started by disassembling the whole thing! I cleaned all the parts with CLOROX(tm) disinfecting wipes because they have a cleaning agent which does no harm and are great for absorbing grime. I put the luggage wrack aside because it just got in the way of mounting/dismounting (and was broken). As for the speedometer, modern bicycle computers (speedometer, odometer, tripometer, etc.) are now inexpensive, under $12 at your local "superstore", and allow you to program your wheel size (unlike the mechanical one), so I replaced the huge speedometer with the little computer.

I took the wheels to the local bike shop and had them trued for $5/wheel. They still have a little jitter, but it's nothing compared to the thumping and wobbling that they had before. They had both lateral (off rotation axis) wobble and radial (tending to not form a circle) wobble. I can ride over 20 mph now and not leave the ground.

While the wheels were being worked on, I was able to remove the plastic grips a lot easier than I expected (these things have tendency to become attached). Rubber grips can be a real pain, but I figured the plastic grips would have been fused to the handlebars. I just inserted a flat-head screw driver (on the bottom side to avoid any visible scratching) in between the grip and the handle bar. I didn't pry with the screwdriver, rather just inserted and removed it several times, spraying WD-40 down the length of the screwdriver each time. After about three times, I was able to break each grip free by trying to twist the grip around the handlebar. Once it came free, I took out the screwdriver and worked it off. I cleaned the lubricant off the bars. I bought some handlebar gel tape from Bartrager, and installed them according to these directions since there were none in the box. I guess I have long handlebars because I didn't have to cut the length of the tape, I just made a cut to taper the distal end of the tape so it was flush with the handlebar cap.

The bicycle has a disk brake in the back which needed to be manually tightened because it didn't lock the wheel. The brakes in the front were also too loose. Tip: If brake pads don't lock the wheel, they need more tension. Take the wheel off of the frame, and ask an assistant to hold the pads together. If your caliper makes it easy to disengage the recoiling springs, you won't need an assistant.
Taking the wheel off will provide enough slack to make the adjustment--I found it was impossible otherwise. Adjust where the steel cable is fastened on the caliper side about a centimeter and tighten everything back up.The bike was back in order but the crank still felt bad because it moved inside the frame. It's a one-piece crank and is a lot simpler than I thought before. Take the chain off first. It has a big locking nut which you'll need a big wrench for. Note that the threads are in the opposite direction from most threads, so you take the nut off by turning it in a clockwise direction. Under that, there were some washers which act as dust covers. There is a big nut that the ball bearings ride on which needs to be finger-tightened, it was loose and this is where the crank wobble came from.
I de-greased the ball bearings and the journals inside the frame. It was essentially dry inside just a lot of black deposits on everything. There was some fatigue on the right-hand (gear) side where the ball bearings either crashed down or the bearing retainer was hitting. With one-piece cranks, look inside where the crank fits and look for the obstructions from the other tubes. Find a place to fit the corner of the crank and angle the crank through as to take the gear away from the frame. This will give you more space and will let the crank rest. After everything was clean (I had to use an abrasive brush in some places), I greased everything up as much as I could and put everything back together. I used a screwdriver to make the final torque on the hand-tightened nut, I stopped as soon as I felt no wobbling of the crank. When you put everything back together, you should be able to spin the crank and it should keep spinning for a while. If it doesn't make it more than a few turns after you let go, it's too tight.
After a few other adjustments, like the kickstand position, everything was ready to go. The bike feels a lot tighter like a new bike would. All the rattling, chirping, and wobbling is gone.

The bike will need new tires soon (the thin tires don't last very long), I have about 300 miles on mine and the tread in the back is disappearing. When I replace these, I'll try to disassemble the bearings on the wheels and try to revive them like I did the crank bearings. I'll also replace the chain at that time.


