Storage Prices for March, 2019

I wanted to make sure that it still makes sense to use Blu-ray Recordable (BD-R) discs for archiving old files. So, I got several samples of storage prices from Amazon and computed the cost per gigabyte for various available sizes of several types of storage.

Name	ASIN	Density (GB)	Count	Price	$/GB
Blu-ray Recordable
Verbatim BD-R 25GB 6X Blu-ray Recordable Media Disc - 50 Pack Spindle - 98397	https://www.amazon.com/Verbatim-BD-R-Blu-ray-Recordable-Media/dp/B00GSQ4DBM	25	50	$38.30	$0.0306
Verbatim BD-R 25GB 6X DataLifePlus White Inkjet Printable, Hub Printable - 50pk Spindle	https://www.amazon.com/Verbatim-DataLifePlus-White-Inkjet-Printable/dp/B004477BQQ	25	50	$41.80	$0.0334
Verbatim Blu-ray Disc 20 Spindle - 50 GB 4X Speed BD-R DL - Printable	https://www.amazon.com/Verbatim-Blu-ray-Disc-Spindle-Printable/dp/B0056DURHW	50	20	$45.29	$0.0453
Verbatim Blu-ray Disc 50 pcs Spindle - 50GB 4X BD-R DL - Inkjet Printable	https://www.amazon.com/Verbatim-Blu-ray-Disc-pcs-Spindle/dp/B0056DV0L4	50	50	$102.88	$0.0412
10 Verbatim Bluray 100gb BD-R XL Triple Layer 4x Speed Blu-ray Inkjet Printable Discs	https://www.amazon.com/Verbatim-Bluray-Triple-Blu-ray-Printable/dp/B01DGG5ZP4	100	10	$85.88	$0.0859
1000 Years Archival Verbatim M-Disc BDXL Inkjet Printable | 100GB 4x Speed | 5 Pack Jewel Case	https://www.amazon.com/dp/B01B99WWXI	100	5	$67.35	$0.1347
USB Flash
SanDisk Ultra Flair 128GB USB 3.0 Flash Drive – SDCZ73-128G-G46	https://www.amazon.com/SanDisk-Ultra-Flair-128GB-Flash/dp/B015CH1PJU	128	1	$23.99	$0.1874
Samsung BAR Plus 32GB - 200MB/s USB 3.1 Flash Drive Champagne Silver (MUF-32BE3/AM)	https://www.amazon.com/Samsung-BAR-Plus-32GB-Champagne/dp/B07BPHML28	32	1	$9.99	$0.3122
PNY Turbo 128GB USB 3.0 Flash Drive - P-FD128GTBOP-GE	https://www.amazon.com/PNY-Turbo-128GB-Flash-Drive/dp/B00FE2N1WS	128	1	$21.99	$0.1718
Lexar JumpDrive S75 64GB USB 3.0 Flash Drive - LJDS75-64GABNL (Green)	https://www.amazon.com/Lexar-JumpDrive-64GB-Flash-Drive/dp/B00S5V5PEC	64	1	$14.99	$0.2342
Verbatim 32GB Store 'n' Go USB Flash Drive - PC / Mac Compatible - 3pk - Red, Blue, Green	https://www.amazon.com/Verbatim-32GB-Store-Flash-Drive/dp/B0722DV8R5	32	3	$35.11	$0.3657
SanDisk 256GB Ultra Fit USB 3.1 Flash Drive - SDCZ430-256G-G46	https://www.amazon.com/SanDisk-256GB-Ultra-Flash-Drive/dp/B07857Y17V	256	1	$44.99	$0.1757
Micro SD
SanDisk Ultra 64GB microSDXC UHS-I card with Adapter - 100MB/s U1 A1 - SDSQUAR-064G-GN6MA	https://www.amazon.com/SanDisk-Ultra-microSDXC-Memory-Adapter/dp/B073JYVKNX	64	1	$11.55	$0.1805
Kingston Canvas Select 32GB microSDHC Class 10 microSD Memory Card UHS-I 80MB/s R Flash Memory Card with Adapter (SDCS/32GB)	https://www.amazon.com/Kingston-32GB-microSDHC-microSD-SDCS/dp/B079GTYCW4	32	1	$5.25	$0.1641
Samsung 128GB 100MB/s (U3) MicroSD EVO Select Memory Card with Adapter (MB-ME128GA/AM)	https://www.amazon.com/Samsung-MicroSD-Adapter-MB-ME128GA-AM/dp/B06XWZWYVP	128	1	$22.99	$0.1796
SanDisk 32GB X2 (64GB) MicroSD HC Ultra Uhs-1 Memory Card, Class 10	https://www.amazon.com/SanDisk-MicroSD-Ultra-UHS-1-Memory/dp/B00CNYV942	32	2	$15.70	$0.2453
Samsung 256GB 100MB/s (U3) MicroSDXC EVO Select Memory Card	https://www.amazon.com/Samsung-MicroSDXC-Adapter-MB-ME256GA-AM/dp/B072HRDM55	256	1	$47.99	$0.1875
Silicon Power-32GB High Speed MicroSD Card with Adapter Compatible with Surveillance Camera Wyze, YI, Wansview, TENVIS	https://www.amazon.com/Silicon-Power-32GB-Compatible-Surveillance-Wansview/dp/B07MKK9NX1	32	1	$5.99	$0.1872
Silicon Power-64GB High Speed MicroSD Card with Adapter	https://www.amazon.com/Silicon-Power-64GB-Speed-MicroSD-Adapter/dp/B07F6TS4TD	64	1	$9.99	$0.1561
SATA SSD
Crucial MX500 500GB 3D NAND SATA 2.5 Inch Internal SSD – CT500MX500SSD1(Z)	https://www.amazon.com/Crucial-MX500-500GB-NAND-Internal/dp/B0784SLQM6	500	1	$69.95	$0.1399
Samsung 860 EVO 500GB 2.5 Inch SATA III Internal SSD (MZ-76E500B/AM)	https://www.amazon.com/Samsung-500GB-Internal-MZ-76E500B-AM/dp/B0781Z7Y3S	500	1	$77.99	$0.1560
Kingston A400 SSD 120GB SATA 3 2.5” Solid State Drive SA400S37/120G - Increase Performance	https://www.amazon.com/Kingston-120GB-Solid-SA400S37-120G/dp/B01N6JQS8C	120	1	$19.99	$0.1666
WD Blue 3D NAND 500GB PC SSD - SATA III 6 Gb/s, 2.5"/7mm - WDS500G2B0A	https://www.amazon.com/Blue-NAND-500GB-SSD-WDS500G2B0A/dp/B073SBZ8YH	500	1	$67.99	$0.1360
SanDisk SSD PLUS 480GB Internal SSD - SATA III 6 Gb/s, 2.5"/7mm - SDSSDA-480G-G26	https://www.amazon.com/SanDisk-480GB-Solid-State-SDSSDA-480G-G26/dp/B01F9G46Q8	480	1	$59.99	$0.1250
SanDisk SSD PLUS 1TB Internal SSD - SATA III 6 Gb/s, 2.5"/7mm - SDSSDA-1T00-G26	https://www.amazon.com/SanDisk-SSD-PLUS-Internal-SDSSDA-1T00-G26/dp/B07D998212	1000	1	$124.99	$0.1250

This yields the following graph:

Conclusion: BD-R is indeed the most affordable medium per gigabyte with most options less than 0.10 USD/gigabyte.

Ubuntu Server btrfs Setup

I wanted to practice setting up btrfs on Ubuntu server. My requirement is nightly backups retained for 30 days. I started with a 10GB virtual disk on a VirtualBox VM. I partitioned it with the following table:

/boot      ext4    200MB
/          ext4   5000MB
/home      btrfs  5000MB
swap       swap    537MB

First boot, Sun Oct 22 19:36:46 MDT 2017

Filesystem      Size  Used Avail Use% Mounted on
udev            477M     0  477M   0% /dev
tmpfs           100M  3.2M   97M   4% /run
/dev/sda2       4.5G  1.4G  2.9G  33% /
tmpfs           497M     0  497M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           497M     0  497M   0% /sys/fs/cgroup
/dev/sda3       4.7G   17M  4.2G   1% /home
/dev/sda1       180M   57M  111M  34% /boot
tmpfs           100M     0  100M   0% /run/user/1000

I installed tools for doing automatic backups with btrfs https://github.com/digint/btrbk
sudo apt install btrbk

I created a configuration file using their 'local time-machine' example as a guide.

ryan@ubuntu:~$ cat /etc/btrbk/btrbk.conf
transaction_log            /var/log/btrbk.log
snapshot_dir               _btrbk_snap

timestamp_format           long
snapshot_preserve_daily    30
snapshot_preserve_weekly   0
snapshot_preserve_monthly  0

volume /mnt/btr_pool
  snapshot_dir btrbk_snapshots
  subvolume @home

I had to mount the btrfs main volume under /mnt/btr_pool to the fstab:
ryan@ubuntu:~$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
#              
# / was on /dev/sda2 during installation
UUID=6ae1fc17-428e-4446-941c-f478c71b9cfd /               ext4    errors=remount-ro 0       1
# /boot was on /dev/sda1 during installation
UUID=683a6648-1598-44d3-930e-62aba3b8a4a5 /boot           ext4    defaults        0       2
# /home was on /dev/sda3 during installation
UUID=50e15cf1-f4d8-4ac3-9116-dfea86eb7c33 /home           btrfs   defaults,subvol=@home 0       2
# swap was on /dev/sda4 during installation
UUID=ace0597d-3b9f-45ca-bab2-9a49ff6dbe51 none            swap    sw              0       0
# mount btrfs for backup
UUID=50e15cf1-f4d8-4ac3-9116-dfea86eb7c33 /mnt/btr_pool   btrfs   defaults,subvolid=0 0       0

/etc/cron.daily/btrbk:

#!/bin/sh
exec /usr/sbin/btrbk -q -c /etc/btrbk/btrbk.conf run

After creating some directories, the backups appear as:
ryan@ubuntu:~$ ls /mnt/btr_pool/btrbk_snapshots/
@home.20171022T2149  @home.20171022T2216    @home.20171022T2223
@home.20171022T2151  @home.20171022T2216_1  @home.20171022T2229
@home.20171022T2152  @home.20171022T2216_2  @home.20171022T2231

I had to change the VirtualBox network adapter type and install openssh in order to get my notes off of the VM http://wiredrevolution.com/virtualbox/setup-ssh-access-between-virtualbox-host-and-guest-vms

It might be a good idea to start using git in the /etc directory to keep track of changes. More info: https://unix.stackexchange.com/a/72764

I should try this again, but combining the / and /home partitions into one btrfs partition so that things like /etc and /var can also be tracked.

Sigma 150-500mm

With reference to this earlier post, I wanted to make a comparison of a Nikon 18-200mm lens and a Sigma 150-500mm on a D90.

	18-200mm	150-500mm
Wide
Telephoto

So, the widest the 150-500mm lens will go is only a little wider than the narrowest the 18-200mm lens goes. The narrowest the 150-500mm lens goes is much narrower than the 18-200mm.

360cities Workflow

For a long time, I held this proprietary, but these days it seems like everyone and their brother (no offense to mine) can make panoramas, so here it goes.

For this demonstration, I used a Nikon D-90 and the Nikon 10.5mm Fisheye lens. The first step is to set up and fix the camera. I put it into manual mode and then use autofocus to focus to infinity using something on the horizon. I then set the camera to manual focus so that it won't change. When shooting outside, I will usually use the "sunny 16 rule" and start setting up the exposure by setting the aperture to f/16. Then, with the camera pointed at the horizon or something interesting in the perspective, I use the exposure meter in the diopter display to balance the exposure. I then enable +/- 3EV exposure bracketing, so the camera will shoot nominal, an under exposure and then an over exposure. This step is not strictly necessary, but it makes it possible to correct the specific exposure of regions later. Although I sometimes forget, you must set the white balance and ISO rating while in manual mode. I usually set the white balance to "sunny" or "cloudy" and the ISO to 200. Recently, I've been using the high-quality JPG format for output. It is more economical than RAW format and in JPG mode the camera applies certain lens corrections to the output such as chromatic aberration.

Then I set up the Nodal Ninja with the 45-degree detent ring (8 positions) and I first shoot a row at a pitch of 30-degrees upward. Once I've gone all the way around, I adjust the pitch to 30-degrees downward. At each position, I press the shutter release three times because I am doing exposure bracketing. If there is something interesting below the tripod, I will fold up the tripod and hold it out to take 3 additional pictures of the nadir using the remote shutter release. The video below demonstrates the sequence of positions and the 48 (16*3) pictures that result.

The next step is to load the images onto the computer and to align them with eachother. I have some scripts that automate this process as demonstrated in the video below.

The script takes the images, by the order in which they were taken, and applies the appropriate pitch and yaw in the resulting Hugin PTO file. It also determines the EV value, focal length and FOV of each exposure and sets that in the PTO. The script also runs a control point generator on each pair of adjacent images in the graph below. Essentially, the two rows (actually, cycles) of +/- 30 degrees are connected horizontally, each of the horizontal positions in these rows are connected vertically, and finally the additional under- and over-exposure images are connected to the nominal exposure image. This drastically reduces the amount of time required to create control points and the number of false control points.

At about 2:00 into the video, I launch the panorama previewer. This allows you to see a coarse rendition of what it will look like. The exposure is set to 0 by default, so I click the 'auto EV' button to set it to the average exposure. The pictures are shown in the positions set by the script. The Nodal Ninja and tripod can move slightly, and therefore there will be some error from the planned angles. To correct this, we use the control point optimizer, which also sets the parameters of the camera model to render the images onto the equirectangular projection with low error. To set up the optimizer, we want to allow it to vary the yaw, pitch and roll of all the frames except the first one or any nominal exposure that is level. This will fix the position of that photo as an anchor. For my camera, I allow the tool to optimize the view (v), barrel (b) and distortion (c). I also enable the x shift (d) and y shift (e) to be optimized if necessary.

An iterative process follows whereby I optimize, remove outlying control points, add additional control points, and (sometimes) tweak the optimizer settings until the maximum distance between control points is less than 0.7 pixels. In the fast panorama preview window, you can "Show control points", which allows you to see where you have false control points or where objects have moved during shooting. This is typical of clouds as is seen in the video. The tripod should also be ignored because it will be removed in the final result. Remove any control points on the tripod or panoramic head. Alternatively, you could use a mask before you optimize, but I haven't yet integrated this into my tools. This iterative process is genuinely carried out in the video (no smoke and mirrors).

The next step is to choose a projection and output size. The Hugin tool does a great job of computing the optimal size for your output. I use the equirectangular projection for my output. I will sometimes choose a small size (2000x1000 pixels) for my output to check for major errors. Not shown in the video is the type of output you want to use. Under panorama outputs, check "Exposure fused from any arrangement", Format: "TIFF", Compression: "Packbits". Under remapped images, check "No exposure correction, low dynamic range". Under Layers, check "Blended layers of similar exposure, without exposure correction". For remapper, I use Nona, enfuse for image fusion and enblend for exposure blender. To save space, I check "Save cropped images" under the remapper options.

The next step in the process is to make corrections to the remapped images. This is demonstrated in the video below.

In this image, I had a subject that moved around and wanted to fix that first. I use the extra outputs that I asked for in the Hugin stitcher tab to make put my subject in one piece. In this particular output process, the exposure was slightly different in the resulting panorama than it was in any of the individual exposure layers. Ideally, one would create the three exposure layers as complete panoramas and then blend the result together. Unfortunately, my subject moved inside the exposure bracket, so I had to blend the exposure myself with the GIMP. There is a slight aura in the result which could be fixed with more care. I was hasty for the sake of keeping the video short.

The following video demonstrates how the remapped images are positioned. The tool can form a complete panorama for each exposure setting, and then blend these together. The exposure blending process essentially starts with the nominal expsoure, fills in highlights from the under exposure, and fills in shadows from the over exposure.

The next thing I do is to is remap the final blended image that we just fixed onto a cubic projection. This is done with the erect2cubic tool. The point of this is that it is easier to fix the zenith and nadir in the cubic projection. There is sometimes a dark spot at the zenith and the tripod is seen at the nadir. The process to remove these is shown in the video. If you took additional pictures for what is below the tripod, this can be integrated into your output in a tutorial to come. Usually, you can just replicate the ground around the tripod to cover it up.

I then map these images back to the equirectangular projection. The final output is ready for uploading after a couple checks. I usually look around it closely for fusion errors. I usually check the histogram to make sure I am using most of the exposure spectrum. Then, I choose a JPEG quality to make sure that the output is less than 25MB. A low-quality JPEG is shown below:

ASUS P9X79 Wake on LAN (WOL)

I built a new system including an ASUS P9X79 motherboard. Wake on LAN (WOL) was just not working for me, and a setting that would enable it did not seem to exist in the BIOS (well, the EFI). There's essentially no explicit documentation on how to enable this feature in the owner's manual. Online, I found many people struggling with the same problem on older ASUS motherboards: they seemed to fix the problem by enabling the Intel LAN PXE Boot ROM (LAN PXE OPROM)---this is not necessary! The PXE ROM is for booting over the LAN, not waking in response to a magic packet. There is also a distractor called  ErP Ready which seems to enable more power features when the system is in a standby state, but also seems to be mutually exclusive of other power-on options.

The only setting you need to enable is Advanced \ APM \ Power On By PCIE/PCI. This makes sense because the Intel LAN controller is almost certainly connected as a PCIE or PCI device. I was able to wake the system from the off state using the wakeonlan program (a perl script) from MacPorts, even over WiFi.

Encrypted FS with LUKS

Partitioning a Seagate FreeAgent

Outline

Check for bad blocks (not required)

badblocks -c 10240 -s -w -t random -v /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0

Partition drive. In this case, I create one 'Linux' partition which spans the whole drive.

cfdisk /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0

Create a LUKS device (partition) on the drive

cryptsetup --verbose --verify-passphrase luksFormat /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1

Open that device

cryptsetup luksOpen /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1 FreeAgent

Create an EXT partition on the device.

mkfs.ext3 -j -m 1 -O dir_index,filetype,sparse_super /dev/mapper/FreeAgent

These options aren't necessary. The following will also work fine:

mkfs.ext3 /dev/mapper/FreeAgent

Mount the encrypted partition

mkdir /media/FreeAgentLuks
mount /dev/mapper/FreeAgent /media/FreeAgentLuks

Transcript

This is something I did before, following instructions from somewhere.

[root@exciter ~]# badblocks -c 10240 -s -w -t random -v /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0
Checking for bad blocks in read-write mode
From block 0 to 488386583
Testing with random pattern: done                                
Reading and comparing: done                                
Pass completed, 0 bad blocks found.
[root@exciter ~]# cfdisk /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0
Disk has been changed.

WARNING: If you have created or modified any
DOS 6.x partitions, please see the cfdisk manual
page for additional information.

[root@exciter ~]# cryptsetup --verbose --verify-passphrase luksFormat /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1 

WARNING!
========
This will overwrite data on /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase: 
Verify passphrase: 
Command successful.
[root@exciter ~]# cryptsetup luksOpen /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1 FreeAgent
Enter LUKS passphrase for /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1: 
key slot 0 unlocked.
Command successful.
[root@exciter ~]# mkfs.ext3 -j -m 1 -O dir_index,filetype,sparse_super /dev/mapper/FreeAgent 
mke2fs 1.41.4 (27-Jan-2009)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
30531584 inodes, 122095871 blocks
1220958 blocks (1.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
3727 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
 102400000

Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 34 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Un-mounting

umount /mnt/FreeAgent
rmdir /mnt/FreeAgent
cryptsetup luksClose /dev/mapper/FreeAgent

(safe to remove)

Notes

• With LUKS, you can easily change the pass-phrase, and even have more than one at a time. See luksAddKey in the manual (man cryptsetup).
  

Zoom Lens Comparison

How useful is a high-power zoom lens? Very useful if it means you don't have to switch lenses and carry an extra lens around. The following table shows how much more powerful the Nikon 18-200mm 1:3.5-5.6 G is over the Nikon 28-80mm 1:3.3-5.6 G.

	D50 28-80mm	D90 18-200mm
Wide	28mm	18mm
Tele	80mm	200mm

Within each column, the only variable is the movement of the zoom ring on the lens. Within each row, I changed out the lens and the camera body, so exposure details may vary. Both cameras were on point-and-shoot mode, since this is merely an illustration of the field of view.

Need more swap space?

When I installed Fedora 10 from scratch, I thought that 2GB was probably enough swap space, given that disk access is slow. With only 4GB of RAM, which used to be a lot, as soon as I started using upwards of 6GB, the swap space I had allocated obviously became filled and the program would fail.

Therefore, I am reverting back to the rule of thumb,
"Your swap space should be about twice your RAM size"
which has never failed me.

However, even though I tried, it's a huge pain to make space when you have three hard drives all allocated into a boot partition, a swap partition, and a bunch of LVM partitions, and a logical volume that uses all the LVM space. Is freeing space for a new swap partition possible? Yes. Fast or convenient? No.

Luckily, I discovered that you can use a file in the file system for swap, just as our favourite operating system from Redmond does. I was successful in doing the following on two different machines:

Become the superuser.

$ sudo su

Create a large file of zeros.

# dd if=/dev/zero of=/swapfile1 bs=1024 count=6144k
6291456+0 records in
6291456+0 records out
6442450944 bytes (6.4 GB) copied, 106.946 s, 60.2 MB/s

Make this file a swap partition.

# mkswap -c /swapfile1
Setting up swapspace version 1, size = 6291452 KiB
no label, UUID=7f5ab06e-4315-4185-b5a2-b3151a5a608c

Add the partition to the swap space.

# swapon /swapfile1

Check that it worked; total swap should be 6GB larger now.

# free -m
          total       used       free     shared    buffers     cached
Mem:          3967       3936         30          0         75       3054
-/+ buffers/cache:        807       3159
Swap:         8144          0       8144

Add the swapon command somewhere in this file. This will add the file to the swap space the at boot.

# vi /etc/rc.local

You're done! However, a partition would be faster (and perhaps safer), so the next time you upgrade, consider re-partitioning.

Also note that Linux will use the partition swap first, as per its priority according to:

# cat /proc/swaps
Filename    Type  Size Used Priority
/dev/sdb1                               partition 2048248 96 -1
/swapfile1                              file  6291448 0 -2

And with that in mind, I can think of all sorts of other fixes for running out of swap space (USB drives, compact flash to SATA adapters, etc.).

Making a Time-lapse Video with the Nikon D50 and gphoto2

Who said you cannot teach a 2-year-old digital camera new tricks?

I was really excited when I found out I could control my Digital SLR camera via the USB cable. Nikon produces a software called Camera Control Pro. The software is $170 USD, and there is no non-pro version that I know of. They give you a 30-day trial, so I thought I would check it out.

At first, my camera wouldn't connect. I had to use the menu on the camera to take the USB mode out of "Mass Storage" and into "PTP". The camera stopped bringing up iPhoto and started working in Camera Control. Life was good.

I made a time-lapse of the sunset through my 10.5mm fisheye lens, taking one frame per minute. I used ImageMagick to convert the JPEGs into an MPEG, although this method lacks a lot of the control that I expected: frame rate, bitrate, etc.

As I was looking for a free way to do this, since I was not going to pay almost $200; I have to eat too you know. I (almost by accident) came across gphoto2. I think I had already heard of this software as I was wondering how I could capture images from a webcam, if I had one. The gphoto2 software (available through YUM in Fedora) provides a command-line method to do what I was doing with Camera Control; taking time-lapse pictures. However, it doesn't provide an interface similar to the one on the camera in a GUI form for changing camera settings. No matter, since you can use the controls on the camera directly in PTP mode.

So I made an AT job:

at 6:00am
cd /home/ryan/Pictures/Timelapse/Sunrise
gphoto2 --capture-image --interval=15
[CTRL+D]

I quickly found out that I need to purchase the Nikon EH-5 AC adapter since the camera spends a lot of power in PTP mode and the battery died overnight. Replaced with a spare just in the nick of time and everything went as planned. I should have increased the rate to four times per minute (every 15 seconds), but I was scared after the battery died that with nearly continuous long-shutter shots the battery would die again. As soon as I have the adapter, I'll try going to four frames per minute (or more).

Then, I used a script that had taken some time to develop to compile the frames into video. There are three major steps:

1. Down-sample the stills
2. Crop the stills to some standard video format (here I chose to move the crop region 131 pixels (all the way down) since the earth was more interesting than the sky
   
3. Use good old piping to massage the frames into an MPEG

The following script performs all this, and assumes that the captured images are named as gphoto2 names them, capt0000.jpg, capt0001.jpg, etc.

Here are the packages I needed:

• netpbm-progs
  
• mjpegtools
  

#!/bin/sh

echo "Resizing images to 720p (1280x720)..."
for JPG in capt????.jpg ;
do
TMPNAME=`echo $JPG | sed 's/capt/Mov/' | sed 's/\.jpg/\.ppm/'`;
if [ ! -f $TMPNAME ] ;
then
      echo "$JPG";
      CORRNAME=$JPG
      # Resizing gives an image that is 1280x851, so I need to also crop and shift by 65 pixels (to crop the top and bottom equally)
      convert -resize 1280x1280 -crop 1280x720+0+131 $CORRNAME $TMPNAME;
fi
done

echo "Compiling video...";
ls *.ppm | xargs cat | pamdepth 255 | ppmtoy4m -F 24:1 -S 420mpeg2 | yuvfps -s 4:1 -r 24:1 | mpeg2enc -f 12 -o timelapse.m1v


echo "Done!"

The last piping command

• makes a list of all the PPM's (this could be more elegant),
  
• makes them all arguments to cat (basically, sends them all to stdout),
  
• decreases the PPM color depth to one byte (255 values),
  
• translates the stream of PPM's to a Y4M stream using some standard MPEG settings (24 FPS, and 4:2:0 chroma subsampling mode),
• repeats frames to make four distinct exposures per second but 24 video frames per second,
• and finally compiles an MPEG using format 12 (ATSC 720p video).
  

And, finally, here is the fruit of my labor, which was originally 720p (1280x720 pixels), and 4 frames per second up-sampled to 24 frames per second. The framerate is a bit slow, so you might change the '-s' argument to 'yuvfps' to '8:1' to make it smoother, but with that I recommend to capture more frames per minute.


Yes, you can see my wall clock because a night-shift construction crew was lighting up my whole apartment.

For better results during daytime or night-time (not a transition between the two), I would recommend going to manual exposure control. This would make things a lot smoother.

So to recap, with gphoto2 you can:

• Take more frames per minute, such as 4
• Try making a scene using manual exposure control (like clouds or a tree)
• Create a 1080p (1920×1080) video since your native resolution is probably higher

Watch more on YouTube:

• http://www.youtube.com/watch?v=iWvCHlPRN-k
• http://www.youtube.com/watch?v=SbcrCUPQf-g

Nikon D-50 Cleaning Dust from Sensor

So, on some bright long-exposure shots, I noticed there were some consistent dust spots. I took some slow-shutter, out-of-focus test shots to see. I changed lenses thinking it could be one of the internal elements, but nothing---the dirt was on the sensor. I decided to clean out the inside of the body. Some dust was accumulating on the reflex mirror and the window for the viewfinder. However, if you move the mirror down and out of the way (ever so carefully!), there is a door in the rear which is called the shutter curtain. I thought I was out of luck and needed help, until (completely by chance) I came across a section in the User Manual about cleaning the low-pass filter.

You can see the dirt at about 4:30 in this test shot.

I knew that the low-pass filter was a fancy term for an IR filter that happens to provide an air-tight seal for the CCD sensor. There's a procedure in the manual that enables you to lock open the reflex mirror and the shutter curtain, exposing the low-pass filter and sensor. I highly recommend that you read these instructions; they're very easy to follow. They recommend not to contact the low-pass filter, and I totally agree.

I followed the instructions for locking the mirror and curtain open, and when I removed the body cap, I could clearly see the debris on the sensor. I simply blew some air in there, and it was freed. I recommend blowing air while holding the body upside-down. Make sure you have a lot of light.

Nikon also has directions for setting up test shots for seeing if you have a dust problem. It is geared for their dust removal software, but it also serves as a diagnostic.

Test shot following Nikon's directions after cleaning

You turn the camera off to exit the special lock mode. Presto! No more dirt on my photos!

MD5SUM reorganization revisited

I was running my shell script on a collection of over 10,000 files, and it was running for over three days. The shell interpreter is just way too slow when this amount of work is involved. So, I re-wrote the script in Perl and it is now lightning fast because I am able to take advantage of Perl hashes (associative arrays).

#!/usr/bin/perl

#

# This file takes a checksum file (output from md5sum utility)

# and attempts to reorganize the files in the directory to

# match the listing in the md5 file.

#

# Files not found in the md5 input are left alone.

# Files already in the right place are left alone.

# Other files have their checksums computed and, if they are found

# in the md5 input, they are moved to the appropriate location.

#

# WARNING: It confuses duplicate files!!!

#

use File::Basename;



if ( $#ARGV < reorgpath =" $ARGV[1];" reorgptah = "." lines =" ;



close(SUMSFILE);



foreach my $line (@lines) {

chomp($line);

$sums{substr($line,0,32)} = substr($line,34);

}



print "Read in ".($#lines+1)." checksums and paths.\n";



&reorg($reorgPath);



sub reorg {

my $dir = shift;

#print "Recurring for $dir\n";



opendir DIR, $dir or return;

my @contents =

 map "$dir/$_",

 sort grep !/^\.\.?$/,

 readdir DIR;

closedir DIR;

foreach my $file (@contents) {

 #print "Considering $file\n";

 #next unless !-1 && -d;

 if ( -d $file ) {

  &reorg($file);

 } else {

#   my @args = ("md5sum", $file);

#   system(@args) == 0 or die("System @args failed: $?\n");

  $tmpLine = `md5sum "$file"`;

  chomp($tmpLine);

  $tmpHash = substr($tmpLine,0,32);

  $tmpPath = substr($tmpLine,34);



  #print "$tmpHash -> $tmpPath\n";

  # Now lookup the hash, if the paths aren't the same, move the file

  if (defined $sums{$tmpHash}) {

   #print $sums{$tmpHash};

   if ($tmpPath ne $sums{$tmpHash}) {

    print "Moving ".$tmpPath." to ".$sums{$tmpHash}."\n";



    # Make directory for move if it doesn't exist

    @args = ("mkdir", "-p", dirname($sums{$tmpHash}) );

    system(@args) == 0 or print STDERR "Couldn't create directory @args: $!\n";

    # Move file to path found in checksum file

    @args = ("mv", $tmpPath, $sums{$tmpHash});

    system(@args) == 0 or print STDERR "Couldn't move: $!\n";

   } else {

    print "File $tmpPath is already in place.\n";

   }

  } else {

   print "No hash for $tmpPath found.\n";

  }

 }

}

return;

}



exit;

A few things still need to be fixed: Inserting a new hash from the file into the hash table should fail if it is already there (hash collision or duplicate files). Also, a reverse lookup in the hash table could save needing to compute the MD5SUM, but at what computational cost?

Data/Documents backup script

I used to always create directory under my home called data. I'd store files which I need to do work and my working files. Then, this would be the only directory I had to synchronize with Unison (http://www.cis.upenn.edu/~bcpierce/unison/)between machines. Since Fedora has included a directory called Documents, which has a similar purpose (and so has Mac OS X), I've just changed the name of this special directory from data to Documents.

That brings me to the topic of this article--how do I manage if I mess up a file in this directory, and I synchronize the mistake to other mirrors? That would mean none of the computers I work on has a copy anymore. The answer is rdiff-backup (http://www.nongnu.org/rdiff-backup/). I used to make just a user cron job to call rdiff-backup and update an incremental backup repo on my behalf, but I wanted this to be more mechanical (what if I added a new user, etc.). So I wrote the following script which replaces an old script I used to have which would only make a mirror with rsync.

Note that, on my configuration, the incremental backup repo is on the same device as the original files, but both are mirrored in an overnight backup job. This means I have (at least) four mirrors of the same files, which is sort of a waste but I don't really worry about it since it's about 200 MB / user. Ideally, you would keep just the local mirror on the disk, and the rdiff-backup repository on the backup device (or system over network). The downside here would be, if the backup device (or remote system) crashes you loose the ability to restore a previous increment.

Following is the script for maintaining incremental backup repositories with rdiff-backup for all users. Caution: the backup path should be as read-only as possible, but it is not if the user logs in to the system. For this reason, you'll have to count on them not to modify the files, or keep the files on a filesystem they cannot mount. For simple remote access over samba (the kind of access in my setup), you would have to configure a per-user share (so Joe can't see Susan's backup, for instance) and make each one read-only (so Joe cannot corrupt his repository by changing the files there).

#!/bin/sh
#
# Make incremental backups of 'Documents' directories for users
#
# Ryan Helinski

LOGFILE=/var/log/rdiff-backup.log

# Close stdout and stderr
1>&-; 2>&-;

# Direct stdout and stderr to logfile
exec 1>>$LOGFILE; exec 2>>$LOGFILE;

echo Begin $0, `date`;

# For debugging
#set -x

# Script-specific configuration
HOMESPATH=/home
BACKUPPATH=/opt/backup
RDIFF_OPTIONS="--print-statistics"
DOCSDIR=Documents

# Max increment life 
# The time interval is an integer followed by the character s, m, h, D, W, 
# M, or Y, indicating seconds,  minutes, hours,  days,  weeks,  months, or 
# years respectively, or a number of these con-catenated.
MAX_LIFE=4W

for USERHOME in $HOMESPATH/* ; do

# USERDIR=`basename $USERHOME`;
 USER=`stat -c "%U" $USERHOME`;

 if [ -d "$USERHOME/$DOCSDIR" ] ; then
  echo "Updating $BACKUPPATH/$USER/$DOCSDIR";

  # Create backup repo if it doesn't exist
  if [ ! -d "$BACKUPPATH/$USER/$DOCSDIR" ] ; then
   echo "Creating rdiff-backup repo";
   mkdir -p "$BACKUPPATH/$USER/$DOCSDIR";
   chown $USER "$BACKUPPATH/$USER";
   chown $USER "$BACKUPPATH/$USER/$DOCSDIR";
  fi

  # Update repo
  rdiff-backup $RDIFF_OPTIONS "$USERHOME/$DOCSDIR" \
   "$BACKUPPATH/$USER/$DOCSDIR" ;

  # Clean up repo
  #rdiff-backup --remove-older-than $MAX_LIFE --force \
  # "$BACKUPPATH/$USER/$DOCSDIR" ;

 fi

done

echo End $0, `date`;
echo

exit

To actually remove old backups, uncomment the code in the "Clean up repo" section.

Shell Script to Tunnel VNC Sessions over SSH (Improved)

Following is a parameterized shell script which you can use with this syntax:

Usage: vnctunnel [ssh session] [screen] [vnc options]
Example: vnctunnel user@server :1

It's "improved" because it's completely parameterized--you can specify the SSH session, screen, and even vnc options.

(Assuming you copy it to your ~/bin directory and make it executable)

#!/bin/sh
# This is a parameterized script to SSH tunnel a VNC session
# Ryan Helinski

# Local settings
# Change this if you don't like `vncviewer'
VNCVIEWER=vncviewer
# Change this if you use ports near 25900 for something else locally
PORTOFFSET=20000
# Apply extra options to vncviewer
VNCOPTS="--AutoSelect=0 $3"

if [ $# -lt 2 ] ; then
        echo "Usage: $0 [ssh session] [screen]";
        echo "Example: $0 user@server :1";
fi

SSHPARM=$1;
SCREEN=`echo $2 | cut -d':' -f2`;
SSHPORT=$[$SCREEN+5900];

echo "Session: $SSHPARM, Screen: $SCREEN, Port: $SSHPORT"

ssh -f -L $[$PORTOFFSET+$SSHPORT]:localhost:$SSHPORT $SSHPARM sleep 10; \
        vncviewer localhost:$[$PORTOFFSET+$SSHPORT]:$SCREEN $VNCOPTS

exit

Using 7-zip for Archival

I've been using 7-zip to archive large collections of files--mainly college and documents which have high potential to be compressed. 7-zip has a high compression ratio--higher than bzip2, but also higher than the evil RAR archive everyone seems to like--and it's GPL and widely available for GNU+Linux, UNIX and Mac via `p7zip'. However, I've been slightly concerned that it doesn't retain POSIX file modes. For files like text, graphics, etc. this doesn't really bother me. However, if I was going to archive something which contains scripts, or a shared directory where user ownership needs to be preserved, I'd need to retain this extra information.

I have seen it a couple times while browsing, and thought "oh...". When I tried to look for it, I really found a lack of examples. After careful reading of the manuals for tar and 7z, following is the solution:

To encode:

tar cf - data/ | 7za a data.tar.7z -si

To decode:

7za x ~/data.tar.7z -so | tar xf -

Trash Management Scripts

Moving files to trash

Today's post is about some trash management scripts I have written for myself and would like to share. They should run on most GNU+Linux environments.

If you're like me, you prefer to mv file ~/.Trash/ rather than rm file, but this is both dangerous and hard to type. I created a (very small) script that will act as a command you can use to move files to your Trash bin without overwriting anything.

Uses 'numbered' backup scheme of 'mv', so the most recently trashed
file will have its name intact, and the backups will have the suffix
".~n~" appended, where the higher the n, the more recent the backup.

#!/bin/sh
#
# Moves files in the command arguments to the trash bin, while keeping 
# backups of any files already in that directory. 
#
# Uses 'numbered' backup scheme of 'mv', so the most recently trashed 
# file will have its name intact, and the backups will have the suffix 
# .~n~ appended where the higher the n, the more recent the backup. 
#
# It uses the basename of the file so that no (absolute or relative) 
# directory is preserved.
#
TRASH="$HOME/.Trash";

while [ $# -gt 0 ];
do
        mv --backup=numbered "$1" "$TRASH/`basename $1`";

        shift;
done

After you copy this file to your favorite bin directory and make it executable, you can use the following syntax

trash file1 file2 path/to/file3 path/to/file4

And, even if file1 and file3 have the same name, you'll still be able to find both in your trash bin.

Rounding up and deleting trash automatically

If your trash is anything like mine, the probability that a file is not trash increases exponentially every day. For this reason, I wanted to somehow tag when I threw something out so I could tell how old it was, and maybe delete everything after a specific number of days.

The solution I came up with was to create siblings of the .Trash directory, using UNIX time-stamps. In conjunction with a cron task, yesterday's trash will be in a directory at ~/.Trash-XXXXXXXXXX where the X's are the time-stamp for 4:00am today. In this manner, you'll have a bin for each day you throw something out. The script follows.

#!/bin/sh
#
# The first step is to move all files under ~/.Trash, other than 
# ., .., and .#bin-n into a new trash bin for yesterday.
# 
#

NEWBIN=`date +%s`;
NEWBINPATH="$HOME/.Trash-$NEWBIN";
OLDESTBIN="20"; # days

OLDESTBINTIME=`date -d "now - $OLDESTBIN days" +%s`;
OLDESTBINPATH="$HOME/.Trash-$OLDESTBINTIME";

echo "Moving current trash to $NEWBINPATH";
mkdir $NEWBINPATH;
mv $HOME/.Trash/* $HOME/.Trash/.[!.]* $NEWBINPATH/;

for BIN in $HOME/.Trash-*; do 

        STAMP=`echo "$BIN" | cut -d'-' -f2`;
#       echo $BIN $STAMP $OLDESTBINTIME;
        if [ $STAMP -lt $OLDESTBINTIME ] ;
        then
                echo "Deleting $BIN";
#               rm -Rf $BIN;
        fi

done

So I copied this file into my ~/bin/ directory and used crontab -e to add the following line to my crontab:

00 4 * * * /home/ryan/bin/trash-roundup.sh

Right now this script will send errors to crond, which should be delivered in your local mail (accessed using mail). Also, deleting old bins is disabled since I haven't had a chance to thoroughly test it.

To actually delete old trash, choose a value for the OLDESTBIN variable in the script, this is the longest time that a bin will hang around. Then, you have to un-comment the line with rm in the script.

Using the Seagate FreeAgent as a (periodic) Mirror Backup

I recently purchased a Seagate FreeAgent 100D USB drive. The drive is quite nice, small, quiet, and has advanced power management. I plugged the device in on a Fedora 7 installation, and it came right up.

The first problem I had was the NTFS partition that came on the device. This is easily fixed with fdisk and mkfs.ext3. Don't forget to use e2label, especially if you have more than one of the same device.

The next thing you want to do is determine the "id" of the device, so that you can uniquely identify it. The problem is that udev sets up devices in order of connection (/dev/sda, /dev/sdb, ...). To remedy this, use the symbolic links that udev creates under /dev/disk/ to identify the disk by one of the following means:

• /dev/disk/by-id/ - Uses connection, make, model, and serial number


• /dev/disk/by-uuid/ - Uses the UUID given to the partition when it was created


• /dev/disk/by-label/ - If you have used e2label, this is more meaningful than the UUID


• /dev/disk/by-path/ - This is the one you do not want to use!

I want to disable normal users (including myself) from modifying the contents of the backup, so the plan is to keep it unmounted and disable users from mounting it. Thefefor, this is the appropriate entry for me in /etc/fstab:

/dev/disk/by-id/usb-Seagate_FreeAgentDesktop_30DFK39D-0:0-part1 /media/FreeAgent        ext3    defaults        1 2

Note this will require the device to be present and consistent at boot-time. If you don't need this, change the last two numbers in the fstab entry to zero.

Before automating backup, I had a problem: The drive spins down automatically, so if you use only a command like rsync it may terminate due to an I/O Error because it simply timed out. This happened with me using ls, but it doesn't always happen -- so you should be more paranoid about the power mode.

The solution is to use a utility called sdparm, which is made to control SCSI (and SATA) disks. This is available for Fedora in a yum package called simply sdparm. This allows you to send a command to the drive to "start" (spin up) or "stop" (spin down).

Then, the final task is to create a backup procedure that is invoked by crond. Rather than writing something to /etc/crontab, I added the following file to /etc/cron.daily/. Make sure it has execute permissions, and call it (as root) a couple times to make sure it's working.

#!/bin/sh
#
# Backup to a USB disk 
#

# Env variables 
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/

# OK to change these
LOGFILE=/var/log/backup-usb.log
DISKDEVICE=/dev/disk/by-id/usb-Seagate_FreeAgentDesktop_30DFK39D-0\:0
DISKPARTITION=$DISKDEVICE-part1
MOUNTPOINT=/media/FreeAgent

# Leave these alone
DEVICEFILE=`readlink -f $DISKDEVICE`
PARTITIONFILE=`readlink -f $DISKPARTITION`

# Close stdout
1>&-;
# Close stderr
2>&-;

# Log needs to be rotated manually
#mv $LOGFILE $LOGFILE.1

# Direct stdout and stderr to logfile
exec 1>>$LOGFILE;
exec 2>>$LOGFILE;

echo Begin $0, `date`;

# Preliminary work (get backup device online)
if [ -e $DISKDEVICE ] ;
then
 echo "Device exists (is connected).";

# /usr/bin/sdparm --all $DISKDEVICE

 echo "Sending start (wake up) signal...";
 /usr/bin/sdparm --command=start $DISKDEVICE
 if [ "$?" -eq "0" ] ; 
 then 
  echo "Success";
 else 
  echo "Failed to start device";
  exit;
 fi

 # Get backup partition mounted
 if [ `grep $PARTITIONFILE /etc/mtab | wc -l` -le "0" ] ;
 then
  echo "Device $PARTITIONFILE is not mounted!";
  mount $PARTITIONFILE $MOUNTPOINT;
  if [ "$?" -eq "0" ] ;
  then
   echo "Mounted OK.";
  else
   echo "Failed to mount.";
   exit;
  fi
  
 else 
  echo "Partition $PARTITIONFILE is already mounted.";
 fi
else
 echo "Device doesn't exist, must not be connected!";
 exit;
fi

# Backup procedure 
# 
# For now I'm just using RSYNC because these files don't often change
# 
rsync --verbose --itemize-changes \
--archive --hard-links --partial --delete-before \
/opt/srv/ $MOUNTPOINT/srv/ && \
date && \
df -h $MOUNTPOINT && \
echo ;

# NOTE might also want to un-mount the device at this point 
# to prevent users from modifying it directly !
echo "Unmounting $PARTITIONFILE"
umount $PARTITIONFILE
echo "Stopping device $DISKDEVICE"
/usr/bin/sdparm --command=stop $DISKDEVICE
# /usr/bin/sdparm --all $DISKDEVICE && \

echo End $0, `date`;

The variables section and the rsync command have to be updated to fit your installation and your backup scheme.

Again, this worked well for me, you may have to make adjustments.

Shell Script to Reorganize a Mirror to Match a Newer Mirror

This script has a very specific purpose, but you've probably had a chance where it could have come in handy. The idea is, if there are two mirrors of a bunch of files, and you reorganize (more around and/or rename) files on one mirror, then when you go to RSYNC the other mirror, it thinks the files that were only moved were added and deleted. 

To use this script, you would first move the files on one mirror, and generate an MD5 sum file, as in:

$ find ./ -type f -exec md5sum {} \; | tee checksums.md5

Then, copy this small file to the root of the other mirror, and invoke the script with the name of the checksums file:

$ md5sum-reorg.sh checksums.md5

The script will process every file under the working directory and attempt to correct its path so that it matches a file in the new 'checksums.md5'. Files not found in the checksums file are left alone. Files already in place are left alone (obviously), without even calculating their md5sum.

Script follows:

#!/bin/bash
#
# This file takes a checksum file (output from md5sum utility)
# and attempts to reorganize the files in the directory to 
# match the listing in the md5 file. 
#
# Files not found in the md5 input are left alone.
# Files already in the right place are left alone.
# Other files have their checksums computed and, if they are found 
# in the md5 input, they are moved to the appropriate location.
#
# WARNING: It confuses duplicate files!!!
#

if [ $# -lt 1 ];
then
 echo "Usage: $0 [checksums file]";
 exit;
fi

declare -a SUMS
declare -a NEWPATHS

exec 10<$1
let count=0

echo "Parsing checksums and paths from input file...";

while read LINE <&10; do
 SUM=`echo "$LINE" | cut -d' ' -f1`;
 NEWPATH=`echo "$LINE" | cut -d' ' -f1 --complement | cut -d' ' -f1 --complement`;
 SUMS[$count]=$SUM;
 NEWPATHS[$count]=$NEWPATH;
 ((count++));
done
# close file 
exec 10>&-

echo "Compiling list of files that need to be checked...";

TMPFILE=`mktemp`;
find ./ -type f -printf "%p\n" > $TMPFILE;

exec 10<$TMPFILE
while read OLDFILE <&10; do
 echo "Trying to find new path for $OLDFILE";

 SKIP=0;
 let count=0;
 while [ $count -lt ${#NEWPATHS[@]} ] ; do
  if [ "${NEWPATHS[$count]}" == "$OLDFILE" ]; then
   echo "File already exists at '${NEWPATHS[$count]}'";
   SKIP=1;
   break;
  fi
  ((count++));
 done

 if [ $SKIP -eq 1 ]; then
  continue; #skip the rest of this iteration
 fi


 echo "Computing checksum of $OLDFILE";
 OLDSUM=`md5sum "$OLDFILE" | cut -d' ' -f1`;

 # iterate over the pair of arrays until we might find a matching sum
 let count=0;
 while [ "$count" -lt ${#SUMS[@]} ]; do 
  SUM=${SUMS[$count]};
  NEWPATH=${NEWPATHS[$count]};

  if [ "$SUM" == "$OLDSUM" ];
  then
   if [ "$OLDFILE" != "$NEWPATH" ] ;
   then
    NEWPARENT=`dirname "$NEWPATH"`;
    if [ ! -d "$NEWPARENT" -a "$NEWPARENT" != "." ];
    then
     echo "Making directory $NEWPARENT";
     mkdir "-p" "$NEWPARENT";
    fi
    echo "Moving $OLDFILE to $NEWPATH";
    mv "$OLDFILE" "$NEWPATH";
   else
    echo "Path hasn't changed.";
   fi
   break;
  fi
  ((count++));
 done

done

exec 10>&-

exit

In case it's not clear, this is offered without any warranty or guarantee whatsoever.

Scripts for Moving Large Files to DVD (GNU+Linux environment)

Often, we have tons of files (videos, ISO's, software packages, etc.) that we're done with but don't want to throw away. I have written a few scripts to pack a directory of these kinds of files up into volumes (without any kind of compression) so that they can be written to DVD. Each volume should have an MD5 sum file generated which provides both a directory for that volume (since you can keep a copy locally and search using grep) and a means to verify its integrity.

The first script attempts to "pack" files at the current directory level (it's non-recursive) into volumes of a specified size. This is really the only part of this post that warrants a real script since it's a nested loop. Note that really small files will always end up in the extra space on the first volume, so you may want to move them first.

#!/bin/sh
#
# Script takes files (or directories) in the current working directory 
# and moves them to subdirectories for writing out to discs
#
# This allows collections of relatively same-sized files or directories
# of files to be packed into volumes for storage on optical media.
#
# Modified to output a shell script instead of making any changes
# Disk capacity can now be entered

# Defaults
discSize="0";
discDefaultSize="4380000";
discInitNumDef="0";
discInitNum="0";
scriptPathDef="pack.sh";
diskPath="disks";

echo -n "Enter the volume number at which to start [$discInitNumDef]: ";
read discNumOffset;

if [ "$discNumOffset" == "" ] ;
then
 discNumOffset=$discInitNumDef;
fi
echo $discNumOffset;

echo -n "Enter the maximum capacity of the media [$discDefaultSize]: ";
read discMaxSize;

if [ "$discMaxSize" == "" ] ;
then 
 discMaxSize=$discDefaultSize;
fi
echo $discMaxSize;

echo -n "A shell script will be output, move files now? [y/N]";
read moveFiles;

if [ "$moveFiles" == "" ];
then
 moveFiles="N";
 echo "Not going to move files.";
fi

echo -n "Enter the path to save the shell script [$scriptPathDef]: ";
read $scriptPath;

if [ "$scriptPath" == "" ] ;
then
 scriptPath=$scriptPathDef;
fi

echo "Going to write shell script to '$scriptPath'.";

# Declare disk size array
diskSizes[0]=0;
arraySize=1;

echo "#!/bin/sh" > $scriptPath;

if [ ! -d "$diskPath" ];
then
 echo "mkdir \"$diskPath\";" >> $scriptPath;
fi

if [ ! -d "$diskPath/`expr $discNum + $discNumOffSet`" ] ;
then
 echo "mkdir \"$diskPath/`expr $discInitNum + $discNumOffset`\";" >> $scriptPath;
fi

for file in * ;
do

if [ "$file" != "$diskPath" -a "$file" != "$scriptPath" ] ;
then 
 echo "$file";

 discNum=$discInitNum;

 newSize=`du -s "$file" | cut -f1`;
 #discSize=`du -s todisk/$discNum/ | cut -f1`;
 #discSize=`expr $diskSizes[$discNum] + $newSize`;
 discSize=${diskSizes[$discNum]};

 echo "newSize = $newSize, discSize = $discSize";

 if [ $newSize -gt $discMaxSize ] ;
 then
  echo "$file is larger than the disc size, skipping it.";
 else
  while [ `expr $discSize + $newSize` -gt $discMaxSize ] 
  do
   echo "Won't fit in $discNum + $discNumOffset: $discSize + $newSize > $discMaxSize";

   discNum=`expr $discNum + 1`;
   
   if [ $discNum -ge $arraySize ] ;
   then
    #diskSizes[$diskNum]=0;
    diskSizes=( ${diskSizes[@]} 0 );
    arraySize=`expr $arraySize + 1`;

    if [ ! -d "$diskPath/`expr $discNum + $discNumOffset`" ];
    then
            echo "mkdir \"$diskPath/`expr $discNum + $discNumOffset`\";" >> $scriptPath;
    fi


   fi
   
   discSize=${diskSizes[$discNum]};
  done
  
  echo "Going to move $file into $discNum to make $discSize kb `expr $discSize + $newSize`";
 
  echo "mv \"$file\" \"$diskPath/`expr $discNum + $discNumOffset`/\";" >> $scriptPath;

  # Update disc size entry
  diskSizes[$discNum]=`expr $discSize + $newSize`;
 
 fi

fi

done

echo "Disk sizes:";

for DISC in ${diskSizes[@]}
do
 echo "$DISC kb";
done

exit;

The next, albeit more simple script creates checksum files (for later use with md5sum -c ...) in each volume and provides the option to save a copy of the checksum file to another location.

#!/bin/sh
#
# Create checksum files for disk volumes generated by 'disk-pack'.
# These files allow the fidelity of the optical media to be 
# evaluated, and allow the contents of the disk to be catalogued.
#
# This file should not change any files; only add new files. 
#

CATDIRDEF="`pwd`";
echo -n "Path to save a duplicate of the MD5 checksums [$CATDIRDEF]: ";
read CATDIR;

if [ "$CATDIR" == "" ];
then
 CATDIR=$CATDIRDEF;
fi

echo "Saving duplicate checksums in '$CATDIR'";

if [ ! -d "$CATDIR" ];
then 
 echo "Directory doesn't exist.";
 exit;
fi

PREFIXDEF="disk";
echo -n "Prefix to use in checksum file names [$PREFIXDEF]: ";
read PREFIX;

if [ "$PREFIX" == "" ];
then
        PREFIX=PREFIXDEF;
fi
echo "Using prefix '$PREFIX'.";

for DISK in [0-9]* ;
do

 if [ "$DISK" != "." -a "$DISK" != ".." ]
 then
  echo "Processing volume $DISK";
  cd $DISK;
  find . -type f -exec md5sum {} \; | tee ../tempsums.md5;
  cd ..;

  if [ -e "$CATDIR/$PREFIX$DISK.md5" ];
  then 
   echo "WARNING: Catalog file already exists, using alternate name.";
   NUM="0";
   while [ -e ""$CATDIR/$PREFIX$DISK-$NUM.md5 ]; do
    NUM=`expr $NUM + 1`;
   done
   cp tempsums.md5 $CATDIR/$PREFIX$DISK-$NUM.md5;
  else
   cp tempsums.md5 $CATDIR/$PREFIX$DISK.md5;
  fi

  if [ -e "$DISK/$PREFIX$DISK.md5" ];
  then
   echo "WARNING: File $DISK/$PREFIX$DISK.md5 already exists, using alternate name.";
   NUM="0";
   while [ -e $DISK/$PREFIX$DISK-$NUM.md5 ]; do
    NUM=`expr $NUM + 1`;
   done
   mv tempsums.md5 $DISK/$PREFIX$DISK-$NUM.md5;
  else
   mv tempsums.md5 $DISK/$PREFIX$DISK.md5;
  fi
 fi
done

Finally, you're ready to put these volumes out to optical media (since you've minimized internal fragmentation, captured a catalog of the files, and took an extra step to preserve integrity). You can do so using your favorite method, but when there are many volumes (like more than three) I prefer to take the following steps.

The following command, if you have genisoimage will create a .iso file for the directory '40' and it will have the volume name "Volume40" when you mount it.

genisoimage -o volume40.iso -J -r -V Volume40 40/

After you have a .iso file, you're almost ready to burn. Always, always, always mount the ISO image (mount -o loop -t iso9660 volume40.iso isotest/), enter it and check some of the MD5 sums to make sure you have a good .iso file! You'll have to check the man page for genisoimage and make sure you're providing the command-line options correctly if the files in the ISO seem corrupted.

If you're familiar with cdrecord, it is now provided by wodim. You need to be root. The command looks like:

wodim -v -eject speed=8 dev='/dev/scd0' volume40.iso

Then, before I delete anything, I always insert the CD, preferably into another optical drive, and run md5sum -c volume40.md5. Now you know you have an exact copy, you can put it in a case and delete the original. Note I'm assuming that if the disc fidelity decays that the files can be found again from the Internet--make sure you have even more redundancy if these are your personal files!

New Split Directory Script

Follow-up to: "Making directory traversals more efficient"

Back at the beginning, I posted a lengthy script that would split up a congested directory alphabetically. Recently, I needed it again, but needed it to be smarter, so I re-wrote it. Also, I figured out how to insert code into Blogger. Enjoy.

#!/bin/sh
#
# by Ryan Helinski, January 2008
#
# This is the second revision of a script that should be used 
# when there are too many files or directories at a single level
# on the file system. 
#
# It now recognizes the word, "the", and that the name should 
# be alphabetized by the words following.
#
# A script is output so the changes can be reviewed before
# any are made. 
#
# The script should add to existing bin directories if they 
# already exist.
#
# A further improvement would be to allow the split to be 
# multi-level.

BINS=(0-9 a b c d e f g h i j k l m n o p q r s t u v w x y z);
BIN_EXPS=(0-9 Aa Bb Cc Dd Ee Ff Gg Hh Ii Jj Kk Ll Mm Nn Oo Pp Qq Rr Ss Tt Uu Vv Ww Xx Yy Zz);

SCRIPT_FILE=".script.sh";

echo "#!/bin/sh" > $SCRIPT_FILE;

for BIN in ${BINS[*]};
do
 if [ -d $BIN ];
 then 
  echo "mv $BIN .$BIN" >> $SCRIPT_FILE;
 else
  echo "mkdir .$BIN" >> $SCRIPT_FILE;
 fi
done

INDEX="0";
while [ "$INDEX" -lt "${#BINS[*]}" ];
do
 echo "mv [Tt][Hh][Ee]\ [${BIN_EXPS[$INDEX]}]* .${BINS[$INDEX]}/" >> $SCRIPT_FILE;
 INDEX=`expr $INDEX + 1`;
done

INDEX="0";
while [ "$INDEX" -lt "${#BINS[*]}" ];
do
 echo "mv [${BIN_EXPS[$INDEX]}]* .${BINS[$INDEX]}/" >> $SCRIPT_FILE;
 INDEX=`expr $INDEX + 1`;
done

for BIN in ${BINS[*]};
do
 echo "mv .$BIN $BIN" >> $SCRIPT_FILE;
done

ANSWER="";
while [ "$ANSWER" != "yes" -a "$ANSWER" != "no" ];
do
 echo "Script written to \"$SCRIPT_FILE\", execute now? (yes, no)";
 read ANSWER;
done

if [ "$ANSWER" == "yes" ];
then
 sh $SCRIPT_FILE;
 
 ANSWER="";
 while [ "$ANSWER" != "yes" -a "$ANSWER" != "no" ];
 do
  echo "Delete script file? (yes, no)";
  read ANSWER;
 done

 if [ "$ANSWER" == "yes" ];
 then
  rm "$SCRIPT_FILE";
 fi
fi

exit;

Make OGG not MP3

In reference to recent news, MP3 is still gaining momentum, as Wal-Mart is replacing their Windows Media Audio (WMA) files with MP3. In my humble opinion, MP3 is still better than WMA because it is so well understood (coders, decoders, support, household name, etc.) and WMA is really just a way for Microsoft to get a piece of the Fraunhofer pie. However, as long as MP3 is really "gaining momentum", then we need to collectively do something about it...

Personally, I've made the following pledges and hope they inspire someone:

1. I will not distribute MP3
   I deal with a lot of original content, and after mix-down there's a choice--how to encode it to let someone hear it over the Internet? I will always choose FLAC or Ogg Vorbis, and refuse to send an MP3.
   
2. I will maintain my personal collection in Ogg Vorbis
   When I buy a new CD, the first thing I do (usually) is run it through GNOME Sound Juicer (which supports OGG and FLAC by default) and create Ogg Vorbis files so I can load it onto music player (software) or portable media player (hardware), and to protect the data from being damaged by wear on the CD. This is the cheif method that OGG could become ubiquitous. Transcoding MP3 to OGG is lossy (it's much better to start from the original sampling), and any audiophile would tell you it's not worth it. Therefor, the most important source of OGG files is at the source (audio CD, FLAC, etc.), and we can become less dependent on MP3 by replacing MP3 files with newly-encoded OGG files.
   
3. I will not buy a portable media player that does not support free codecs like Ogg Vorbis
   Okay, I still want to be able to listen to MP3, but in order to support pledge #2, I need my portable media device to support OGG out of the box. I personally have a COWON (iAudio) T2, but there are many other options from abroad.
4. I prefer Ogg Vorbis Internet radio over other formats
   Hey, if OGG has the same fidelity at a lower bandwidth (you'll note the rates on Internet streams have a lower average), then we're saving the host bandwidth by preferring an Ogg Vorbis stream. Also, since Internet radio is struggling with royalties, why don't they save themselves money by dropping their MP3 streams and providing only Ogg Vorbis streams? I feel that most Internet radio listeners would not mind having to download an additional codec (if they even need to) to make the switch.
   

These are reasonable commitments to make, and the second big corporations catch on, we'll start to see MP3 dwindle away. Ogg Vorbis will have to make a statistical win (what is the probability of an audio file being an OGG?), and a compatibility win (what is the probability I can play this OGG?) in order to acheive this.

Note that DRM is not even in my vocabulary--it seems like the only "solution" to DRM is to avoid it at all costs. This business of limiting where, when and who can listen to a file (that someone has paid for, mind you) is nonsense. This is exactly the reason we dropped vinyl albums and magnetic tapes for compact discs--so we could hear the same thing again whenever and wherever we want.