Tuesday, June 09, 2020

HP Envy 15-as133cl Laptop Hard Drive Upgrade

2.5" SSDs are dense enough and cheap enough now that it makes sense to replace 2.5" HDDs. I have been using a M.2 NVMe SSD in this laptop for GNU+Linux, but I still had Windows on the original hard drive. However, whenever I had to boot into Windows, I was reminded just how slow Windows and hard drives are.

First, I used the dd command to mirror the one hard drive to the other. I used a StarTech USB 3.0 to SATA converter. I unmounted the hard drive, then ran dd, which took the better part of a day. Then, it was time for the physical replacement.

Before


The old hard drive weighed in at 89 grams. The new SSD weighed 38 grams, less than half the weight.

After


The real savings should be in power and response times. I tested that Windows still boots, and it only took several seconds. I did not measure the boot times, but it seemed much faster. Time will tell.

Saturday, March 02, 2019

Storage Prices for March, 2019

I wanted to make sure that it still makes sense to use Blu-ray Recordable (BD-R) discs for archiving old files. So, I got several samples of storage prices from Amazon and computed the cost per gigabyte for various available sizes of several types of storage.
Name ASIN Density (GB) Count Price $/GB
Blu-ray Recordable




Verbatim BD-R 25GB 6X Blu-ray Recordable Media Disc - 50 Pack Spindle - 98397 https://www.amazon.com/Verbatim-BD-R-Blu-ray-Recordable-Media/dp/B00GSQ4DBM 25 50 $38.30 $0.0306
Verbatim BD-R 25GB 6X DataLifePlus White Inkjet Printable, Hub Printable - 50pk Spindle https://www.amazon.com/Verbatim-DataLifePlus-White-Inkjet-Printable/dp/B004477BQQ 25 50 $41.80 $0.0334
Verbatim Blu-ray Disc 20 Spindle - 50 GB 4X Speed BD-R DL - Printable https://www.amazon.com/Verbatim-Blu-ray-Disc-Spindle-Printable/dp/B0056DURHW 50 20 $45.29 $0.0453
Verbatim Blu-ray Disc 50 pcs Spindle - 50GB 4X BD-R DL - Inkjet Printable https://www.amazon.com/Verbatim-Blu-ray-Disc-pcs-Spindle/dp/B0056DV0L4 50 50 $102.88 $0.0412
10 Verbatim Bluray 100gb BD-R XL Triple Layer 4x Speed Blu-ray Inkjet Printable Discs https://www.amazon.com/Verbatim-Bluray-Triple-Blu-ray-Printable/dp/B01DGG5ZP4 100 10 $85.88 $0.0859
1000 Years Archival Verbatim M-Disc BDXL Inkjet Printable | 100GB 4x Speed | 5 Pack Jewel Case https://www.amazon.com/dp/B01B99WWXI 100 5 $67.35 $0.1347
USB Flash




SanDisk Ultra Flair 128GB USB 3.0 Flash Drive – SDCZ73-128G-G46 https://www.amazon.com/SanDisk-Ultra-Flair-128GB-Flash/dp/B015CH1PJU 128 1 $23.99 $0.1874
Samsung BAR Plus 32GB - 200MB/s USB 3.1 Flash Drive Champagne Silver (MUF-32BE3/AM) https://www.amazon.com/Samsung-BAR-Plus-32GB-Champagne/dp/B07BPHML28 32 1 $9.99 $0.3122
PNY Turbo 128GB USB 3.0 Flash Drive - P-FD128GTBOP-GE https://www.amazon.com/PNY-Turbo-128GB-Flash-Drive/dp/B00FE2N1WS 128 1 $21.99 $0.1718
Lexar JumpDrive S75 64GB USB 3.0 Flash Drive - LJDS75-64GABNL (Green) https://www.amazon.com/Lexar-JumpDrive-64GB-Flash-Drive/dp/B00S5V5PEC 64 1 $14.99 $0.2342
Verbatim 32GB Store 'n' Go USB Flash Drive - PC / Mac Compatible - 3pk - Red, Blue, Green https://www.amazon.com/Verbatim-32GB-Store-Flash-Drive/dp/B0722DV8R5 32 3 $35.11 $0.3657
SanDisk 256GB Ultra Fit USB 3.1 Flash Drive - SDCZ430-256G-G46 https://www.amazon.com/SanDisk-256GB-Ultra-Flash-Drive/dp/B07857Y17V 256 1 $44.99 $0.1757
Micro SD




SanDisk Ultra 64GB microSDXC UHS-I card with Adapter - 100MB/s U1 A1 - SDSQUAR-064G-GN6MA https://www.amazon.com/SanDisk-Ultra-microSDXC-Memory-Adapter/dp/B073JYVKNX 64 1 $11.55 $0.1805
Kingston Canvas Select 32GB microSDHC Class 10 microSD Memory Card UHS-I 80MB/s R Flash Memory Card with Adapter (SDCS/32GB) https://www.amazon.com/Kingston-32GB-microSDHC-microSD-SDCS/dp/B079GTYCW4 32 1 $5.25 $0.1641
Samsung 128GB 100MB/s (U3) MicroSD EVO Select Memory Card with Adapter (MB-ME128GA/AM) https://www.amazon.com/Samsung-MicroSD-Adapter-MB-ME128GA-AM/dp/B06XWZWYVP 128 1 $22.99 $0.1796
SanDisk 32GB X2 (64GB) MicroSD HC Ultra Uhs-1 Memory Card, Class 10 https://www.amazon.com/SanDisk-MicroSD-Ultra-UHS-1-Memory/dp/B00CNYV942 32 2 $15.70 $0.2453
Samsung 256GB 100MB/s (U3) MicroSDXC EVO Select Memory Card https://www.amazon.com/Samsung-MicroSDXC-Adapter-MB-ME256GA-AM/dp/B072HRDM55 256 1 $47.99 $0.1875
Silicon Power-32GB High Speed MicroSD Card with Adapter Compatible with Surveillance Camera Wyze, YI, Wansview, TENVIS https://www.amazon.com/Silicon-Power-32GB-Compatible-Surveillance-Wansview/dp/B07MKK9NX1 32 1 $5.99 $0.1872
Silicon Power-64GB High Speed MicroSD Card with Adapter https://www.amazon.com/Silicon-Power-64GB-Speed-MicroSD-Adapter/dp/B07F6TS4TD 64 1 $9.99 $0.1561
SATA SSD




Crucial MX500 500GB 3D NAND SATA 2.5 Inch Internal SSD – CT500MX500SSD1(Z) https://www.amazon.com/Crucial-MX500-500GB-NAND-Internal/dp/B0784SLQM6 500 1 $69.95 $0.1399
Samsung 860 EVO 500GB 2.5 Inch SATA III Internal SSD (MZ-76E500B/AM) https://www.amazon.com/Samsung-500GB-Internal-MZ-76E500B-AM/dp/B0781Z7Y3S 500 1 $77.99 $0.1560
Kingston A400 SSD 120GB SATA 3 2.5” Solid State Drive SA400S37/120G - Increase Performance https://www.amazon.com/Kingston-120GB-Solid-SA400S37-120G/dp/B01N6JQS8C 120 1 $19.99 $0.1666
WD Blue 3D NAND 500GB PC SSD - SATA III 6 Gb/s, 2.5"/7mm - WDS500G2B0A https://www.amazon.com/Blue-NAND-500GB-SSD-WDS500G2B0A/dp/B073SBZ8YH 500 1 $67.99 $0.1360
SanDisk SSD PLUS 480GB Internal SSD - SATA III 6 Gb/s, 2.5"/7mm - SDSSDA-480G-G26 https://www.amazon.com/SanDisk-480GB-Solid-State-SDSSDA-480G-G26/dp/B01F9G46Q8 480 1 $59.99 $0.1250
SanDisk SSD PLUS 1TB Internal SSD - SATA III 6 Gb/s, 2.5"/7mm - SDSSDA-1T00-G26 https://www.amazon.com/SanDisk-SSD-PLUS-Internal-SDSSDA-1T00-G26/dp/B07D998212 1000 1 $124.99 $0.1250

This yields the following graph:


Conclusion: BD-R is indeed the most affordable medium per gigabyte with most options less than 0.10 USD/gigabyte.

Tuesday, October 24, 2017

Ubuntu Server btrfs Setup

I wanted to practice setting up btrfs on Ubuntu server. My requirement is nightly backups retained for 30 days. I started with a 10GB virtual disk on a VirtualBox VM. I partitioned it with the following table:

/boot      ext4    200MB
/          ext4   5000MB
/home      btrfs  5000MB
swap       swap    537MB

First boot, Sun Oct 22 19:36:46 MDT 2017

Filesystem      Size  Used Avail Use% Mounted on
udev            477M     0  477M   0% /dev
tmpfs           100M  3.2M   97M   4% /run
/dev/sda2       4.5G  1.4G  2.9G  33% /
tmpfs           497M     0  497M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           497M     0  497M   0% /sys/fs/cgroup
/dev/sda3       4.7G   17M  4.2G   1% /home
/dev/sda1       180M   57M  111M  34% /boot
tmpfs           100M     0  100M   0% /run/user/1000


I installed tools for doing automatic backups with btrfs https://github.com/digint/btrbk
sudo apt install btrbk

I created a configuration file using their 'local time-machine' example as a guide.

ryan@ubuntu:~$ cat /etc/btrbk/btrbk.conf
transaction_log            /var/log/btrbk.log
snapshot_dir               _btrbk_snap

timestamp_format           long
snapshot_preserve_daily    30
snapshot_preserve_weekly   0
snapshot_preserve_monthly  0

volume /mnt/btr_pool
  snapshot_dir btrbk_snapshots
  subvolume @home


I had to mount the btrfs main volume under /mnt/btr_pool to the fstab:
ryan@ubuntu:~$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
#              
# / was on /dev/sda2 during installation
UUID=6ae1fc17-428e-4446-941c-f478c71b9cfd /               ext4    errors=remount-ro 0       1
# /boot was on /dev/sda1 during installation
UUID=683a6648-1598-44d3-930e-62aba3b8a4a5 /boot           ext4    defaults        0       2
# /home was on /dev/sda3 during installation
UUID=50e15cf1-f4d8-4ac3-9116-dfea86eb7c33 /home           btrfs   defaults,subvol=@home 0       2
# swap was on /dev/sda4 during installation
UUID=ace0597d-3b9f-45ca-bab2-9a49ff6dbe51 none            swap    sw              0       0
# mount btrfs for backup
UUID=50e15cf1-f4d8-4ac3-9116-dfea86eb7c33 /mnt/btr_pool   btrfs   defaults,subvolid=0 0       0


/etc/cron.daily/btrbk:
#!/bin/sh
exec /usr/sbin/btrbk -q -c /etc/btrbk/btrbk.conf run

After creating some directories, the backups appear as:
ryan@ubuntu:~$ ls /mnt/btr_pool/btrbk_snapshots/
@home.20171022T2149  @home.20171022T2216    @home.20171022T2223
@home.20171022T2151  @home.20171022T2216_1  @home.20171022T2229
@home.20171022T2152  @home.20171022T2216_2  @home.20171022T2231


I had to change the VirtualBox network adapter type and install openssh in order to get my notes off of the VM http://wiredrevolution.com/virtualbox/setup-ssh-access-between-virtualbox-host-and-guest-vms

It might be a good idea to start using git in the /etc directory to keep track of changes. More info: https://unix.stackexchange.com/a/72764

I should try this again, but combining the / and /home partitions into one btrfs partition so that things like /etc and /var can also be tracked.

Thursday, October 22, 2015

Sigma 150-500mm

With reference to this earlier post, I wanted to make a comparison of a Nikon 18-200mm lens and a Sigma 150-500mm on a D90.



18-200mm 150-500mm
Wide
Telephoto

So, the widest the 150-500mm lens will go is only a little wider than the narrowest the 18-200mm lens goes. The narrowest the 150-500mm lens goes is much narrower than the 18-200mm.

Thursday, December 13, 2012

360cities Workflow

For a long time, I held this proprietary, but these days it seems like everyone and their brother (no offense to mine) can make panoramas, so here it goes.

For this demonstration, I used a Nikon D-90 and the Nikon 10.5mm Fisheye lens. The first step is to set up and fix the camera. I put it into manual mode and then use autofocus to focus to infinity using something on the horizon. I then set the camera to manual focus so that it won't change. When shooting outside, I will usually use the "sunny 16 rule" and start setting up the exposure by setting the aperture to f/16. Then, with the camera pointed at the horizon or something interesting in the perspective, I use the exposure meter in the diopter display to balance the exposure. I then enable +/- 3EV exposure bracketing, so the camera will shoot nominal, an under exposure and then an over exposure. This step is not strictly necessary, but it makes it possible to correct the specific exposure of regions later. Although I sometimes forget, you must set the white balance and ISO rating while in manual mode. I usually set the white balance to "sunny" or "cloudy" and the ISO to 200. Recently, I've been using the high-quality JPG format for output. It is more economical than RAW format and in JPG mode the camera applies certain lens corrections to the output such as chromatic aberration.

Then I set up the Nodal Ninja with the 45-degree detent ring (8 positions) and I first shoot a row at a pitch of 30-degrees upward. Once I've gone all the way around, I adjust the pitch to 30-degrees downward. At each position, I press the shutter release three times because I am doing exposure bracketing. If there is something interesting below the tripod, I will fold up the tripod and hold it out to take 3 additional pictures of the nadir using the remote shutter release. The video below demonstrates the sequence of positions and the 48 (16*3) pictures that result.


The next step is to load the images onto the computer and to align them with eachother. I have some scripts that automate this process as demonstrated in the video below.

The script takes the images, by the order in which they were taken, and applies the appropriate pitch and yaw in the resulting Hugin PTO file. It also determines the EV value, focal length and FOV of each exposure and sets that in the PTO. The script also runs a control point generator on each pair of adjacent images in the graph below. Essentially, the two rows (actually, cycles) of +/- 30 degrees are connected horizontally, each of the horizontal positions in these rows are connected vertically, and finally the additional under- and over-exposure images are connected to the nominal exposure image. This drastically reduces the amount of time required to create control points and the number of false control points.


At about 2:00 into the video, I launch the panorama previewer. This allows you to see a coarse rendition of what it will look like. The exposure is set to 0 by default, so I click the 'auto EV' button to set it to the average exposure. The pictures are shown in the positions set by the script. The Nodal Ninja and tripod can move slightly, and therefore there will be some error from the planned angles. To correct this, we use the control point optimizer, which also sets the parameters of the camera model to render the images onto the equirectangular projection with low error. To set up the optimizer, we want to allow it to vary the yaw, pitch and roll of all the frames except the first one or any nominal exposure that is level. This will fix the position of that photo as an anchor. For my camera, I allow the tool to optimize the view (v), barrel (b) and distortion (c). I also enable the x shift (d) and y shift (e) to be optimized if necessary.

An iterative process follows whereby I optimize, remove outlying control points, add additional control points, and (sometimes) tweak the optimizer settings until the maximum distance between control points is less than 0.7 pixels. In the fast panorama preview window, you can "Show control points", which allows you to see where you have false control points or where objects have moved during shooting. This is typical of clouds as is seen in the video. The tripod should also be ignored because it will be removed in the final result. Remove any control points on the tripod or panoramic head. Alternatively, you could use a mask before you optimize, but I haven't yet integrated this into my tools. This iterative process is genuinely carried out in the video (no smoke and mirrors).

The next step is to choose a projection and output size. The Hugin tool does a great job of computing the optimal size for your output. I use the equirectangular projection for my output. I will sometimes choose a small size (2000x1000 pixels) for my output to check for major errors. Not shown in the video is the type of output you want to use. Under panorama outputs, check "Exposure fused from any arrangement", Format: "TIFF", Compression: "Packbits". Under remapped images, check "No exposure correction, low dynamic range". Under Layers, check "Blended layers of similar exposure, without exposure correction". For remapper, I use Nona, enfuse for image fusion and enblend for exposure blender. To save space, I check "Save cropped images" under the remapper options.

The next step in the process is to make corrections to the remapped images. This is demonstrated in the video below.



In this image, I had a subject that moved around and wanted to fix that first. I use the extra outputs that I asked for in the Hugin stitcher tab to make put my subject in one piece. In this particular output process, the exposure was slightly different in the resulting panorama than it was in any of the individual exposure layers. Ideally, one would create the three exposure layers as complete panoramas and then blend the result together. Unfortunately, my subject moved inside the exposure bracket, so I had to blend the exposure myself with the GIMP. There is a slight aura in the result which could be fixed with more care. I was hasty for the sake of keeping the video short.

The following video demonstrates how the remapped images are positioned. The tool can form a complete panorama for each exposure setting, and then blend these together. The exposure blending process essentially starts with the nominal expsoure, fills in highlights from the under exposure, and fills in shadows from the over exposure.


The next thing I do is to is remap the final blended image that we just fixed onto a cubic projection. This is done with the erect2cubic tool. The point of this is that it is easier to fix the zenith and nadir in the cubic projection. There is sometimes a dark spot at the zenith and the tripod is seen at the nadir. The process to remove these is shown in the video. If you took additional pictures for what is below the tripod, this can be integrated into your output in a tutorial to come. Usually, you can just replicate the ground around the tripod to cover it up.

I then map these images back to the equirectangular projection. The final output is ready for uploading after a couple checks. I usually look around it closely for fusion errors. I usually check the histogram to make sure I am using most of the exposure spectrum. Then, I choose a JPEG quality to make sure that the output is less than 25MB. A low-quality JPEG is shown below:


Friday, May 25, 2012

ASUS P9X79 Wake on LAN (WOL)

I built a new system including an ASUS P9X79 motherboard. Wake on LAN (WOL) was just not working for me, and a setting that would enable it did not seem to exist in the BIOS (well, the EFI). There's essentially no explicit documentation on how to enable this feature in the owner's manual. Online, I found many people struggling with the same problem on older ASUS motherboards: they seemed to fix the problem by enabling the Intel LAN PXE Boot ROM (LAN PXE OPROM)---this is not necessary! The PXE ROM is for booting over the LAN, not waking in response to a magic packet. There is also a distractor called  ErP Ready which seems to enable more power features when the system is in a standby state, but also seems to be mutually exclusive of other power-on options.

The only setting you need to enable is Advanced \ APM \ Power On By PCIE/PCI. This makes sense because the Intel LAN controller is almost certainly connected as a PCIE or PCI device. I was able to wake the system from the off state using the wakeonlan program (a perl script) from MacPorts, even over WiFi.

Saturday, January 09, 2010

Encrypted FS with LUKS

Partitioning a Seagate FreeAgent



Outline



Check for bad blocks (not required)
badblocks -c 10240 -s -w -t random -v /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0


Partition drive. In this case, I create one 'Linux' partition which spans the whole drive.
cfdisk /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0


Create a LUKS device (partition) on the drive
cryptsetup --verbose --verify-passphrase luksFormat /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1


Open that device
cryptsetup luksOpen /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1 FreeAgent


Create an EXT partition on the device.
mkfs.ext3 -j -m 1 -O dir_index,filetype,sparse_super /dev/mapper/FreeAgent


These options aren't necessary. The following will also work fine:
mkfs.ext3 /dev/mapper/FreeAgent


Mount the encrypted partition
mkdir /media/FreeAgentLuks
mount /dev/mapper/FreeAgent /media/FreeAgentLuks


Transcript



This is something I did before, following instructions from somewhere.

[root@exciter ~]# badblocks -c 10240 -s -w -t random -v /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0
Checking for bad blocks in read-write mode
From block 0 to 488386583
Testing with random pattern: done
Reading and comparing: done
Pass completed, 0 bad blocks found.
[root@exciter ~]# cfdisk /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0
Disk has been changed.

WARNING: If you have created or modified any
DOS 6.x partitions, please see the cfdisk manual
page for additional information.

[root@exciter ~]# cryptsetup --verbose --verify-passphrase luksFormat /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1

WARNING!
========
This will overwrite data on /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase:
Verify passphrase:
Command successful.
[root@exciter ~]# cryptsetup luksOpen /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1 FreeAgent
Enter LUKS passphrase for /dev/disk/by-id/usb-Seagate_FreeAgentDesktop-0:0-part1:
key slot 0 unlocked.
Command successful.
[root@exciter ~]# mkfs.ext3 -j -m 1 -O dir_index,filetype,sparse_super /dev/mapper/FreeAgent
mke2fs 1.41.4 (27-Jan-2009)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
30531584 inodes, 122095871 blocks
1220958 blocks (1.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
3727 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 34 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.


Un-mounting


umount /mnt/FreeAgent
rmdir /mnt/FreeAgent
cryptsetup luksClose /dev/mapper/FreeAgent

(safe to remove)

Notes




  • With LUKS, you can easily change the pass-phrase, and even have more than one at a time. See luksAddKey in the manual (man cryptsetup).