New Split Directory Script

Follow-up to: "Making directory traversals more efficient"

Back at the beginning, I posted a lengthy script that would split up a congested directory alphabetically. Recently, I needed it again, but needed it to be smarter, so I re-wrote it. Also, I figured out how to insert code into Blogger. Enjoy.

#!/bin/sh
#
# by Ryan Helinski, January 2008
#
# This is the second revision of a script that should be used 
# when there are too many files or directories at a single level
# on the file system. 
#
# It now recognizes the word, "the", and that the name should 
# be alphabetized by the words following.
#
# A script is output so the changes can be reviewed before
# any are made. 
#
# The script should add to existing bin directories if they 
# already exist.
#
# A further improvement would be to allow the split to be 
# multi-level.

BINS=(0-9 a b c d e f g h i j k l m n o p q r s t u v w x y z);
BIN_EXPS=(0-9 Aa Bb Cc Dd Ee Ff Gg Hh Ii Jj Kk Ll Mm Nn Oo Pp Qq Rr Ss Tt Uu Vv Ww Xx Yy Zz);

SCRIPT_FILE=".script.sh";

echo "#!/bin/sh" > $SCRIPT_FILE;

for BIN in ${BINS[*]};
do
 if [ -d $BIN ];
 then 
  echo "mv $BIN .$BIN" >> $SCRIPT_FILE;
 else
  echo "mkdir .$BIN" >> $SCRIPT_FILE;
 fi
done

INDEX="0";
while [ "$INDEX" -lt "${#BINS[*]}" ];
do
 echo "mv [Tt][Hh][Ee]\ [${BIN_EXPS[$INDEX]}]* .${BINS[$INDEX]}/" >> $SCRIPT_FILE;
 INDEX=`expr $INDEX + 1`;
done

INDEX="0";
while [ "$INDEX" -lt "${#BINS[*]}" ];
do
 echo "mv [${BIN_EXPS[$INDEX]}]* .${BINS[$INDEX]}/" >> $SCRIPT_FILE;
 INDEX=`expr $INDEX + 1`;
done

for BIN in ${BINS[*]};
do
 echo "mv .$BIN $BIN" >> $SCRIPT_FILE;
done

ANSWER="";
while [ "$ANSWER" != "yes" -a "$ANSWER" != "no" ];
do
 echo "Script written to \"$SCRIPT_FILE\", execute now? (yes, no)";
 read ANSWER;
done

if [ "$ANSWER" == "yes" ];
then
 sh $SCRIPT_FILE;
 
 ANSWER="";
 while [ "$ANSWER" != "yes" -a "$ANSWER" != "no" ];
 do
  echo "Delete script file? (yes, no)";
  read ANSWER;
 done

 if [ "$ANSWER" == "yes" ];
 then
  rm "$SCRIPT_FILE";
 fi
fi

exit;

Web File Search CGI Program for Linux

So you can install Fedora, and choose the Web Server package. If you are running any kind of file server, you will probably want to find a file over the web. This script allows users to search the mlocate database which is usually compiled by your Linux installation every night. It's a fast database which circumvents searching the filesystem manually. I had to write it, so I extend my source code to everyone in hopes that it will help someone.

It has some thought put into security. The results are displayed as links to facilitate viewing/downloading the located file. It should be used to search one part of the file system. In my case, the public area is /srv/. This part of the filesystem should be linked into the Apache document root via

ln -s /srv /var/www/html/srv

to allow the users access to the files with your HTTP server.

Following is the Perl-CGI code. Name it locate.cgi and it should go in your cgi-bin directory. Don't forget to chmod 755 the file so it's executable. Because of technical limitations of Blogger, I can't paste the Perl-CGI code here--please follow the following link:

http://www.cs.umbc.edu/~rhelins1/getfile.php?id=26

Shell Scripts for making links unique or sole copies

Here are scripts to convert soft links to unique copies, and to sole copied (which moves the target of the link to the location of the link).

#!/bin/sh

# Check that link exists
ls "$1" > /dev/null
if test $? -eq 0
then
echo -n "File found. "
else
echo "Error: File not found."
exit 1
fi

linktarget=`find "$1" -printf "%l\0"`
linktargettype=`stat --format=%F "$linktarget"`

# Check that link target exists
if test -z "$linktarget"
then
  echo "Error: Null link target, not a valid soft link."
  exit 1
else
  echo -n "Found soft link. "
fi

# Check that link target is NOT a directory
#if test "$linktargettype" = "directory"
#  then
#    echo "Link to directory, skipping."
#    exit 0
#  else
#    echo "Not a directory."
#fi

# Remove soft link
echo -n "Unlinking $1... "
unlink "$1"
if test $? -eq 0
then
  echo "Done."
else
  echo "Error!"
  exit 1
fi

# Replace soft link with unique copy
echo "Creating copy from $1"
echo -n " to $linktarget... "
cp -a "$linktarget" "$1"
if test $? -eq 0
then
  echo "Done."
else
  echo "Error!"
  exit 1
fi


exit 0

Here's the same script for changing the link to the unique copy:

# Check that link exists
ls "$1" > /dev/null
if test $? -eq 0
then
echo -n "File found. "
else
echo "Error: File not found."
exit 1
fi

linktarget=`find "$1" -printf "%l\0"`
linktargettype=`stat --format=%F "$linktarget"`

# Check that link target exists
if test -z "$linktarget"
 then
   echo "Error: Null link target, not a valid soft link."
   exit 1
 else
   echo -n "Found soft link. "
fi

# Check that link target is NOT a directory
#if test "$linktargettype" = "directory"
#  then
#    echo "Link to directory, skipping."
#    exit 0
#  else
#    echo "Not a directory."
#fi

# Remove soft link
echo -n "Unlinking $1... "
unlink "$1"
if test $? -eq 0
 then
   echo "Done."
 else
   echo "Error!"
   exit 1
fi

# Replace soft link with only copy
echo "Moving $1"
echo -n " to $linktarget... "
mv "$linktarget" "$1"
if test $? -eq 0
 then
   echo "Done."
 else
   echo "Error!"
   exit 1
fi


exit 0

Convert Soft Links to Hard Links

Here's a Shell script I wrote to convert large numbers of soft links to hard links on Linux.
It now handles soft links on different file systems properly.

#!/bin/sh
# soft2hard.sh by Ryan Helinski
# Replace a soft (symbolic) link with a hard one.
#
# $1 is name of soft link
# Returns 0 on success, 1 otherwise
#
# Example: To replace all the soft links in a particular directory:
# find ./ -type l | tr \\n \\0 | xargs -0 -n 1 soft2hard.sh
#
# finds all files under ./ of type link (l), replaces (tr) the newline
# characters with null characters and then pipes each filename one-by-one
# to soft2hard.sh

# Check that link exists
ls "$1" > /dev/null
if test $? -eq 0
then
echo -n "File found. "
else
echo "Error: File not found."
exit 1
fi

linktarget=`find "$1" -printf "%l\0"`
linktargettype=`stat --format=%F "$linktarget"`

# Check that link target exists
if test -z "$linktarget"
 then
   echo "Error: Null link target, not a soft link."
   exit 1
 else
   echo -n "Found soft link. "
fi


# Check that link target is NOT a directory
if test "$linktargettype" = "directory"
 then
   echo "Link to directory, skipping."
   exit 0
 else
   echo "Not a directory."
fi

# Remove soft link
echo -n "Unlinking $1... "
unlink "$1"
if test $? -eq 0
 then
   echo "Done."
 else
   echo "Error!"
   exit 1
fi

# Replace with hard link
echo "Creating hard link from $1"
echo -n " to $linktarget... "
ln "$linktarget" "$1"
if test $? -eq 0
 then
   echo "Done."
 else
   echo "Error creating hard link, replacing soft link"
   ln -s "$linktarget" "$1"
   exit 1
fi


exit 0

Optimizing Multimedia and Backup Storage with Hard or Soft Links

Here are the notes on this subject in my Server Log with some annotations. Basically, it's (hard links are) a quick fix for duplicate files to free up space.

I had md5 files generated for everything that was copied onto the server, so using the shell commands sort, grep, and uniq, I was able to clean up a lot of space from files that had been copied over twice as a result of using WinMerge to prepare for deleting the backups that were on one of the hard drives that went into the RAID.

I was about to start playing with FSlint, but by chance came across a perl program called dupseek. I was looking for a script which replaced one of two duplicate files with a link.

As far as I can tell, this is an excellent program with a well thought-out algorithm for finding duplicate files on a Unix system (based on personal experience, and what it says on their page), but more importantly to me, it has a function for creating Unix soft links in place of the duplicate files. Although it's text-mode, this is the best program I've used for dealing with duplicate files. And text-mode is just fine! This is a real life-saver because I don't want duplicates sitting on the file system, and I'd like to keep some files cross-referenced in directories. Also, removing duplicates in directories which are already backed up will make the copy on hard disk seem like it has fewer files than that on the CD.

Hard links would really be better (for me), because soft (symbolic) links in Unix probably raise some compatibility issues when, for instance, trying to but the directory on a CD. Unless interpreted correctly, these links are just files. With hard links, however, the same file system inode is just referenced by two different directories. Since the directories in question won't be changing, this wouldn't raise any issues (deletion, separation, etc.).

In fact, gnomebaker currently has a bug where soft links are dereferenced for computer the size of the CD image, but the link file itself if put onto the CD.

I was able to save less than 10 GB by realizing there were duplicate media files as a result of combining directories before the big move to the server RAID, and more than 2 GB by using dupseek.

http://www.pixelbeat.org/fslint/
http://www.beautylabs.net/software/dupseek.html

Notes:

You can use a text file, generated yourself or by filtering the report generated the “-b report” function of dupseek, which contains filenames to be removed. You can pipe these names to xargs which calls rm. This is useful in case files have been copied to more specific directories and many duplicates lie in a general directory (e.g. “downloads” vs. “singles”).

cat [name of report file] | grep “/Downloads/” | tr \\n \\0 | xargs -0 rm

Which accomplishes the task of filtering the report to only files which are in a path which includes
“/Downloads/”, replacing the new line characters with null characters, and xargs passes each of these lines to rm. This removes the duplicate files which are in the common directory (which you want to clean out, and not preserve). Also, this must be executed from the same directory the report file was created in reference to (to make the relative filenames match up). For safety, try replacing rm with ls before you do anything to make sure you're about to remove the right files:

cat [name of report file] | tr \\n \\0 | xargs -0 ls | less

Only after checking over this output, hit q and then run:

cat [name of report file] | tr \\n \\0 | xargs -0 rm

Note: A much safer plan is to use the interactive mode of dupseek, or use the FSlint GUI (takes into account hard links). The interactive mode of dupseek got too repetitive and so I just used it to identify duplicates in one case, but in general the interactive or batch mode is fine (be careful of the batch mode, unless you're running my version of dupseek).

In the end, dupseek is better for batch jobs (but I feel that only with my hard link modification) and FSLint is better for compatibility and running on directories like home folders where it is the case that you want to leave some unique files alone. Of course, a compressed file system would be a step better, but who has the (CPU) time for that?

Making directory traversals more efficient

I wrote a script which takes less than a second to run, and which splits bulky folders up into 28 different subfolders folders. If there's a well-organized folder with more than 30-50 subfolders under it, then they're all the same type of subdirectory and they only differ in their name. More than thirty or fifty subfolders is too much for your users to digest at once, and it slows down traversing the file system. It takes longer for Linux to get all the files necessary to list all the subfolders, and if you're browsing with Apache, it might take a while to download the whole listing.

A very popular solution for this type of directory is to subdivide the collection of similar subfolders by their name. I wrote a script which does this based on the first character of the folder name. This way, you can divide a huge directory into chunks that make traversals more efficient for the computer and more responsive to your users (even over CIFS, FTP, NFS, etc.).

Following is the script code:

#!/bin/sh

mkdir /tmp/myindex

mkdir /tmp/myindex/0-9
mkdir /tmp/myindex/misc
mkdir /tmp/myindex/a
mkdir /tmp/myindex/b
mkdir /tmp/myindex/c
mkdir /tmp/myindex/d
mkdir /tmp/myindex/e
mkdir /tmp/myindex/f
mkdir /tmp/myindex/g
mkdir /tmp/myindex/h
mkdir /tmp/myindex/i
mkdir /tmp/myindex/j
mkdir /tmp/myindex/k
mkdir /tmp/myindex/l
mkdir /tmp/myindex/m
mkdir /tmp/myindex/n
mkdir /tmp/myindex/o
mkdir /tmp/myindex/p
mkdir /tmp/myindex/q
mkdir /tmp/myindex/r
mkdir /tmp/myindex/s
mkdir /tmp/myindex/t
mkdir /tmp/myindex/u
mkdir /tmp/myindex/v
mkdir /tmp/myindex/w
mkdir /tmp/myindex/x
mkdir /tmp/myindex/y
mkdir /tmp/myindex/z

mv ./[0-9]* /tmp/myindex/0-9/
mv ./[!0-9a-zA-Z]* /tmp/myindex/misc/
mv ./[Aa]* /tmp/myindex/a/
mv ./[Bb]* /tmp/myindex/b/
mv ./[Cc]* /tmp/myindex/c/
mv ./[Dd]* /tmp/myindex/d/
mv ./[Ee]* /tmp/myindex/e/
mv ./[Ff]* /tmp/myindex/f/
mv ./[Gg]* /tmp/myindex/g/
mv ./[Hh]* /tmp/myindex/h/
mv ./[Ii]* /tmp/myindex/i/
mv ./[Jj]* /tmp/myindex/j/
mv ./[Kk]* /tmp/myindex/k/
mv ./[Ll]* /tmp/myindex/l/
mv ./[Mm]* /tmp/myindex/m/
mv ./[Nn]* /tmp/myindex/n/
mv ./[Oo]* /tmp/myindex/o/
mv ./[Pp]* /tmp/myindex/p/
mv ./[Qq]* /tmp/myindex/q/
mv ./[Rr]* /tmp/myindex/r/
mv ./[Ss]* /tmp/myindex/s/
mv ./[Tt]* /tmp/myindex/t/
mv ./[Uu]* /tmp/myindex/u/
mv ./[Vv]* /tmp/myindex/v/
mv ./[Ww]* /tmp/myindex/w/
mv ./[Xx]* /tmp/myindex/x/
mv ./[Yy]* /tmp/myindex/y/
mv ./[Zz]* /tmp/myindex/z/

mv /tmp/myindex/* ./
rmdir /tmp/myindex

exit 0

Some 'for' loops would make this code substantially smaller, and if someone makes a suggestion I'll try it.