Wrangling Personal Digital Assets (DAM)

One of the side effects of our times is the amount of digital content we produce. Everyday we're increasingly prone to taking pictures and videos with our pocket computers (read: currently iPhone 4S). Blink, and suddenly there are 12 years and thousands of media files (> 20GB) scattered across mismatched folders, each one with a late attempt at organizing it by event, year, people, etc. Trying to find something becomes a challenge, and grouping things into albums is something that cannot easily be done without much time and replicating pictures (and wasting disk space).

This is something that was weighing me down for a while. I knew that I had to spend some hours researching the best way manage digital media, and then process my files while making sure I didn't miss anything important I would later regret. However, finding the uninterrupted time to do this was a luxury I didn't have until recently.

I broke down the process into the following areas:

  • File organization in the filesystem
    • Folder and file naming convention
  • Migrating media folders and files
    • Ensuring no file loss due to automated migration due to multiple extensions, etc
    • Fixing filename consistency (all lower case)
  • Using a media database application to browse and easily process files
    • Picasa vs. iPhoto vs. ?
  • Defining a process to import media from devices to maintain organization/consistency

File Organization & Media Migration

After a bit of researching and thinking about it, it makes sense to organize files in a chronological folder structure; year, month, day, with each day folder housing the actual media
I found a great article which shared these views, and also pointed me to a brilliant little tool called DIM (Digital Image Mover).

DIM is dead simple to use, and I only made some minor tweaks to get it going. One warning is to do some testing to make sure it's behaving as intended. By default, it only processes JPGs and MOV files. Therefore, you have to specify which file extensions it should process, which becomes a daunting tasks with this amount of data. This is where command line goodness comes into play. Lastly, I wanted to have naming consistency across files, for which I used some more command line (perl).

The following are the steps I took to organize the files into the desired folder and naming conventions:

  1. Move all the images/folders into a single parent folder. Really, just throw them all in there
    1. I chose ~/Pictures/old as the starting bucket for all the media
    2. I created ~/Pictures/new as the destination for the processed media
  2. To determine all the file extensions that I cared about, I ran the following command:
    1. find ~/Pictures/old -type f | grep -oe "\.[a-zA-Z]\{3\}$" > /tmp/extentions; sort /tmp/extentions | uniq
  3. I noticed that DIM did not standardize the case of the output. Rather, it retained the case of the filename and extension. This meant that some files would be *.jpg and others *.JPG. My semi-ocd nature decided to kick in, so I did the following
    1. Tried to use the rename Perl utility, but it turns out that OSX Lion does not have this built in as does Linux. This is where brew comes in (if you don't know about brew, check this out) - to install:
    2. brew install rename
    3. With tool in hand, it was now time to rename all the files with the following command
    4. rename -cf `find ~/Pictures/old -type f`
    5. Note: I actually had to run this command multiple times because there were too many files, however, it only took me a few tries to get them all
    6. Done - filename case consistency goodness
  4. Next, it was time to use some DIM magic
    1. Non-standard configurations used:
    2. Day subdirectory name pattern: %D_%M_%Y
    3. Renamed file pattern: %D_%M_%Y-%H%m%S
    4. File types to process (from above): jpg;mov;avi;wmv;tif
    5. Select source and destination paths
    6. Process files Success!

Media Database Application - Picasa Wins

Now that the folders and files were under control, I had to find a good way to interact with them. I first tried to use the native application that comes with the Mac, iPhoto. Wow... I really dislike it. Namely, it wants to import all the files and abstract them from the filesystem. That means that you can only interact with them through the app itself, and anything you do from another app/command line runs the risk of corrupting data. This also means that you duplicate all your files; this becomes expensive when we're talking about gigabytes of data. I did find that there's a way for it to not import images, and just use links, but reviews and documentation on that functionality were poor at best.

I remembered liking Picasa from my Linux desktop using days, so I started going down that road. To my surprise, it's come a long way in terms of functionality. What's more, I had already decided that I was going to put my shared galleries on Google (Picasaweb, Google+, all the same now - more on the online experience later), and Picasa desktop provides a simple interface to create albums, upload them, and share them. As a bonus, you can also upload videos to Youtube from Picasa. Success!

One thing I don't like about Picasa is that you don't have an initial choice to tell it what to import. It has two options: Scan Desktop, Pictures, and a couple other folders; or scan the whole computer. This means that you will have a lot of images that you may not want in the database. However, this was a minor irritation. I selected the former option, and manually deselected the unwanted folders from the application (best way: File --> Add Folder to Picasa --> Deselect folders). Once I only had the ~/Pictures folder selected, Picasa processed all the images, and I could start playing with the media.

As a side note, I had to rebuild the Picasa database a couple of times. To do this, delete the following folder:

~/Library/Application Support/Google/Picasa3/db3/

Note: Careful; doing this will delete all the albums

This is where the fun begins. I was now able to easily play with the files, group them into albums, share them, relive memories. This is where the chore turned into a game I wanted to play.

Repeatable Process Going Forward

Great. I managed to take control the files, and found a great way to play with my media. Now, I just have to make sure that I can do this in a repeatable way. What is the best way to process files from a device (camera drive, iPhone, etc)? These are my initial thoughts:

  1. Copy the files to a temporary files outside of the ~/Pictures directory (so that Picasa does not automatically scan them in)
  2. Standardize the filenames using the command line tool above
  3. Use DIM to process the files (rename and place in correct folders) from the temporary location

Hopefully writing this helps to keep me honest. I'd be curious to know if there are better ways to do this. Comments appreciated.

comments powered by Disqus