Cross-application URL cache searching!

I was out taking a (filesystem) walk, and wandered into /var/folders. Not remembering where / when this came from, I was pleased to find that the hier(7) manpage is aware of this directory, describing it as ‘per-user temporary files and caches’. More research reveals that this directory structure is created as defined by some code in Apple’s Libc as of Leopard. Some of the constants are defined in dirhelper_priv.h, and the corresponding _dirhelper.c does the creating and locating of directories – note the fancy two-level hashing.

Walking around more, I find that I’ve got a directory for myself (as observed via the file permissions) with the following contents:

drwx------  35 andre  staff  1190 Oct 27 19:34 -Caches-
drwx------  12 andre  staff   408 Oct 27 21:03 -Tmp-
drwxrwxrwx@  3 andre  staff   102 Oct 27 20:42 Cleanup At Startup
drwxr-xr-x   2 andre  staff    68 Oct 27 19:26 TemporaryItems

The -Caches- folder contains application directories in the reverse domain notation, e.g. com.apple.Safari, or com.apple.mail or com.blizzard.worldofwarcraft. All of these directories seem to have at least one file: Cache.db. hmm… ok, database. fun. Let’s see what’s in it!

{56} andre@gyro [-Caches-/com.apple.Safari] % file Cache.db
Cache.db: SQLite database (Version 3)

Cool! I can do sqlite3…

{74} andre@gyro [-Caches-/com.apple.Safari] % sqlite3 Cache.db
SQLite version 3.4.0
Enter ".help" for instructions
sqlite> .tables
cfurl_cache_blob_data       cfurl_cache_schema_version
cfurl_cache_response     
sqlite> .schema cfurl_cache_response

(this produces output with really long lines, so let’s hop back to the shell right quick)

sqlite3 Cache.db ".schema cfurl_cache_response" | fold -s
CREATE TABLE cfurl_cache_response(entry_ID INTEGER PRIMARY KEY AUTOINCREMENT
UNIQUE, 		     version INTEGER, hash_value INTEGER,
storage_policy INTEGER, request_key TEXT UNIQUE, 		     time_stamp
NOT NULL DEFAULT CURRENT_TIMESTAMP);
CREATE INDEX request_key_index ON cfurl_cache_response(request_key);
CREATE INDEX time_stamp_index ON cfurl_cache_response(time_stamp);

ok, back in sqlite3 interactive mode now:

sqlite> .mode line
sqlite> select * from cfurl_cache_response where entry_ID=1;
      entry_ID = 1
       version = 0
    hash_value = -1475899696
storage_policy = 0
   request_key = http://i.cdn.turner.com/cnn/.element/css/2.0/pgaleader.css
    time_stamp = 2008-10-28 01:55:10
sqlite>
sqlite> select * from cfurl_cache_response where entry_ID<4;
      entry_ID = 1
       version = 0
    hash_value = -1475899696
storage_policy = 0
   request_key = http://i.cdn.turner.com/cnn/.element/css/2.0/pgaleader.css
    time_stamp = 2008-10-28 01:55:10

      entry_ID = 2
       version = 0
    hash_value = -763438389
storage_policy = 0
   request_key = http://i.cdn.turner.com/cnn/.element/css/2.0/main.css
    time_stamp = 2008-10-28 01:55:10

      entry_ID = 3
       version = 0
    hash_value = 1822950361
storage_policy = 0
   request_key = https://www.cnn.com/
    time_stamp = 2008-10-28 01:55:10

Neat! Ok ok, here’s the cross-app url cache searching code:

#!/bin/bash
# Here's a little ditty that does keyword searching across all the URLs in your
# /var/folders caches directory - the one returned by
# /usr/bin/getconf DARWIN_USER_CACHE_DIR

# Search term comes in as first command line argument
# -v is an optional second argument to list databases are they are searched
SEARCH=$1
VERBOSE=0

if [ ! $SEARCH ]
then echo "Usage: $0 [search term] [-v]"
exit
fi

if ( [ -n "$2" ] && [ $2 == "-v" ] )
then VERBOSE=1
fi

CACHE=`getconf DARWIN_USER_CACHE_DIR`

ls -1 $CACHE/*/Cache.db | while read db
	do
	echo -n ""
	if [ $VERBOSE == "1" ] ; then echo "   Searching $db" ; fi
	sqlite3 $db "select request_key from cfurl_cache_response where \
		request_key LIKE'%$SEARCH%';"
	done

Probably safe to not depend on any of this being available or staying the way it is in future versions of Mac OS X… but for now at least, it’s here and contains data. In case you were wondering, Safari doesn’t log here when private browsing is enabled :)