{"id":73,"date":"2008-10-27T22:52:16","date_gmt":"2008-10-28T05:52:16","guid":{"rendered":"http:\/\/www.dreness.com\/blog\/?p=73"},"modified":"2019-12-07T17:10:32","modified_gmt":"2019-12-08T00:10:32","slug":"cross-application-url-cache-searching","status":"publish","type":"post","link":"https:\/\/dreness.com\/blog\/archives\/73","title":{"rendered":"Cross-application URL cache searching!"},"content":{"rendered":"

I was out taking a (filesystem) walk, and wandered into \/var\/folders. Not remembering where \/ when this came from, I was pleased to find that the hier(7) manpage<\/a> is aware of this directory, describing it as ‘per-user temporary files and caches’. More research reveals that this directory structure is created as defined by some code in Apple’s Libc <\/a>as of Leopard. Some of the constants are defined in dirhelper_priv.h<\/a>, and the corresponding _dirhelper.c<\/a> does the creating and locating of directories – note the fancy two-level hashing.<\/p>\n

Walking around more, I find that I’ve got a directory for myself (as observed via the file permissions) with the following contents:<\/p>\n

drwx------  35 andre  staff  1190 Oct 27 19:34 -Caches-\r\ndrwx------  12 andre  staff   408 Oct 27 21:03 -Tmp-\r\ndrwxrwxrwx@  3 andre  staff   102 Oct 27 20:42 Cleanup At Startup\r\ndrwxr-xr-x   2 andre  staff    68 Oct 27 19:26 TemporaryItems<\/pre>\n
The -Caches- folder contains application directories in the reverse domain notation, e.g. com.apple.Safari, or com.apple.mail or com.blizzard.worldofwarcraft. All of these directories seem to have at least one file: Cache.db. hmm… ok, database. fun. Let’s see what’s in it!<\/p>\n
{56} andre@gyro [-Caches-\/com.apple.Safari] % file Cache.db\r\nCache.db: SQLite database (Version 3)<\/pre>\n
Cool! I can do sqlite3…<\/p>\n
{74} andre@gyro [-Caches-\/com.apple.Safari] % sqlite3 Cache.db\r\nSQLite version 3.4.0\r\nEnter \".help\" for instructions\r\nsqlite> .tables\r\ncfurl_cache_blob_data\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 cfurl_cache_schema_version\r\ncfurl_cache_response\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\r\nsqlite> .schema cfurl_cache_response<\/pre>\n
(this produces output with really long lines, so let’s hop back to the shell right quick)<\/p>\n
sqlite3 Cache.db \".schema cfurl_cache_response\" | fold -s\r\nCREATE TABLE cfurl_cache_response(entry_ID INTEGER PRIMARY KEY AUTOINCREMENT\r\nUNIQUE, \t\t     version INTEGER, hash_value INTEGER,\r\nstorage_policy INTEGER, request_key TEXT UNIQUE, \t\t     time_stamp\r\nNOT NULL DEFAULT CURRENT_TIMESTAMP);\r\nCREATE INDEX request_key_index ON cfurl_cache_response(request_key);\r\nCREATE INDEX time_stamp_index ON cfurl_cache_response(time_stamp);<\/pre>\n
ok, back in sqlite3 interactive mode now:<\/p>\n
sqlite> .mode line\r\nsqlite> select * from cfurl_cache_response where entry_ID=1;\r\n      entry_ID = 1\r\n       version = 0\r\n    hash_value = -1475899696\r\nstorage_policy = 0\r\n   request_key = http:\/\/i.cdn.turner.com\/cnn\/.element\/css\/2.0\/pgaleader.css\r\n    time_stamp = 2008-10-28 01:55:10\r\nsqlite>\r\nsqlite> select * from cfurl_cache_response where entry_ID<4;\r\n      entry_ID = 1\r\n       version = 0\r\n    hash_value = -1475899696\r\nstorage_policy = 0\r\n   request_key = http:\/\/i.cdn.turner.com\/cnn\/.element\/css\/2.0\/pgaleader.css\r\n    time_stamp = 2008-10-28 01:55:10\r\n\r\n      entry_ID = 2\r\n       version = 0\r\n    hash_value = -763438389\r\nstorage_policy = 0\r\n   request_key = http:\/\/i.cdn.turner.com\/cnn\/.element\/css\/2.0\/main.css\r\n    time_stamp = 2008-10-28 01:55:10\r\n\r\n      entry_ID = 3\r\n       version = 0\r\n    hash_value = 1822950361\r\nstorage_policy = 0\r\n   request_key = https:\/\/www.cnn.com\/\r\n    time_stamp = 2008-10-28 01:55:10<\/pre>\n
Neat! Ok ok, here’s the cross-app url cache searching code:<\/p>\n
#!\/bin\/bash\r\n# Here's a little ditty that does keyword searching across all the URLs in your\r\n# \/var\/folders caches directory - the one returned by\r\n# \/usr\/bin\/getconf DARWIN_USER_CACHE_DIR\r\n\r\n# Search term comes in as first command line argument\r\n# -v is an optional second argument to list databases are they are searched\r\nSEARCH=$1\r\nVERBOSE=0\r\n\r\nif [ ! $SEARCH ]\r\nthen echo \"Usage: $0 [search term] [-v]\"\r\nexit\r\nfi\r\n\r\nif ( [ -n \"$2\" ] && [ $2 == \"-v\" ] )\r\nthen VERBOSE=1\r\nfi\r\n\r\nCACHE=`getconf DARWIN_USER_CACHE_DIR`\r\n\r\nls -1 $CACHE\/*\/Cache.db | while read db\r\n\tdo\r\n\techo -n \"\"\r\n\tif [ $VERBOSE == \"1\" ] ; then echo \"   Searching $db\" ; fi\r\n\tsqlite3 $db \"select request_key from cfurl_cache_response where \\\r\n\t\trequest_key LIKE'%$SEARCH%';\"\r\n\tdone<\/pre>\n
Probably safe to not depend on any of this being available or staying the way it is in future versions of Mac OS X… but for now at least, it’s here and contains data. In case you were wondering, Safari doesn’t log here when private browsing is enabled :)<\/p>\n","protected":false},"excerpt":{"rendered":"
I was out taking a (filesystem) walk, and wandered into \/var\/folders. Not remembering where \/ when this came from, I was pleased to find that the hier(7) manpage is aware of this directory, describing it as ‘per-user temporary files and … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,6,13],"tags":[],"class_list":["post-73","post","type-post","status-publish","format-standard","hentry","category-os-x","category-scripts","category-the-more-you-know"],"_links":{"self":[{"href":"https:\/\/dreness.com\/blog\/wp-json\/wp\/v2\/posts\/73","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dreness.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dreness.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dreness.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dreness.com\/blog\/wp-json\/wp\/v2\/comments?post=73"}],"version-history":[{"count":4,"href":"https:\/\/dreness.com\/blog\/wp-json\/wp\/v2\/posts\/73\/revisions"}],"predecessor-version":[{"id":1271,"href":"https:\/\/dreness.com\/blog\/wp-json\/wp\/v2\/posts\/73\/revisions\/1271"}],"wp:attachment":[{"href":"https:\/\/dreness.com\/blog\/wp-json\/wp\/v2\/media?parent=73"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dreness.com\/blog\/wp-json\/wp\/v2\/categories?post=73"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dreness.com\/blog\/wp-json\/wp\/v2\/tags?post=73"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}