Permissions updated…

Spambots have been running wild lately, so I’ve tightened this site a little. To post a comment, users must be logged in and must have a previously approved comment – otherwise the comment goes into moderation queue.


They really went crazy for reals, so I added a captcha for registration, and also mass-deleted hundreds of bogus accounts. I’m pretty sure I deleted some legit ones as well, as there’s no easy means for distinguishing them in the wordpress user list… so sorry, babies :/

Posted in bit bucket | Leave a comment

How I fixed broken internal audio by zapping pram

Oops. I think I stole my own thunder. It was a serious wtf moment though… by all rights, I really don’t think this is supposed to happen. The worst part of this experience is not the increasingly frantic hour of me trying all the ‘right’ things to fix my apparently dead internal audio, failing, and then feeling queazy about the prospect of having to get a new mac pro right now. That was uncomfortable, but temporary. The worst part is that I can no longer mock people who zap pram in attempts to fix random problems that were never intended to be fixable by a pram reset. I wonder what mysteriously beneficial effect permissions repair has…

To continue the upside-down nature of this post, I should mention that the initial damage appears to have been done by windows 7, which I installed on a dedicated internal HDD via the BootCamp Assistant. It all appeared to be going well for the first couple days (of my Tera blitz, during which I didn’t boot over to OS X even once), but then I bounced back to OS X and was stymied by both a much longer than average boot time, and also… no internal audio. Apple System Profiler didn’t see much of anything:

system profiler window showing a basically empty window where the details of the internal audio device should be

I did several normal reboots, shutdown –> boot cycles, disconnection of various extra hardware and peripherals, SMC reset… but nothing was helping. The first time I booted back to win7, it offered to re-install the drivers. Windows could see the internal audio device (which it believes is of the RealTek variety) and allow me to select it as the default device, but still no actual sound was emitted. I figured it was kinda toasted at that point, but then, not expecting it to work, I zapped pram, and lo and behold:

system profiler window showing normal internal audio device details

Boot time is back to normal, too. Hopefully I don’t need to add any permanent pram cheese to this bootcamp sandwich…

UPDATE 1: oh noes, this is totally reproducible. Basically every time I go from win to mac, the audio device goes away until I zap pram. a bug has been filed.

Posted in mac pro, OS X | Leave a comment

Ubuntu Development Bootstrap

Linux hackers surely already know this, but I’m only sorta a Linux hacker. This post documents a nice process for using apt to prepare an Ubuntu machine for development. In my case, I was trying to build a custom version of irssi that is patched to add SSL support to the irssi proxy module. Sure, I could have gone about this in the traditional manner, but given that irssi has a fair number of dependencies, it’s easier to let apt do the grunt work for me.

First, get the toolchain set up.

sudo apt-get install build-essential checkinstall

The build-essential and ‘check install’ targets are news to me. Build-essential, as the name suggests, includes some compilers and related tools that would be needed for building just about anything.

Checkinstall keeps track of all files installed by a “make install” or equivalent, creates a Slackware, RPM, or Debian package with those files, and adds it to the installed packages database, allowing for easy package removal or distribution.”

Next, in preparation for building irssi itself, get all the dependencies for irssi. This works because irssi is already in the apt package database, so apt knows what the dependencies are.

sudo apt-get build-dep irssi

From there, download the irssi source, patch it, and proceed to build / run / install it (perhaps installing with checkinstall so you can easily remove it later!)

p.s. this irssi proxy SSL patch isn’t working. bummer.

p.p.s stunnel works though!

Posted in development | Leave a comment

Gnuplot time-series buckets quickie

Let’s say you’ve got a log file of events with time stamps; the nature of the events isn’t relevant, and let’s say the stamps look like this:


You want to create a graph of the frequency of events in the log file, with time on the x and frequency on the y. An easy tactic here is to put the events into buckets, and then find the count of items in each bucket. If we use a bucket size of one minute, we can simply lop off the seconds field, and then use ‘uniq -c’ to produce our frequency count for each minute:

cat test | cut -d: -f 1-3 | uniq -c
 6 14/Mar/2012:17:49
 2 14/Mar/2012:17:51
 4 14/Mar/2012:17:52
 3 14/Mar/2012:17:53

Then with gnuplot:

set xdata time
set timefmt "%d/%b/%Y:%H:%M"
plot 'events_per_minute.txt' using 2:1 title "events per minute" with impulses
Posted in scripts | Leave a comment

The case of the excessively congratulatory Memory Slot Utility

A while ago I filled up my final unused memory slot in my awesome Mac Pro with an additional 4 gigorams (for a grand total of 16). Since then, whenever I reboot I’m showered with praise by the Memory Slot Utility for a job well done. This made me feel pretty good for the first couple months, but eventually it got old and I was ready to say goodbye for the last time. Googling around, I found the usual colorful variety of suggestions, ranging from “do a permissions repair” to “reinstall Mac OS X” to “reformat your hard drive”. Manual invocations of Memory Slot Utility as root (e.g. sudo open -a Memory\ Slot\ didn’t help, although it was obvious that it needed to twiddle some bit somewhere, and my hunch was that this was most likely failing for authorization reasons.

The winning move turned out to be logging in briefly as root. Generally not a good idea to log in as root, but in this case it’s the easiest way to clear this condition. All you need to do is enable the root account by setting a password for it, and then log in as root from the login window.

  1. In Finder, choose Go –> Go to Folder… (or shift command g), and enter /System/Library/CoreServices
  2. In the CoreServices folder, open Directory Utility.
  3. From the Edit menu, choose “Enable Root User”, and supply a password.
  4. Open System Preferences –> Users & Groups prefpane, and under Login Options, turn off Automatic Login, and set the login window display to Name and password.
  5. Reboot. Log in as root from the login window. Perform one final heroic click of Memory Slot Utility’s OK button, going for maximum style points.
  6. Log out, log back in as yourself, put settings back the way you like them. I suggest using Directory Utility to disable the root account.
Posted in mac pro, OS X, Pro Tip, The More You Know | 1 Comment


I only worked under Steve’s leadership for a few years, and never spoke to him in person. Even so, he earned my respect for thinking big and really wanting people to *understand* what he was after. The technology is definitely just a means to an end.

I admire his dedication to building tools, both for developers and end-users. It’s pretty central for us as a species, and it’s been fascinating to see how such tools enable people to communicate better, find information better, organize thoughts better… all of it. Not to have cool gadgets, but to be empowered with abilities that would be impossible without the gadgets. And yeah, if you’re going to do gadgets, might as well make them beautiful.

For a while I was pretty conflicted about Apple’s direction with iOS and the conclusions one might draw about how it might affect the future of Mac OS X. I felt like my beloved computing environment was about to get dumbed down, and my concerns marginalized as an edge case. I still feel this way, but in the last year or so I’ve come to understand (and observe) that it’s totally worth it. Making an iPhone this easy to use means it gets *much* more widespread use than something else that might be more appealing (at least, at first blush) to geeks. The overall value of countless computer illiterates gaining access to iOS is surely far greater than the cost of some graybeards occasionally yelling about stuff on their lawns. My desire for people to be able to handle larger amounts of complexity in their computing devices seems completely selfish in retrospect.

So thanks, Steve, for the relentless focus on the ‘high order bit’. Your presence will be missed, but your message not forgotten.

Posted in bit bucket | Leave a comment

Exploring web apps in Lion Server

There is some new glue in Lion Server to help manage infrastructure sharing between services, mostly targeted at web-related services referred to as web apps. This post aims to shed a bit of light on the ‘web apps’ layer to see how it interacts with the server admin tools and with launchd.

Much of what you need to know can be found in the man pages for webappctl and webapp.plist (the man page links only work on Lion Server). I won’t repeat what’s already there; go read them ;)

Going backwards

First, let’s take a look at a fairly vanilla config to demonstrate how to work backwards from a webapp configuration to find who / what it is. On a server with no (visibly configured) services enabled, let’s use webappctl to get a list of the web apps that are loaded:

bash-3.2# webappctl status -
web:webAppState:_array_index:0:state = "RUNNING"
web:webAppState:_array_index:0:virtualHostName = ""
web:webAppState:_array_index:0:webAppName = ""

What is this ACSServer? Looking at the, we see:

bash-3.2# /usr/libexec/PlistBuddy -c print \
Dict {
    name =
    proxies = Dict {
        /AccountsConfigService/api/ = Dict {
            urls = Array {


            path = /AccountsConfigService/api/
    sslPolicy = 1
    requiredModuleNames = Array {
    includeFiles = Array {
    launchKeys = Array {

The “launchKeys” entry specifies the name of the associated launched job,, so let’s look at that:

bash-3.2# /usr/libexec/PlistBuddy -c print \
Dict {
    Program = /usr/libexec/scsd
    OnDemand = true
    Disabled = true
    Sockets = Dict {
        Listeners = Array {
            Dict {
                SockFamily = IPv4
                SockNodeName =
                SockServiceName = 31415
                SockType = stream
    UserName = _scsd
    ProgramArguments = Array {
    GroupName = _scsd
    Label =

… lest you think this job should actually be disabled, don’t forget about the ol’ overrides file. Let’s query it to see if this job’s state is overridden there:

bash-3.2# /usr/libexec/PlistBuddy -c "print" \
Dict {
    Disabled = false

We’ve determined what the associated process is (scsd) and that this webapp should be loaded based on the launchd config. Since it’s configured to load on demand, we might not expect to see the scsd process running; it’s not running on my system. There is a tiny man page for scsd. Don’t stop things you didn’t start ;)

Going forward

webappctl has a method called ‘tree’ that “displays the hierarchy of webapps declared by the requiredWebApps property.” Let’s look:

bash-3.2# webappctl trees - | fold -s,,,,,,,,, org.postgresql.postgres
.,, org.postgresql.postgres
. com.example.placeholder, org.postgresql.postgres

org.calendarserver: org.calendarserver.calendarserver, org.postgresql.postgres

It might be tempting to think that the above is a complete expression of the dependency graph of related web apps. This is not true. For example, we see that org.calendarserver expresses dependencies on org.calendarserver.calendarserver and org.postgres.postgres. These two things are not web apps, but launchd jobs.

Diversion: just because you can…

So then, can we deduce that by starting the ‘org.calendarserver’ web app, we’d end up with a running calendar and postgres service? Well, we could, but we’d be wrong.

bash-3.2# webappctl start org.calendarserver
web:state = "RUNNING"
bash-3.2# webappctl status -
web:webAppState:_array_index:0:state = "RUNNING"
web:webAppState:_array_index:0:virtualHostName = ""
web:webAppState:_array_index:0:webAppName = ""
web:webAppState:_array_index:1:state = "RUNNING"
web:webAppState:_array_index:1:virtualHostName = ""
web:webAppState:_array_index:1:webAppName = "org.calendarserver"

Where’s postgres? What gives?

bash-3.2# tail -n 4 /var/log/system.log | cut -d' ' -f6-20
Reading configuration from file: /etc/caldavd/caldavd.plist
Neither EnableCalDAV nor EnableCardDAV are set to True.
(org.calendarserver.calendarserver[11258]): Exited with code: 1
(org.calendarserver.calendarserver): Throttling respawn: Will start in 59 seconds

Not so fast, smart guy. What makes you think you can just go around using new and mostly unknown tools for starting and stopping services, without being given express permission?! Don’t tell me you read it on the Internet. The point is that things are becoming more complex, and you really need to use the highlest-level interface available for doing things. In this case, that means or the serveradmin command line tool. In this specific case, the same Calendar Server software can provide both CalDAV (for calendaring) and CardDAV (for address book) services, but each is only enabled when they are configured to be enabled (in /etc/caldavd/caldavd.plist). It is only through / serveradmin that these config keys get enabled prior to loading the launchd job.

What about that dep graph again?

Now that you are sufficiently chastised, let’s get back to our example and see what happens when we start calendar server the right way, after first undoing the damage from earlier:

bash-3.2# webappctl stop org.calendarserver
web:state = "STOPPED"
bash-3.2# serveradmin start calendar
calendar:state = "RUNNING"
calendar:setStateVersion = 1
calendar:readWriteSettingsVersion = 1
bash-3.2# webappctl status -
web:webAppState:_array_index:0:state = "RUNNING"
web:webAppState:_array_index:0:virtualHostName = ""
web:webAppState:_array_index:0:webAppName = ""
web:webAppState:_array_index:1:state = "RUNNING"
web:webAppState:_array_index:1:virtualHostName = ""
web:webAppState:_array_index:1:webAppName = ""
web:webAppState:_array_index:2:state = "RUNNING"
web:webAppState:_array_index:2:virtualHostName = ""
web:webAppState:_array_index:2:webAppName = ""
web:webAppState:_array_index:3:state = "RUNNING"
web:webAppState:_array_index:3:virtualHostName = ""
web:webAppState:_array_index:3:webAppName = "org.calendarserver"

Who said anything about or, and where is postgres? Well, first of all remember that postgres won’t show up here, because ‘org.postgresql.postgres’ refers to a launchd job, not a web app. If we check launchd and a process listing, we see that postgres is indeed running:

bash-3.2# ps ax | grep -i post | fold -s
11452   ??  Ss     0:00.15 /usr/bin/postgres -D /var/pgsql -c listen_addresses=
-c log_connections=on -c log_directory=/Library/Logs -c
log_filename=PostgreSQL.log -c log_lock_waits=on -c log_statement=ddl -c
log_line_prefix=%t  -c logging_collector=on -c
unix_socket_directory=/var/pgsql_socket -c unix_socket_group=_postgres -c
11459   ??  Ss     0:00.08 postgres: logger process
11461   ??  Ss     0:00.29 postgres: writer process
11462   ??  Ss     0:00.22 postgres: wal writer process
11463   ??  Ss     0:00.16 postgres: autovacuum launcher process
11464   ??  Ss     0:00.35 postgres: stats collector process
11837 s001  R+     0:00.00 grep -i post
bash-3.2# launchctl list | grep -i postgres
11452	-	org.postgresql.postgres

Ok, but what about the ‘auth’ and ‘webcal’ web apps? Well, calendar server is cool and has a nifty log that keeps track of what happens through its servermgrd bundle. Let’s see if it’s doing any webcal stuff:

bash-3.2# grep webcal /var/log/caldavd/servermgr_calendar.log | tail -n 1
[cal] 7/24/11 2:48:15 PM PDT : Changing WebAppState to START

We could then look at the webcal webapp plist to see that it also starts the ‘auth’ one. Whew, ok. Maybe you liked it better when it was mysterious and magical :)

The take home point here is that services can and do directly start web apps on their own. The policies controlling this behavior may be codified in the app, so you should have no expectation of being able to directly lay eyes on these policies, although you can get a good feel for it through experimentation. It’s probably a good idea not to use webappctl to twiddle the state of any web apps that you didn’t create, lest you invalidate somebody else’s expectations.

Posted in OS X Server | 4 Comments

Mailman members only

Recently I had occasion to cook up some glue for a Mailman deployment that wanted a privacy and membership management policy that would not be facilitated by the Mailman mailing list software in its current form. We wanted to host a private list (disabling web subscriptions), and we wanted to allow any member to invite new members.

The solution comes in a few pieces:

  • a python script to synchronize the Mailman username / password credentials to an AuthUserFile that can be used by Apache to authorize access to a cgi that wraps the mailman add_member cli tool. Find the source here: Put this in root’s crontab at some reasonable interval.
  • the aforementioned script that wraps the add_member tool. Find the source here:
  • this guy’s modified add_members cli tool. The add_members tool supplied by Mailman bypasses confirmation, but we want invitees to respond to a confirmation message before being added. This script makes that happen.
  • Disable web subscriptions by deleting stuff from /etc/mailman/en/listinfo.html
Posted in development, scripts | Leave a comment

ATI 5870 now available!

and there was much rejoicing.

Posted in bit bucket | Leave a comment

The Server Solution

Following Apple’s announcement of the XServe’s cancellation yesterday, there has been an expected amount of chatter revolving around Apple’s server-oriented efforts in general, and also specifically about Apple’s software and hardware offerings. As somebody who has lived and worked in the Apple IT world for a good while, here are some thoughts.

Over the years, there have been (in my opinion) two primary use cases for Mac OS X Server which both stem from the same underlying concept: any place where there are more than a few macs, those macs can benefit from some form of centralized binding element. The more macs there are, the greater the possible benefit. The two use cases I see are: 1) to provide a turnkey client management solution, and 2) to provide turnkey services. I hate the word ‘turnkey’ cause it sounds blingy, but in this case I mean it: the ratio of ‘ease of use’ to ‘features provided’ is definitely the key metric, and the key ‘win’ for OSXS.

The management use case has opened up a lot since ye old days, and now there are a variety of ways to perform client management that don’t even require Mac OS X Server. However, I would still argue (tentatively, as somebody who doesn’t do a lot of client management these days) that OSXS still provides the best ratio of ease of use to features provided, for client management. The need for client management absolutely still exists, and now there are more options. This is a good thing.

The ‘services’ side is more significant now than it used to be, because it wasn’t until the more recent server releases that OSXS provided services that really broke through and attained critical mass (this is mostly gauged by observing the mac IT community and its users). In my opinion, it’s not that Apple makes the best single wiki, calendar, mail, file, podcast production(, …) service, but that they are pretty good, integrated in one box, and … sigh… ‘turnkey’.

Time is money. Money is money, too. It’s pretty easy for a geek to be all like “pshaw, I don’t need all the autoconfiguration glue crutch annoyance water wing bumper car stuff that you call ‘added value’, I has the power of open source!”, and then proceed to provide all the solutions that are needed using a potpourri of software and perhaps even hardware. Usually, the skilled open source person wins in terms of the ‘absolute value’ of the features provided, and pretty much always loses in terms of the ratio of ease of use (from the implementor’s perspective) to features provided. “Ease of use” is important because it expands to “time, money”, but it’s easy to overlook as a skilled (especially open source) professional, because “easy” is a relative concept, and because *time* is a relative concept. Geeks might not mind plugging away at ${annoying_technical_challenge} all day long, where as most normal people would never think of it.

Of course, it’s possible to get off into the weeds with just about anything related to computers – and this certainly includes OSXS. However, with the right OSXS approach, it’s absolutely possible to deploy and maintain useful solutions, and only expend minimal time and money in the process. Many OSXS sites fit this description exactly, and many OSXS consultants make their living based on this concept.

I used to run my own private instances of all the network services I cared about, on a FreeBSD colo box on which I had root. It was fun for a while, and a great learning experience. Eventually it turned from fun to janitorial in nature – that’s when I got my DreamHost account, and I haven’t looked back. I now prefer the solution that conserves time and money while, and I still think that’s the primary win for OSXS, for its target audience.

The XServe will be missed, though. It’s some kickass hardware. For the people who actually have racks of XServes, eventually something will have to give. It’s simple physics :) Mac sales are only going up, so the big picture isn’t a lot different than it was last week in terms of *needs*. In terms of *options*, it is different. Just as the death of Apple’s XServe RAID did not spell the death of direct-attached storage for Macs, I’m hoping the same will be true for high-density datacenter-grade OSXS deployments. I’m as curious as anybody to see how this plays out…

Posted in OS X Server | Leave a comment