the bits

There is some new glue in Lion Server to help manage infrastructure sharing between services, mostly targeted at web-related services referred to as web apps. This post aims to shed a bit of light on the ‘web apps’ layer to see how it interacts with the server admin tools and with launchd.

Much of what you need to know can be found in the man pages for webappctl and webapp.plist (the man page links only work on Lion Server). I won’t repeat what’s already there; go read them ;)

Going backwards

First, let’s take a look at a fairly vanilla config to demonstrate how to work backwards from a webapp configuration to find who / what it is. On a server with no (visibly configured) services enabled, let’s use webappctl to get a list of the web apps that are loaded:

bash-3.2# webappctl status -
web:webAppState:_array_index:0:state = "RUNNING"
web:webAppState:_array_index:0:virtualHostName = ""
web:webAppState:_array_index:0:webAppName = "com.apple.webapp.ACSServer"

What is this ACSServer? Looking at the com.apple.webapp.ACSServer.plist, we see:

bash-3.2# /usr/libexec/PlistBuddy -c print \
/etc/apache2/webapps/com.apple.webapp.ACSServer.plist
Dict {
    name = com.apple.webapp.ACSServer
    proxies = Dict {
        /AccountsConfigService/api/ = Dict {
            urls = Array {
                http://localhost:31415/AccountsConfigService/api
            }
            path = /AccountsConfigService/api/
        }
    }
    sslPolicy = 1
    requiredModuleNames = Array {
        mod_rewrite.so
    }
    includeFiles = Array {
        /etc/apache2/httpd_ACSServer.conf
    }
    launchKeys = Array {
        com.apple.AccountsConfigService
    }
}

The “launchKeys” entry specifies the name of the associated launched job, com.apple.AccountsConfigService, so let’s look at that:

bash-3.2# /usr/libexec/PlistBuddy -c print \
/System/Library/LaunchDaemons/com.apple.AccountsConfigService.plist
Dict {

    Program = /usr/libexec/scsd
    OnDemand = true
    Disabled = true
    Sockets = Dict {
        Listeners = Array {
            Dict {
                SockFamily = IPv4
                SockNodeName = 127.0.0.1
                SockServiceName = 31415
                SockType = stream
            }
        }
    }
    UserName = _scsd
    ProgramArguments = Array {
        /usr/libexec/scsd
        -l
    }
    GroupName = _scsd
    Label = com.apple.AccountsConfigService
}

… lest you think this job should actually be disabled, don’t forget about the ol’ overrides file. Let’s query it to see if this job’s state is overridden there:

bash-3.2# /usr/libexec/PlistBuddy -c "print :com.apple.AccountsConfigService" \
/var/db/launchd.db/com.apple.launchd/overrides.plist

Dict {
    Disabled = false
}

We’ve determined what the associated process is (scsd) and that this webapp should be loaded based on the launchd config. Since it’s configured to load on demand, we might not expect to see the scsd process running; it’s not running on my system. There is a tiny man page for scsd. Don’t stop things you didn’t start ;)

Going forward

webappctl has a method called ‘tree’ that “displays the hierarchy of webapps declared by the requiredWebApps property.” Let’s look:

bash-3.2# webappctl trees - | fold -s

com.apple.webapp.ACSServer: com.apple.AccountsConfigService

com.apple.webapp.auth: com.apple.collabauthd

com.apple.webapp.collab: com.apple.collabcored1, com.apple.collabcored2,
com.apple.collabcored3, com.apple.collabcored4, com.apple.collabsandboxd,
com.apple.collabfeedd, com.apple.collabd, com.apple.collabd.notifications,
com.apple.collabd.quicklook, org.postgresql.postgres
. com.apple.webapp.auth: com.apple.collabauthd

com.apple.webapp.devicemgr: com.apple.devicemanager,
com.apple.DeviceManagement.SCEPHelper, org.postgresql.postgres
. com.apple.webapp.auth: com.apple.collabauthd

com.apple.webapp.mailman

com.apple.webapp.passwordreset: com.apple.passwordreset

com.apple.webapp.php: 

com.apple.webapp.podcastwikiui: com.apple.collabd.podcast-cache-updater

com.apple.webapp.webcal: com.apple.wikid.compatibility
. com.apple.webapp.auth: com.apple.collabauthd

com.apple.webapp.webcalssl: com.apple.wikid.compatibility
. com.apple.webapp.auth: com.apple.collabauthd

com.apple.webapp.webdavsharing: 

com.apple.webapp.webmailserver: com.example.placeholder, org.postgresql.postgres
. com.apple.webapp.php: 

org.calendarserver: org.calendarserver.calendarserver, org.postgresql.postgres

It might be tempting to think that the above is a complete expression of the dependency graph of related web apps. This is not true. For example, we see that org.calendarserver expresses dependencies on org.calendarserver.calendarserver and org.postgres.postgres. These two things are not web apps, but launchd jobs.

Diversion: just because you can…

So then, can we deduce that by starting the ‘org.calendarserver’ web app, we’d end up with a running calendar and postgres service? Well, we could, but we’d be wrong.

bash-3.2# webappctl start org.calendarserver
web:state = "RUNNING"

bash-3.2# webappctl status -
web:webAppState:_array_index:0:state = "RUNNING"
web:webAppState:_array_index:0:virtualHostName = ""
web:webAppState:_array_index:0:webAppName = "com.apple.webapp.ACSServer"
web:webAppState:_array_index:1:state = "RUNNING"
web:webAppState:_array_index:1:virtualHostName = ""
web:webAppState:_array_index:1:webAppName = "org.calendarserver"

Where’s postgres? What gives?

bash-3.2# tail -n 4 /var/log/system.log | cut -d' ' -f6-20
Reading configuration from file: /etc/caldavd/caldavd.plist
Neither EnableCalDAV nor EnableCardDAV are set to True.
(org.calendarserver.calendarserver[11258]): Exited with code: 1
(org.calendarserver.calendarserver): Throttling respawn: Will start in 59 seconds

Not so fast, smart guy. What makes you think you can just go around using new and mostly unknown tools for starting and stopping services, without being given express permission?! Don’t tell me you read it on the Internet. The point is that things are becoming more complex, and you really need to use the highlest-level interface available for doing things. In this case, that means Server.app or the serveradmin command line tool. In this specific case, the same Calendar Server software can provide both CalDAV (for calendaring) and CardDAV (for address book) services, but each is only enabled when they are configured to be enabled (in /etc/caldavd/caldavd.plist). It is only through Server.app / serveradmin that these config keys get enabled prior to loading the launchd job.

What about that dep graph again?

Now that you are sufficiently chastised, let’s get back to our example and see what happens when we start calendar server the right way, after first undoing the damage from earlier:

bash-3.2# webappctl stop org.calendarserver
web:state = "STOPPED"

bash-3.2# serveradmin start calendar
calendar:state = "RUNNING"
calendar:setStateVersion = 1
calendar:readWriteSettingsVersion = 1

bash-3.2# webappctl status -
web:webAppState:_array_index:0:state = "RUNNING"
web:webAppState:_array_index:0:virtualHostName = ""
web:webAppState:_array_index:0:webAppName = "com.apple.webapp.ACSServer"
web:webAppState:_array_index:1:state = "RUNNING"
web:webAppState:_array_index:1:virtualHostName = ""
web:webAppState:_array_index:1:webAppName = "com.apple.webapp.webcal"
web:webAppState:_array_index:2:state = "RUNNING"
web:webAppState:_array_index:2:virtualHostName = ""
web:webAppState:_array_index:2:webAppName = "com.apple.webapp.auth"
web:webAppState:_array_index:3:state = "RUNNING"
web:webAppState:_array_index:3:virtualHostName = ""
web:webAppState:_array_index:3:webAppName = "org.calendarserver"

Who said anything about com.apple.webapp.auth or com.apple.webapp.webcal, and where is postgres? Well, first of all remember that postgres won’t show up here, because ‘org.postgresql.postgres’ refers to a launchd job, not a web app. If we check launchd and a process listing, we see that postgres is indeed running:

bash-3.2# ps ax | grep -i post | fold -s
11452   ??  Ss     0:00.15 /usr/bin/postgres -D /var/pgsql -c listen_addresses=
-c log_connections=on -c log_directory=/Library/Logs -c
log_filename=PostgreSQL.log -c log_lock_waits=on -c log_statement=ddl -c
log_line_prefix=%t  -c logging_collector=on -c
unix_socket_directory=/var/pgsql_socket -c unix_socket_group=_postgres -c
unix_socket_permissions=0770
11459   ??  Ss     0:00.08 postgres: logger process
11461   ??  Ss     0:00.29 postgres: writer process
11462   ??  Ss     0:00.22 postgres: wal writer process
11463   ??  Ss     0:00.16 postgres: autovacuum launcher process
11464   ??  Ss     0:00.35 postgres: stats collector process
11837 s001  R+     0:00.00 grep -i post

bash-3.2# launchctl list | grep -i postgres
11452	-	org.postgresql.postgres

Ok, but what about the ‘auth’ and ‘webcal’ web apps? Well, calendar server is cool and has a nifty log that keeps track of what happens through its servermgrd bundle. Let’s see if it’s doing any webcal stuff:

bash-3.2# grep webcal /var/log/caldavd/servermgr_calendar.log | tail -n 1
[cal] 7/24/11 2:48:15 PM PDT : Changing com.apple.webapp.webcal WebAppState to START

We could then look at the webcal webapp plist to see that it also starts the ‘auth’ one. Whew, ok. Maybe you liked it better when it was mysterious and magical :)

The take home point here is that services can and do directly start web apps on their own. The policies controlling this behavior may be codified in the app, so you should have no expectation of being able to directly lay eyes on these policies, although you can get a good feel for it through experimentation. It’s probably a good idea not to use webappctl to twiddle the state of any web apps that you didn’t create, lest you invalidate somebody else’s expectations.

Following Apple’s announcement of the XServe’s cancellation yesterday, there has been an expected amount of chatter revolving around Apple’s server-oriented efforts in general, and also specifically about Apple’s software and hardware offerings. As somebody who has lived and worked in the Apple IT world for a good while, here are some thoughts.

Over the years, there have been (in my opinion) two primary use cases for Mac OS X Server which both stem from the same underlying concept: any place where there are more than a few macs, those macs can benefit from some form of centralized binding element. The more macs there are, the greater the possible benefit. The two use cases I see are: 1) to provide a turnkey client management solution, and 2) to provide turnkey services. I hate the word ‘turnkey’ cause it sounds blingy, but in this case I mean it: the ratio of ‘ease of use’ to ‘features provided’ is definitely the key metric, and the key ‘win’ for OSXS.

The management use case has opened up a lot since ye old days, and now there are a variety of ways to perform client management that don’t even require Mac OS X Server. However, I would still argue (tentatively, as somebody who doesn’t do a lot of client management these days) that OSXS still provides the best ratio of ease of use to features provided, for client management. The need for client management absolutely still exists, and now there are more options. This is a good thing.

The ‘services’ side is more significant now than it used to be, because it wasn’t until the more recent server releases that OSXS provided services that really broke through and attained critical mass (this is mostly gauged by observing the mac IT community and its users). In my opinion, it’s not that Apple makes the best single wiki, calendar, mail, file, podcast production(, …) service, but that they are pretty good, integrated in one box, and … sigh… ‘turnkey’.

Time is money. Money is money, too. It’s pretty easy for a geek to be all like “pshaw, I don’t need all the autoconfiguration glue crutch annoyance water wing bumper car stuff that you call ‘added value’, I has the power of open source!”, and then proceed to provide all the solutions that are needed using a potpourri of software and perhaps even hardware. Usually, the skilled open source person wins in terms of the ‘absolute value’ of the features provided, and pretty much always loses in terms of the ratio of ease of use (from the implementor’s perspective) to features provided. “Ease of use” is important because it expands to “time, money”, but it’s easy to overlook as a skilled (especially open source) professional, because “easy” is a relative concept, and because *time* is a relative concept. Geeks might not mind plugging away at ${annoying_technical_challenge} all day long, where as most normal people would never think of it.

Of course, it’s possible to get off into the weeds with just about anything related to computers – and this certainly includes OSXS. However, with the right OSXS approach, it’s absolutely possible to deploy and maintain useful solutions, and only expend minimal time and money in the process. Many OSXS sites fit this description exactly, and many OSXS consultants make their living based on this concept.

I used to run my own private instances of all the network services I cared about, on a FreeBSD colo box on which I had root. It was fun for a while, and a great learning experience. Eventually it turned from fun to janitorial in nature – that’s when I got my DreamHost account, and I haven’t looked back. I now prefer the solution that conserves time and money while, and I still think that’s the primary win for OSXS, for its target audience.

The XServe will be missed, though. It’s some kickass hardware. For the people who actually have racks of XServes, eventually something will have to give. It’s simple physics :) Mac sales are only going up, so the big picture isn’t a lot different than it was last week in terms of *needs*. In terms of *options*, it is different. Just as the death of Apple’s XServe RAID did not spell the death of direct-attached storage for Macs, I’m hoping the same will be true for high-density datacenter-grade OSXS deployments. I’m as curious as anybody to see how this plays out…