Exploring web apps in Lion Server

There is some new glue in Lion Server to help manage infrastructure sharing between services, mostly targeted at web-related services referred to as web apps. This post aims to shed a bit of light on the ‘web apps’ layer to see how it interacts with the server admin tools and with launchd.

Much of what you need to know can be found in the man pages for webappctl and webapp.plist (the man page links only work on Lion Server). I won’t repeat what’s already there; go read them ;)

Going backwards

First, let’s take a look at a fairly vanilla config to demonstrate how to work backwards from a webapp configuration to find who / what it is. On a server with no (visibly configured) services enabled, let’s use webappctl to get a list of the web apps that are loaded:

bash-3.2# webappctl status -
web:webAppState:_array_index:0:state = "RUNNING"
web:webAppState:_array_index:0:virtualHostName = ""
web:webAppState:_array_index:0:webAppName = "com.apple.webapp.ACSServer"

What is this ACSServer? Looking at the com.apple.webapp.ACSServer.plist, we see:

bash-3.2# /usr/libexec/PlistBuddy -c print \
/etc/apache2/webapps/com.apple.webapp.ACSServer.plist
Dict {
    name = com.apple.webapp.ACSServer
    proxies = Dict {
        /AccountsConfigService/api/ = Dict {
            urls = Array {
                http://localhost:31415/AccountsConfigService/api
            }
            path = /AccountsConfigService/api/
        }
    }
    sslPolicy = 1
    requiredModuleNames = Array {
        mod_rewrite.so
    }
    includeFiles = Array {
        /etc/apache2/httpd_ACSServer.conf
    }
    launchKeys = Array {
        com.apple.AccountsConfigService
    }
}

The “launchKeys” entry specifies the name of the associated launched job, com.apple.AccountsConfigService, so let’s look at that:

bash-3.2# /usr/libexec/PlistBuddy -c print \
/System/Library/LaunchDaemons/com.apple.AccountsConfigService.plist
Dict {

    Program = /usr/libexec/scsd
    OnDemand = true
    Disabled = true
    Sockets = Dict {
        Listeners = Array {
            Dict {
                SockFamily = IPv4
                SockNodeName = 127.0.0.1
                SockServiceName = 31415
                SockType = stream
            }
        }
    }
    UserName = _scsd
    ProgramArguments = Array {
        /usr/libexec/scsd
        -l
    }
    GroupName = _scsd
    Label = com.apple.AccountsConfigService
}

… lest you think this job should actually be disabled, don’t forget about the ol’ overrides file. Let’s query it to see if this job’s state is overridden there:

bash-3.2# /usr/libexec/PlistBuddy -c "print :com.apple.AccountsConfigService" \
/var/db/launchd.db/com.apple.launchd/overrides.plist

Dict {
    Disabled = false
}

We’ve determined what the associated process is (scsd) and that this webapp should be loaded based on the launchd config. Since it’s configured to load on demand, we might not expect to see the scsd process running; it’s not running on my system. There is a tiny man page for scsd. Don’t stop things you didn’t start ;)

Going forward

webappctl has a method called ‘tree’ that “displays the hierarchy of webapps declared by the requiredWebApps property.” Let’s look:

bash-3.2# webappctl trees - | fold -s

com.apple.webapp.ACSServer: com.apple.AccountsConfigService

com.apple.webapp.auth: com.apple.collabauthd

com.apple.webapp.collab: com.apple.collabcored1, com.apple.collabcored2,
com.apple.collabcored3, com.apple.collabcored4, com.apple.collabsandboxd,
com.apple.collabfeedd, com.apple.collabd, com.apple.collabd.notifications,
com.apple.collabd.quicklook, org.postgresql.postgres
. com.apple.webapp.auth: com.apple.collabauthd

com.apple.webapp.devicemgr: com.apple.devicemanager,
com.apple.DeviceManagement.SCEPHelper, org.postgresql.postgres
. com.apple.webapp.auth: com.apple.collabauthd

com.apple.webapp.mailman

com.apple.webapp.passwordreset: com.apple.passwordreset

com.apple.webapp.php: 

com.apple.webapp.podcastwikiui: com.apple.collabd.podcast-cache-updater

com.apple.webapp.webcal: com.apple.wikid.compatibility
. com.apple.webapp.auth: com.apple.collabauthd

com.apple.webapp.webcalssl: com.apple.wikid.compatibility
. com.apple.webapp.auth: com.apple.collabauthd

com.apple.webapp.webdavsharing: 

com.apple.webapp.webmailserver: com.example.placeholder, org.postgresql.postgres
. com.apple.webapp.php: 

org.calendarserver: org.calendarserver.calendarserver, org.postgresql.postgres

It might be tempting to think that the above is a complete expression of the dependency graph of related web apps. This is not true. For example, we see that org.calendarserver expresses dependencies on org.calendarserver.calendarserver and org.postgres.postgres. These two things are not web apps, but launchd jobs.

Diversion: just because you can…

So then, can we deduce that by starting the ‘org.calendarserver’ web app, we’d end up with a running calendar and postgres service? Well, we could, but we’d be wrong.

bash-3.2# webappctl start org.calendarserver
web:state = "RUNNING"

bash-3.2# webappctl status -
web:webAppState:_array_index:0:state = "RUNNING"
web:webAppState:_array_index:0:virtualHostName = ""
web:webAppState:_array_index:0:webAppName = "com.apple.webapp.ACSServer"
web:webAppState:_array_index:1:state = "RUNNING"
web:webAppState:_array_index:1:virtualHostName = ""
web:webAppState:_array_index:1:webAppName = "org.calendarserver"

Where’s postgres? What gives?

bash-3.2# tail -n 4 /var/log/system.log | cut -d' ' -f6-20
Reading configuration from file: /etc/caldavd/caldavd.plist
Neither EnableCalDAV nor EnableCardDAV are set to True.
(org.calendarserver.calendarserver[11258]): Exited with code: 1
(org.calendarserver.calendarserver): Throttling respawn: Will start in 59 seconds

Not so fast, smart guy. What makes you think you can just go around using new and mostly unknown tools for starting and stopping services, without being given express permission?! Don’t tell me you read it on the Internet. The point is that things are becoming more complex, and you really need to use the highlest-level interface available for doing things. In this case, that means Server.app or the serveradmin command line tool. In this specific case, the same Calendar Server software can provide both CalDAV (for calendaring) and CardDAV (for address book) services, but each is only enabled when they are configured to be enabled (in /etc/caldavd/caldavd.plist). It is only through Server.app / serveradmin that these config keys get enabled prior to loading the launchd job.

What about that dep graph again?

Now that you are sufficiently chastised, let’s get back to our example and see what happens when we start calendar server the right way, after first undoing the damage from earlier:

bash-3.2# webappctl stop org.calendarserver
web:state = "STOPPED"

bash-3.2# serveradmin start calendar
calendar:state = "RUNNING"
calendar:setStateVersion = 1
calendar:readWriteSettingsVersion = 1

bash-3.2# webappctl status -
web:webAppState:_array_index:0:state = "RUNNING"
web:webAppState:_array_index:0:virtualHostName = ""
web:webAppState:_array_index:0:webAppName = "com.apple.webapp.ACSServer"
web:webAppState:_array_index:1:state = "RUNNING"
web:webAppState:_array_index:1:virtualHostName = ""
web:webAppState:_array_index:1:webAppName = "com.apple.webapp.webcal"
web:webAppState:_array_index:2:state = "RUNNING"
web:webAppState:_array_index:2:virtualHostName = ""
web:webAppState:_array_index:2:webAppName = "com.apple.webapp.auth"
web:webAppState:_array_index:3:state = "RUNNING"
web:webAppState:_array_index:3:virtualHostName = ""
web:webAppState:_array_index:3:webAppName = "org.calendarserver"

Who said anything about com.apple.webapp.auth or com.apple.webapp.webcal, and where is postgres? Well, first of all remember that postgres won’t show up here, because ‘org.postgresql.postgres’ refers to a launchd job, not a web app. If we check launchd and a process listing, we see that postgres is indeed running:

bash-3.2# ps ax | grep -i post | fold -s
11452   ??  Ss     0:00.15 /usr/bin/postgres -D /var/pgsql -c listen_addresses=
-c log_connections=on -c log_directory=/Library/Logs -c
log_filename=PostgreSQL.log -c log_lock_waits=on -c log_statement=ddl -c
log_line_prefix=%t  -c logging_collector=on -c
unix_socket_directory=/var/pgsql_socket -c unix_socket_group=_postgres -c
unix_socket_permissions=0770
11459   ??  Ss     0:00.08 postgres: logger process
11461   ??  Ss     0:00.29 postgres: writer process
11462   ??  Ss     0:00.22 postgres: wal writer process
11463   ??  Ss     0:00.16 postgres: autovacuum launcher process
11464   ??  Ss     0:00.35 postgres: stats collector process
11837 s001  R+     0:00.00 grep -i post

bash-3.2# launchctl list | grep -i postgres
11452	-	org.postgresql.postgres

Ok, but what about the ‘auth’ and ‘webcal’ web apps? Well, calendar server is cool and has a nifty log that keeps track of what happens through its servermgrd bundle. Let’s see if it’s doing any webcal stuff:

bash-3.2# grep webcal /var/log/caldavd/servermgr_calendar.log | tail -n 1
[cal] 7/24/11 2:48:15 PM PDT : Changing com.apple.webapp.webcal WebAppState to START

We could then look at the webcal webapp plist to see that it also starts the ‘auth’ one. Whew, ok. Maybe you liked it better when it was mysterious and magical :)

The take home point here is that services can and do directly start web apps on their own. The policies controlling this behavior may be codified in the app, so you should have no expectation of being able to directly lay eyes on these policies, although you can get a good feel for it through experimentation. It’s probably a good idea not to use webappctl to twiddle the state of any web apps that you didn’t create, lest you invalidate somebody else’s expectations.