Tiger Server servermgrd library for Python

motivation
I finally found a reason to write some Python. Being something of a monitoring and data junkie, I’ve had a fair amount of experience with snmp, data mining scripts, etc. After writing this post to the server list, I figured I’d make some templates for snmp that published interesting pieces of data about the server. A lot of good stuff can be retrieved through servermgrd, the ‘Mac OS X Server administrative daemon’, which is basically a little web service that uses xml plists to do request / response transactions. Usually the only software that talks to servrmgrd is Apple’s Server Admin utility, but the enterprising sysadmin can strike up his or her own conversations. This post documents my python baby steps, as well as the birth of a tiny python library I wrote for simplifying servermgrd interactions.

If you happen to have a Mac OS X Server handy, point your web browser at https://your.server:311. Accept the ol’ SSL warnings, and then you’ll be presented with a list of servermgrd modules. Each module has its own html / cgi wrapper which provides a menu of request templates, e.g.

<?xml version="1.0" encoding="UTF-8"?>
<plist version="0.9">
<dict>
    <key>command</key>
    <string>getState</string>
    <key>variant</key>
    <string>withDetails</string>
</dict>
</plist>

Different modules support different commands, some with optional arguments (e.g. variant, timescale). Clicking ‘Send Command’ will do just that, and you’ll see the results as returned by servermgrd. Not surprisingly, the result is also an xml plist.

We’ll definitely want some sort of plist parsing library to let us grab ahold of this data in a fairly painless way. This was ultimately why I chose Python for the task; because Mac OS X ships with a little Python library called plistlib. plistlib is pretty basic; you give it a plist and it will hand over a data structure with all the stuff in it.

Dive in: the python interpreter, arrays, and dicts in like 3 minutes
First let’s find a plist we can use to test. We’ll need to convert it to xml format, as many plists are stored on disk in a biniary format these days. Let’s take our Dock plist and make an xml copy of it at ~/test.plist

plutil -convert xml1 -o test.plist ~/Library/Preferences/com.apple.dock.plist

Now let’s try plistlib through an interactive python session.

{29} andre@donk [~] % python
Python 2.3.5 (#1, Aug 12 2006, 00:08:11)
[GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import plistlib
>>> pl = plistlib.Plist.fromFile('test.plist')
>>>

Cool, no errors! (right? ;) Simply say the name of the object to see its contents:

>>> pl

Woof, lots of output. Here’s where it gets fun. This is a dict, which a collection of property / value pairs. let’s iterate through the items.

>>> for item in pl : print item
...
wvous-br-corner
orientation
mod-count
tilesize
trash-full
persistent-apps
pinning
wvous-br-modifier
version
launchanim
autohide
persistent-others
checked-for-dashboard

Those are properties. Properties have values.

>>> pl['trash-full']
True
>>> pl['launchanim']
False

Iterate through all the items and display them as key –> value pairs:

>>> for item in pl : print item, ' --> ', pl[item]
...

Let’s drill down into that persistent-apps item. It appears to be a bunch of nested stuff, all wrapped in a single list (array). Let’s list the properties of one of the apps; we’ll pick the first one (list index 0).

>>> for item in pl['persistent-apps'][0] : print item
...
tile-data
tile-type
GUID

That’s a bit more managable. Let’s examine these.

>>> pl['persistent-apps'][0]['GUID']
289528741
>>> pl['persistent-apps'][0]['tile-type']
'file-tile'
>>> pl['persistent-apps'][0]['tile-data']
Dict(**{'parent-mod-date': 3264304302L, 'file-label': 'System Preferences', 'file-data': ...

tile-data has more nested stuff, but now we can easily pick out file-label as the human readable name of the thing. Remember how we got here? Iterate through all the persistent-apps, look into the tile-data dict, then print the value of the file-label property

>>> for app in pl['persistent-apps'] : print app['tile-data']['file-label']
...
System Preferences
iChat
Mail
iTunes
Safari
Terminal
Quake 4
TextEdit
World of Warcraft
Hex Fiend

Pretty much everything that comes out of plistlib is either a dict or an array, and there’s often some nesting, so you sorta have to grope the format a bit to figure out how you want to access the data. Definitely beats the crap out of hand parsing :)

what about servermgrd?
servermgrd can be harnessed either via http or a shell. We’ll use a shell. In that mode, the request is delivered via standard input, and the result is… well ya know. output. Let’s build a request. After playing with the web interface for a while, its obvious that the various requests are formated the same way. The value of the ‘command’ property is the most significant part. Some commands have additional parameters, such as ‘variant’ or ‘timescale’. Our request-building function shall be called buildXML.

def buildXML ( command, variant, timescale ) :
  request = """<?xml version="1.0" encoding="UTF-8"?>
<plist version="0.9">
<dict>
        <key>command</key>
        <string>"""
  request = request + command
  request = request + '</string>'
  if timescale != '' :
    request = request + """
        <key>timeScale</key>
        <integer>"""
    request = request + timescale
    request = request + '</integer>'
  if variant != '' :
    request = request + """
        <key>variant</key>
        <string>"""
    request = request + variant
    request = request + '</string>'
  request = request + """
</dict>
</plist>"""
  return request

buildXML is called like this:

request = buildXML('getHistory', 'v1+v2', '60')

Sometimes you need not specify anything more than command. In these cases, supply a null value for any unused parameters using empty single quotes.

Now that we have a request, we can send it to servermgrd by opening a pipe. We’ll use popen2 so we can grab both STDIN and STDOUT. The filesystem path we use depends on the name of the module we’re targetting. Here is our sendXML function, which is called with the name of the servermgrd module and the xml request.

def sendXML ( servermgrdModule, request ) :
  modulePath = '/usr/share/servermgrd/cgi-bin/'+servermgrdModule
  pipeIn, pipeOut = os.popen2(`modulePath`)
  print >>pipeIn, request
  pipeIn.close()
  xmlresult = pipeOut.read(20480)
  pipeOut.close()

We now have xmlresult, which is a string containing the entire result body from servermgrd. Plistlib is accustomed to parsing plists from files, but we don’t want to write this data to the filesystem because we don’t want to keep it. Instead, we’ll finish this function by creating a file-like object (like a file, but without any of that annoying disk access) which contains the result, using the StringIO library, then hand that file-like object to plistlib. plistlib parses the xml into native data structures (dicts, arrays), and we return the result.

  xmlFauxFile = StringIO.StringIO(xmlresult)
  return plistlib.Plist.fromFile(xmlFauxFile)

Examples
I recommend exploring the XML plist you can get from servermgrd through the eyes of the python interpreter using the techniques demonstrated with the Dock example. To get started, simply download the servermgrd.py library, then enter the python interpreter as root on a Tiger Server, while your current directory is the same as servermgrd.py’s directory (or else the import fails).

{131} root@tiny [~] # python
Python 2.3.5 (#1, Jul 25 2006, 00:38:48)
[GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import srvrmgrdIO
>>> request = srvrmgrdIO.buildXML('getHistory', 'v1', '60')
>>> pl = srvrmgrdIO.sendXML('servermgr_info', request)
>>> pl
Plist(**{'v2Legend': 'NETWORK_THROUGHPUT', 'v5Legend': 'NETWORK_THROUGHPUT_EN0', ...

Here’s some quick examples of how this library can be used. There is much more available data than is being shown.

#!/usr/bin/python
# We require the srvrmgrdIO module to prepare the request and talk to servermgrd
import re
import srvrmgrdIO
import time

print 'network bytes / second over the last 15 minutes:'
request = srvrmgrdIO.buildXML('getHistory', 'v1+v2', '900')
pl = srvrmgrdIO.sendXML('servermgr_info', request)
for s in pl['samplesArray'] :
  print s['v1'], 'at', time.ctime(s['t'])
print ""

# dns - this one's real slow for some reason...
#request = srvrmgrdIO.buildXML('getStatistics', '', '')
#pl = srvrmgrdIO.sendXML('servermgr_dns', request)

#print "DNS: success / fail / recursive / referral / nxdomain"
#print `pl['success']` + ' /',
#`pl['failure']` + ' /',
#`pl['recursion']` + ' /',
#`pl['referral']` + ' /',
#`pl['nxdomain']`
#print ""

# afp connected users
request = srvrmgrdIO.buildXML('getConnectedUsers', '', '')
pl = srvrmgrdIO.sendXML('servermgr_afp', request)
print "AFP Users:"
for u in pl['usersArray'] :
  print u['ipAddress'] + " ==> " + u['name']
print ""

# dirserv
print "Directory Services"
request = srvrmgrdIO.buildXML('getState', 'withDetails', '')
pl = srvrmgrdIO.sendXML('servermgr_dirserv', request)

for s in pl :
  if re.search("stat", s, re.I) : print s," ==> ",`pl[s]`

When executed on my server:

{4} root@tiny [~] # ./satest.py
network bytes / second over the last 15 minutes:
38 at Mon Jun 11 19:56:14 2007
35 at Mon Jun 11 19:55:14 2007
35 at Mon Jun 11 19:54:14 2007
35 at Mon Jun 11 19:53:14 2007
38 at Mon Jun 11 19:52:14 2007
35 at Mon Jun 11 19:51:14 2007
43 at Mon Jun 11 19:50:14 2007
43 at Mon Jun 11 19:49:14 2007
33 at Mon Jun 11 19:48:14 2007
38 at Mon Jun 11 19:47:14 2007
40 at Mon Jun 11 19:46:14 2007
36 at Mon Jun 11 19:45:14 2007
35 at Mon Jun 11 19:44:14 2007
26 at Mon Jun 11 19:43:14 2007
36 at Mon Jun 11 19:42:14 2007

AFP Users:
10.0.1.201 ==> andre
10.0.1.6 ==> andre

Directory Services
timState  ==>  'STOPPED'
ldapdState  ==>  'RUNNING'
kdcStatus  ==>  'RUNNING'
lookupdState  ==>  'RUNNING'
passwordServiceState  ==>  'RUNNING'
state  ==>  'RUNNING'
netinfodState  ==>  'RUNNING'
netinfodParentState  ==>  'STOPPED'

About dre

I like all kinds of food.
This entry was posted in development, OS X Server, scripts, tutorials. Bookmark the permalink.

17 Responses to Tiger Server servermgrd library for Python

  1. filipp says:

    Very cool, thanks for sharing!

    This reminds me of an idea I had of creating a cross-platform Server Admin. Python would be perfect for this since it runs on anything and also has GUI bindings.

    plistlib is very cool. Thanks for bringing it up, I had no idea it even existed. I’m working on alot of plist processing right now and the solution I’m using involves alot of XSLT. I also came up with another one, similar to plistlib using ObjC and the NSPropertyListSerialization class.

  2. felimwhiteley says:

    Does this work for a remote server at all ? It’s very cool libraries. I’m trying to run them from a separate (non Mac) box. Python is not my strong point so any pointers would be welcome. Appreciate the code though !

    :-)

  3. dre says:

    Felimwhiteley:

    The srvrmgrdIO library needs to run on the Mac OS X Server itself, as it is currently written. This is because the library executes another command line tool (servermgrd) and then communicates with it via stdin / stdout.

    However, servermgrd also supports an xml protocol via http which provides all the same functionality (https://your.server:311). The sendXML function could be re-written to communicate with a remote Mac OS X Server by doing xml communication with servermgrd via https on port 311 instead of local communication with servermgrd via popen2.

    My purpose for writing this library was to feed data to a local Net SNMP daemon, which can be queried remotely, so I have no need to add remote querying to srvrmgrdIO.

    If you want to add this, check out urllib2 (in the python standard library). Remember that you’ll need to authenticate to the servermgrd xml interface using an admin username and password.

    Good luck :)

  4. felimwhiteley says:

    Ah cheers for the response Dre, I’m probably going to have to… better get my python learning up to scratch asap. I have to monitor multiple Servers across multiple sites, so avoiding installing anything is a major factor. Thanks for the code though, it’ll really make it doable, and cheers for the pointers too :-)

  5. felimwhiteley says:

    Don’t know if you want this code or not but I have modified your srvrmgrdIO as you suggested… First off I really do not know Python.. this is a hobbled together collection of code from various sites and some help from some good friends, but it works. I added in and extra def and modified the sendXML. I can run your test example above against a remote server and it works perfectly the same. I know for sure this can be done better but maybe it’ll help someone else out.

    Thanks for your intial code it’s saved my bacon :-)

    def download_file(url, webuser = None, webpass = None):
    request = urllib2.Request(url)
    if webuser:
    base64string = base64.encodestring(‘%s:%s’ % (webuser, webpass))[:-1]
    request.add_header(“Authorization”, “Basic %s” % base64string)
    htmlFile = urllib2.urlopen(request)
    htmlData = htmlFile.read()
    htmlFile.close()

    return htmlData

    def sendXML ( servermgrdModule, request ) :
    webuser = “some_admin_on_server”
    webpass = “the_password”
    url = ‘https://mac_server_address:311/commands/’+servermgrdModule+’?input=’+urllib.quote(request)
    xmlresult = download_file(url, webuser, webpass)
    xmlFauxFile = StringIO.StringIO(xmlresult)
    return plistlib.Plist.fromFile(xmlFauxFile)

  6. dre says:

    Felimwhiteley:

    I haven’t tested this yet, but it looks great! Since you’ve done all the hard work, I will probably roll this new functionality into the existing code, allowing it to work either locally or remotely.

    Thanks!

  7. felimwhiteley says:

    You are very welcome. Trust me if it weren’t for your code this never would have been possible. Start of last week I knew no python :-)

    I made a couple more minor edits to make it a bit more usable.

    def sendXML ( servermgrdModule, request, server, port, webuser, webpass ) :
    url = ‘https://’+server+’:’+port+’/commands/’+servermgrdModule+’?input=’+urllib.quote(request)
    #print “*** DEBUG ***”
    #print url
    #print “*** DEBUG ***”
    xmlresult = request_data(url, webuser, webpass)
    xmlFauxFile = StringIO.StringIO(xmlresult)
    return plistlib.Plist.fromFile(xmlFauxFile)

    So it doesn’t have it stored in the library, then you would need to edit your test code to having a few lines like this:

    import sys

    server = sys.argv[1]
    port = sys.argv[2]
    webuser = sys.argv[3]
    webpass = sys.argv[4]

    So we can enter them at command line. Reason I added port, I was testing this via a rmeote SSL Tunnel, so I had set up port 8311 localhost to point to the server’s port 311.

    Then any check code needs to use:

    print ‘network bytes / second over the last 15 minutes:’
    request = srvrmgrdIO.buildXML(‘getHistory’, ‘v1+v2’, ‘900’)
    pl = srvrmgrdIO.sendXML(‘servermgr_info’, request, server, port, webuser, webpass)
    for s in pl[‘samplesArray’] :
    print s[‘v2’], ‘at’, time.ctime(s[‘t’])
    print “”

    Thanks again for your code :-)

  8. felimwhiteley says:

    Dre I wonder if you have a chnce to help me understand something with the returned info from srvrmgrdIO.

    I am trying to get info from this command on CPU Temprature:

    request = srvrmgrdIO.buildXML(‘status’, ”, ”)
    pl = srvrmgrdIO.sendXML(‘servermgr_xserve’, request, server, port, webuser, webpass)

    # print len(pl)
    #
    print “******”

    for s in pl :
    print s

    print “******”

    for s in pl[‘Status’] :
    print s

    print “******”

    I’ve stuck in some lines to try understand what is happening at each branch. In all of your examples it’s bene nice in that it was not nested very deeply. Unfortunatly when I use pl[‘Status’] I have a Temprature section I need to get into then about another two below it. I’ve been puzzled by this, I’m sure there is a way but being so green on Python I’m stumped.

    I’m half tempted to just

    print `pl[‘Status’]` and use BASH/Grep/SED etc. to prune out what I need but that defeats the purpose.

    HAve you any pointers ? I’m sorry to bother you with this but finding code like this out on the web has been impossible.

  9. dre says:

    Felimwhiteley:

    These data structures allow targeting of nested items in a fairly straight-forward fashion. Take another look at all the Dock examples in the first half of the article. For example:

    pl[‘persistent-apps’][0][‘GUID’]

    This is a reference to the “GUID” key in the 1st array element (0-based index) of the persistent-apps key of the “pl” plist object. You can nest as far down as you need to, and you may need to have some logic for deciding which ‘branches’ you are interested in, etc.

    Another thing that may help is to retrieve the entire plist result from the remote server and then view it in a structured form, so you can see the indenting of each block. In the Mac OS X Developer tools you will find a utility called Property List Editor that can provide a hierarchical view of your plist.

    Hope this helps!

  10. felimwhiteley says:

    Meant to thank you. I’m not sure what I was doing but I was pretty close… Anyway works like a charm now :-)

  11. felimwhiteley says:

    Hi Again,

    Just in case you have Leopard. Had to modify code a bit. They stick a faux header at the top of the packet, so need to split it off when we reach the end of it at “\r\n\r\n” sequence. Also they implement realms to gain access and have changed the order of when it ask for a password. The code is hideously ugly… but it works :-) Oh and the if re.match(“SupportsBinaryPlist” checks to see if it is in fact leopard, otherwise it performs the older method which I’ve tested back to Panther.

    Anyway here’s the code !

    def download_file(url, webuser = None, webpass = None):
    from urllib2 import (HTTPPasswordMgr, HTTPBasicAuthHandler, build_opener, install_opener, urlopen, HTTPError)
    password_mgr = HTTPPasswordMgr()
    password_mgr.add_password(“Server Admin”, url, webuser, webpass)
    handler = HTTPBasicAuthHandler(password_mgr)
    opener = build_opener(handler)
    install_opener(opener)
    request = urllib2.Request(url)
    if webuser:
    base64string = base64.encodestring(‘%s:%s’ % (webuser, webpass))[:-1]
    request.add_header(“Authorization”, “Basic %s” % base64string)
    request.add_header(‘WWW-Authenticate’, ‘Basic realm=”Server Admin”‘)
    try:
    htmlFile = urllib2.urlopen(request)
    htmlData = htmlFile.read()
    htmlFile.close()
    if re.match(“SupportsBinaryPlist”, htmlData):
    xmlDump = re.split(“\r\n\r\n”, htmlData, 1)
    return xmlDump[1]
    else:
    return htmlData
    except HTTPError, e:
    print e.code
    print e.headers
    sys.exit(2)

  12. felimwhiteley says:

    Dre a quick question on this code, I was gonna stick it and a bunch of scripts into Google-code, would you have issue with me tagging it as GPL ? My thinking is it’d be nice if it was useful to others and they fixed/improved/modified it they’d give it back. I’m not sure I could claim my changes are fantastic but I’d certainly like it to stay open as a useful tool to folk.

  13. dre says:

    Heya,

    I’m all for sharing! I haven’t actually touched this in a while, and at this point it might be fair to say you’ve spent more time on this than I have :) Since I didn’t put a license on it, I’ll leave it up to you. One comment: GPL is actually too restrictive for some uses; basically it prevents use in closed-source software (and there is plenty legitimately closed-source software :) If the goal is to make the code as useful as possible, then I would aim for a BSD style license.

  14. felimwhiteley says:

    Excellent, glad you are up for it. I guess this is were my slightly biased Linux to BSD ratio shows through. Maybe an option is to dual license it then, although I still prefer GPL as it does force contributing back, as the next [insert rich IT person’s name here] walks off with it and makes millions lol

    I’ll let you know once I get it set up etc. anyway. And thanks for the initial idea ! :-)

  15. felimwhiteley says:

    Hi Dre,

    I’ve finally chosen LGPL, means any changes to the lib if distributed must be given back but folk are free to use it in commercial software, think it meets the halfway point quiet well. I have modified it from yours a bit, as it can be used by multiple apps at once, say Cacti for graphs and Nagios for checking. Anyway please feel free to ask for any changes and I didn’t know your full name to credit you. I did search but couldn’t find it.

    http://code.google.com/p/libsrvrmgrd-osx/

  16. dre says:

    @ felimwhiteley: fantastic! This is a great example of why open source is cool :)

    Also, I’ve added a link to your blog to my site.

    Cheers!

  17. felimwhiteley says:

    Ah cool thanks for the link :-)

    Just let you know minor update to it, 0.5.0, it’s got logging ability now but should still maintain backwards computability with your scripts or with very little change.

Leave a Reply