Traffic Shaping with pfSense and HFSC

Traffic Shaping with pfSense and HFSC (youtube)

This screencast demonstrates the use of a pfSense device for traffic shaping on a typical home network, with the goals of minimizing latency and maximizing throughput. In particular, we use a three-tier queue configuration where a parent speedboost queue on each interface contains leaf queues that catch all the traffic. The speedboost queues use HFSC’s non-linear service curve to match the behavior of the comcast speedboost. The leaf queues are configured to partition the available bandwidth, and automatically allow ‘borrowing’ when there is no contention.

Section links:
Installation / Setup: http://www.youtube.com/watch?v=EfXImr5q-sw&t=3m01s
Monitoring: http://www.youtube.com/watch?v=EfXImr5q-sw&t=6m30s
Traffic Shaping: http://www.youtube.com/watch?v=EfXImr5q-sw&t=15m34s

In a future video, I will go into more detail about how the sharing happens when there is contention, and some additional techniques for testing these usage patterns.

Feedback is welcome! Especially from people who know more about this stuff than I do…

Here’s the slide deck.

Posted in tutorials | Leave a comment

Backfill

In preparation for a dreness.com site redesign, I am migrating away from LiveJournal, my longtime (but mostly abandoned) personal blog. WordPress offers import support from various sources, including LiveJournal. I let it grind for about 30 minutes and it got all 300+ LJ entries and all their comments. Awesome! Any non-public entires are set to be password protected by WordPress; I’ll probably figure out some way to remove those from the public listing…

The combination of this blog’s existing content plus the LiveJournal stuff results in pretty complete coverage of my activities back to May 2001. Woot!

This post also marks the introduction of RETINA GRAPHICS, via the WP Retina 2x plugin.

backfill

Posted in bit bucket, lj | Leave a comment

Permissions updated…

Spambots have been running wild lately, so I’ve tightened this site a little. To post a comment, users must be logged in and must have a previously approved comment – otherwise the comment goes into moderation queue.

** UPDATE

They really went crazy for reals, so I added a captcha for registration, and also mass-deleted hundreds of bogus accounts. I’m pretty sure I deleted some legit ones as well, as there’s no easy means for distinguishing them in the wordpress user list… so sorry, babies :/

Posted in bit bucket | Leave a comment

How I fixed broken internal audio by zapping pram

Oops. I think I stole my own thunder. It was a serious wtf moment though… by all rights, I really don’t think this is supposed to happen. The worst part of this experience is not the increasingly frantic hour of me trying all the ‘right’ things to fix my apparently dead internal audio, failing, and then feeling queazy about the prospect of having to get a new mac pro right now. That was uncomfortable, but temporary. The worst part is that I can no longer mock people who zap pram in attempts to fix random problems that were never intended to be fixable by a pram reset. I wonder what mysteriously beneficial effect permissions repair has…

To continue the upside-down nature of this post, I should mention that the initial damage appears to have been done by windows 7, which I installed on a dedicated internal HDD via the BootCamp Assistant. It all appeared to be going well for the first couple days (of my Tera blitz, during which I didn’t boot over to OS X even once), but then I bounced back to OS X and was stymied by both a much longer than average boot time, and also… no internal audio. Apple System Profiler didn’t see much of anything:

system profiler window showing a basically empty window where the details of the internal audio device should be

I did several normal reboots, shutdown –> boot cycles, disconnection of various extra hardware and peripherals, SMC reset… but nothing was helping. The first time I booted back to win7, it offered to re-install the drivers. Windows could see the internal audio device (which it believes is of the RealTek variety) and allow me to select it as the default device, but still no actual sound was emitted. I figured it was kinda toasted at that point, but then, not expecting it to work, I zapped pram, and lo and behold:

system profiler window showing normal internal audio device details

Boot time is back to normal, too. Hopefully I don’t need to add any permanent pram cheese to this bootcamp sandwich…

UPDATE 1: oh noes, this is totally reproducible. Basically every time I go from win to mac, the audio device goes away until I zap pram. a bug has been filed.

Posted in mac pro, OS X | Leave a comment

Ubuntu Development Bootstrap

Linux hackers surely already know this, but I’m only sorta a Linux hacker. This post documents a nice process for using apt to prepare an Ubuntu machine for development. In my case, I was trying to build a custom version of irssi that is patched to add SSL support to the irssi proxy module. Sure, I could have gone about this in the traditional manner, but given that irssi has a fair number of dependencies, it’s easier to let apt do the grunt work for me.

First, get the toolchain set up.

sudo apt-get install build-essential checkinstall

The build-essential and ‘check install’ targets are news to me. Build-essential, as the name suggests, includes some compilers and related tools that would be needed for building just about anything.

Checkinstall keeps track of all files installed by a “make install” or equivalent, creates a Slackware, RPM, or Debian package with those files, and adds it to the installed packages database, allowing for easy package removal or distribution.”

Next, in preparation for building irssi itself, get all the dependencies for irssi. This works because irssi is already in the apt package database, so apt knows what the dependencies are.

sudo apt-get build-dep irssi

From there, download the irssi source, patch it, and proceed to build / run / install it (perhaps installing with checkinstall so you can easily remove it later!)

p.s. this irssi proxy SSL patch isn’t working. bummer.

p.p.s stunnel works though!

Posted in development | Leave a comment

Gnuplot time-series buckets quickie

Let’s say you’ve got a log file of events with time stamps; the nature of the events isn’t relevant, and let’s say the stamps look like this:

14/Mar/2012:17:49:34
14/Mar/2012:17:49:34
14/Mar/2012:17:49:39
14/Mar/2012:17:49:40
14/Mar/2012:17:49:41
14/Mar/2012:17:49:58
14/Mar/2012:17:51:46
14/Mar/2012:17:51:46
14/Mar/2012:17:52:56
14/Mar/2012:17:52:56
14/Mar/2012:17:52:57
14/Mar/2012:17:52:57
14/Mar/2012:17:53:16
14/Mar/2012:17:53:17
14/Mar/2012:17:53:17

You want to create a graph of the frequency of events in the log file, with time on the x and frequency on the y. An easy tactic here is to put the events into buckets, and then find the count of items in each bucket. If we use a bucket size of one minute, we can simply lop off the seconds field, and then use ‘uniq -c’ to produce our frequency count for each minute:

cat test | cut -d: -f 1-3 | uniq -c
 6 14/Mar/2012:17:49
 2 14/Mar/2012:17:51
 4 14/Mar/2012:17:52
 3 14/Mar/2012:17:53

Then with gnuplot:

set xdata time
set timefmt "%d/%b/%Y:%H:%M"
plot 'events_per_minute.txt' using 2:1 title "events per minute" with impulses
Posted in scripts | Leave a comment

The case of the excessively congratulatory Memory Slot Utility

A while ago I filled up my final unused memory slot in my awesome Mac Pro with an additional 4 gigorams (for a grand total of 16). Since then, whenever I reboot I’m showered with praise by the Memory Slot Utility for a job well done. This made me feel pretty good for the first couple months, but eventually it got old and I was ready to say goodbye for the last time. Googling around, I found the usual colorful variety of suggestions, ranging from “do a permissions repair” to “reinstall Mac OS X” to “reformat your hard drive”. Manual invocations of Memory Slot Utility as root (e.g. sudo open -a Memory\ Slot\ Utility.app) didn’t help, although it was obvious that it needed to twiddle some bit somewhere, and my hunch was that this was most likely failing for authorization reasons.

The winning move turned out to be logging in briefly as root. Generally not a good idea to log in as root, but in this case it’s the easiest way to clear this condition. All you need to do is enable the root account by setting a password for it, and then log in as root from the login window.

  1. In Finder, choose Go –> Go to Folder… (or shift command g), and enter /System/Library/CoreServices
  2. In the CoreServices folder, open Directory Utility.
  3. From the Edit menu, choose “Enable Root User”, and supply a password.
  4. Open System Preferences –> Users & Groups prefpane, and under Login Options, turn off Automatic Login, and set the login window display to Name and password.
  5. Reboot. Log in as root from the login window. Perform one final heroic click of Memory Slot Utility’s OK button, going for maximum style points.
  6. Log out, log back in as yourself, put settings back the way you like them. I suggest using Directory Utility to disable the root account.
Posted in mac pro, OS X, Pro Tip, The More You Know | 1 Comment

SJ

I only worked under Steve’s leadership for a few years, and never spoke to him in person. Even so, he earned my respect for thinking big and really wanting people to *understand* what he was after. The technology is definitely just a means to an end.

I admire his dedication to building tools, both for developers and end-users. It’s pretty central for us as a species, and it’s been fascinating to see how such tools enable people to communicate better, find information better, organize thoughts better… all of it. Not to have cool gadgets, but to be empowered with abilities that would be impossible without the gadgets. And yeah, if you’re going to do gadgets, might as well make them beautiful.

For a while I was pretty conflicted about Apple’s direction with iOS and the conclusions one might draw about how it might affect the future of Mac OS X. I felt like my beloved computing environment was about to get dumbed down, and my concerns marginalized as an edge case. I still feel this way, but in the last year or so I’ve come to understand (and observe) that it’s totally worth it. Making an iPhone this easy to use means it gets *much* more widespread use than something else that might be more appealing (at least, at first blush) to geeks. The overall value of countless computer illiterates gaining access to iOS is surely far greater than the cost of some graybeards occasionally yelling about stuff on their lawns. My desire for people to be able to handle larger amounts of complexity in their computing devices seems completely selfish in retrospect.

So thanks, Steve, for the relentless focus on the ‘high order bit’. Your presence will be missed, but your message not forgotten.

Posted in bit bucket | Leave a comment

Exploring web apps in Lion Server

There is some new glue in Lion Server to help manage infrastructure sharing between services, mostly targeted at web-related services referred to as web apps. This post aims to shed a bit of light on the ‘web apps’ layer to see how it interacts with the server admin tools and with launchd.

Much of what you need to know can be found in the man pages for webappctl and webapp.plist (the man page links only work on Lion Server). I won’t repeat what’s already there; go read them ;)

Going backwards

First, let’s take a look at a fairly vanilla config to demonstrate how to work backwards from a webapp configuration to find who / what it is. On a server with no (visibly configured) services enabled, let’s use webappctl to get a list of the web apps that are loaded:

bash-3.2# webappctl status -
web:webAppState:_array_index:0:state = "RUNNING"
web:webAppState:_array_index:0:virtualHostName = ""
web:webAppState:_array_index:0:webAppName = "com.apple.webapp.ACSServer"

What is this ACSServer? Looking at the com.apple.webapp.ACSServer.plist, we see:

bash-3.2# /usr/libexec/PlistBuddy -c print \
/etc/apache2/webapps/com.apple.webapp.ACSServer.plist
Dict {
    name = com.apple.webapp.ACSServer
    proxies = Dict {
        /AccountsConfigService/api/ = Dict {
            urls = Array {

http://localhost:31415/AccountsConfigService/api

            }
            path = /AccountsConfigService/api/
        }
    }
    sslPolicy = 1
    requiredModuleNames = Array {
        mod_rewrite.so
    }
    includeFiles = Array {
        /etc/apache2/httpd_ACSServer.conf
    }
    launchKeys = Array {
        com.apple.AccountsConfigService
    }
}

The “launchKeys” entry specifies the name of the associated launched job, com.apple.AccountsConfigService, so let’s look at that:

bash-3.2# /usr/libexec/PlistBuddy -c print \
/System/Library/LaunchDaemons/com.apple.AccountsConfigService.plist
Dict {
    Program = /usr/libexec/scsd
    OnDemand = true
    Disabled = true
    Sockets = Dict {
        Listeners = Array {
            Dict {
                SockFamily = IPv4
                SockNodeName = 127.0.0.1
                SockServiceName = 31415
                SockType = stream
            }
        }
    }
    UserName = _scsd
    ProgramArguments = Array {
        /usr/libexec/scsd
        -l
    }
    GroupName = _scsd
    Label = com.apple.AccountsConfigService
}

… lest you think this job should actually be disabled, don’t forget about the ol’ overrides file. Let’s query it to see if this job’s state is overridden there:

bash-3.2# /usr/libexec/PlistBuddy -c "print :com.apple.AccountsConfigService" \
/var/db/launchd.db/com.apple.launchd/overrides.plist
Dict {
    Disabled = false
}

We’ve determined what the associated process is (scsd) and that this webapp should be loaded based on the launchd config. Since it’s configured to load on demand, we might not expect to see the scsd process running; it’s not running on my system. There is a tiny man page for scsd. Don’t stop things you didn’t start ;)

Going forward

webappctl has a method called ‘tree’ that “displays the hierarchy of webapps declared by the requiredWebApps property.” Let’s look:

bash-3.2# webappctl trees - | fold -s

com.apple.webapp.ACSServer: com.apple.AccountsConfigService

com.apple.webapp.auth: com.apple.collabauthd

com.apple.webapp.collab: com.apple.collabcored1, com.apple.collabcored2,
com.apple.collabcored3, com.apple.collabcored4, com.apple.collabsandboxd,
com.apple.collabfeedd, com.apple.collabd, com.apple.collabd.notifications,
com.apple.collabd.quicklook, org.postgresql.postgres
. com.apple.webapp.auth: com.apple.collabauthd

com.apple.webapp.devicemgr: com.apple.devicemanager,
com.apple.DeviceManagement.SCEPHelper, org.postgresql.postgres
. com.apple.webapp.auth: com.apple.collabauthd

com.apple.webapp.mailman

com.apple.webapp.passwordreset: com.apple.passwordreset

com.apple.webapp.php: 

com.apple.webapp.podcastwikiui: com.apple.collabd.podcast-cache-updater

com.apple.webapp.webcal: com.apple.wikid.compatibility
. com.apple.webapp.auth: com.apple.collabauthd

com.apple.webapp.webcalssl: com.apple.wikid.compatibility
. com.apple.webapp.auth: com.apple.collabauthd

com.apple.webapp.webdavsharing: 

com.apple.webapp.webmailserver: com.example.placeholder, org.postgresql.postgres
. com.apple.webapp.php: 

org.calendarserver: org.calendarserver.calendarserver, org.postgresql.postgres

It might be tempting to think that the above is a complete expression of the dependency graph of related web apps. This is not true. For example, we see that org.calendarserver expresses dependencies on org.calendarserver.calendarserver and org.postgres.postgres. These two things are not web apps, but launchd jobs.

Diversion: just because you can…

So then, can we deduce that by starting the ‘org.calendarserver’ web app, we’d end up with a running calendar and postgres service? Well, we could, but we’d be wrong.

bash-3.2# webappctl start org.calendarserver
web:state = "RUNNING"
bash-3.2# webappctl status -
web:webAppState:_array_index:0:state = "RUNNING"
web:webAppState:_array_index:0:virtualHostName = ""
web:webAppState:_array_index:0:webAppName = "com.apple.webapp.ACSServer"
web:webAppState:_array_index:1:state = "RUNNING"
web:webAppState:_array_index:1:virtualHostName = ""
web:webAppState:_array_index:1:webAppName = "org.calendarserver"

Where’s postgres? What gives?

bash-3.2# tail -n 4 /var/log/system.log | cut -d' ' -f6-20
Reading configuration from file: /etc/caldavd/caldavd.plist
Neither EnableCalDAV nor EnableCardDAV are set to True.
(org.calendarserver.calendarserver[11258]): Exited with code: 1
(org.calendarserver.calendarserver): Throttling respawn: Will start in 59 seconds

Not so fast, smart guy. What makes you think you can just go around using new and mostly unknown tools for starting and stopping services, without being given express permission?! Don’t tell me you read it on the Internet. The point is that things are becoming more complex, and you really need to use the highlest-level interface available for doing things. In this case, that means Server.app or the serveradmin command line tool. In this specific case, the same Calendar Server software can provide both CalDAV (for calendaring) and CardDAV (for address book) services, but each is only enabled when they are configured to be enabled (in /etc/caldavd/caldavd.plist). It is only through Server.app / serveradmin that these config keys get enabled prior to loading the launchd job.

What about that dep graph again?

Now that you are sufficiently chastised, let’s get back to our example and see what happens when we start calendar server the right way, after first undoing the damage from earlier:

bash-3.2# webappctl stop org.calendarserver
web:state = "STOPPED"
bash-3.2# serveradmin start calendar
calendar:state = "RUNNING"
calendar:setStateVersion = 1
calendar:readWriteSettingsVersion = 1
bash-3.2# webappctl status -
web:webAppState:_array_index:0:state = "RUNNING"
web:webAppState:_array_index:0:virtualHostName = ""
web:webAppState:_array_index:0:webAppName = "com.apple.webapp.ACSServer"
web:webAppState:_array_index:1:state = "RUNNING"
web:webAppState:_array_index:1:virtualHostName = ""
web:webAppState:_array_index:1:webAppName = "com.apple.webapp.webcal"
web:webAppState:_array_index:2:state = "RUNNING"
web:webAppState:_array_index:2:virtualHostName = ""
web:webAppState:_array_index:2:webAppName = "com.apple.webapp.auth"
web:webAppState:_array_index:3:state = "RUNNING"
web:webAppState:_array_index:3:virtualHostName = ""
web:webAppState:_array_index:3:webAppName = "org.calendarserver"

Who said anything about com.apple.webapp.auth or com.apple.webapp.webcal, and where is postgres? Well, first of all remember that postgres won’t show up here, because ‘org.postgresql.postgres’ refers to a launchd job, not a web app. If we check launchd and a process listing, we see that postgres is indeed running:

bash-3.2# ps ax | grep -i post | fold -s
11452   ??  Ss     0:00.15 /usr/bin/postgres -D /var/pgsql -c listen_addresses=
-c log_connections=on -c log_directory=/Library/Logs -c
log_filename=PostgreSQL.log -c log_lock_waits=on -c log_statement=ddl -c
log_line_prefix=%t  -c logging_collector=on -c
unix_socket_directory=/var/pgsql_socket -c unix_socket_group=_postgres -c
unix_socket_permissions=0770
11459   ??  Ss     0:00.08 postgres: logger process
11461   ??  Ss     0:00.29 postgres: writer process
11462   ??  Ss     0:00.22 postgres: wal writer process
11463   ??  Ss     0:00.16 postgres: autovacuum launcher process
11464   ??  Ss     0:00.35 postgres: stats collector process
11837 s001  R+     0:00.00 grep -i post
bash-3.2# launchctl list | grep -i postgres
11452	-	org.postgresql.postgres

Ok, but what about the ‘auth’ and ‘webcal’ web apps? Well, calendar server is cool and has a nifty log that keeps track of what happens through its servermgrd bundle. Let’s see if it’s doing any webcal stuff:

bash-3.2# grep webcal /var/log/caldavd/servermgr_calendar.log | tail -n 1
[cal] 7/24/11 2:48:15 PM PDT : Changing com.apple.webapp.webcal WebAppState to START

We could then look at the webcal webapp plist to see that it also starts the ‘auth’ one. Whew, ok. Maybe you liked it better when it was mysterious and magical :)

The take home point here is that services can and do directly start web apps on their own. The policies controlling this behavior may be codified in the app, so you should have no expectation of being able to directly lay eyes on these policies, although you can get a good feel for it through experimentation. It’s probably a good idea not to use webappctl to twiddle the state of any web apps that you didn’t create, lest you invalidate somebody else’s expectations.

Posted in OS X Server | 4 Comments

Mailman members only

Recently I had occasion to cook up some glue for a Mailman deployment that wanted a privacy and membership management policy that would not be facilitated by the Mailman mailing list software in its current form. We wanted to host a private list (disabling web subscriptions), and we wanted to allow any member to invite new members.

The solution comes in a few pieces:

  • a python script to synchronize the Mailman username / password credentials to an AuthUserFile that can be used by Apache to authorize access to a cgi that wraps the mailman add_member cli tool. Find the source here: mailman_cred_export.py. Put this in root’s crontab at some reasonable interval.
  • the aforementioned script that wraps the add_member tool. Find the source here: addmember.py.
  • this guy’s modified add_members cli tool. The add_members tool supplied by Mailman bypasses confirmation, but we want invitees to respond to a confirmation message before being added. This script makes that happen.
  • Disable web subscriptions by deleting stuff from /etc/mailman/en/listinfo.html
Posted in development, scripts | Leave a comment