Traffic micro-management: limit network bandwidth used by an OS X process

Problem: you have some silly uploader app that only knows one speed: maximum. You would love a way to make that app back off so it doesn’t saturate your uplink and badly impact the other things using your network. Previously in OS X, you would accomplish this using ipfw, but these days you’d use pfctl and dnctl.

None of that is breaking news. The reason for this post is that I thought of an easy way to make the limits apply only to a specific app’s traffic, and all of that traffic. Traditionally you’d have to identify the traffic using some combination of source or destination IP address or port, which can get quite cumbersome. PF also supports matching packets by uid and gid. Let’s use gid, and then run the target app with a custom gid :)

First, make a new unix group called throttle:

sudo dseditgroup -o create throttle

Next, give yourself permission to run things with the group id of the throttle group. To do this, edit /etc/sudoers to add a line such as this, substituting your actual user name.

username ALL=(ALL:throttle) ALL

To test the new group and sudo configuration, run ‘id -g’, and then again with a custom group id (using sudo’s -g option). The results should be different.

$ id -g
20 
$ sudo -g throttle id -g
502

Now we can instantiate the requisite PF and dummynet setup. Here’s a little script that wants to be run as root.

#!/bin/bash

# Reset dummynet to default config
dnctl -f flush

# Compose an addendum to the default config to create a new anchor
read -d '' -r PF <<EOF
dummynet-anchor "throttle"
anchor "throttle"
EOF

# Reset PF to default config and apply our addendum
(cat /etc/pf.conf && echo "$PF") | pfctl -q -f -

# Configure the new anchor
cat <<EOF | pfctl -q -a throttle -f -
no dummynet quick on lo0 all
dummynet out proto tcp group throttle pipe 1
dummynet out proto udp group throttle pipe 1
EOF

# Create the dummynet queue - adjust speed as desired
dnctl pipe 1 config bw 1Mbit/s

# Show new configs
printf "\nGlobal pf dummynet anchors:\n"
pfctl -q -s dummynet
printf "\nthrottle anchor config:\n"
pfctl -q -s dummynet -a throttle
printf "\ndummynet config:\n"
dnctl show queue

# Enable PF
pfctl -E

Finally, start the target app as follows:

sudo -g throttle open -a "Whatever"

To watch the counters on the bandwidth limiting queue:

sudo dnctl show queue

To clear all PF config / state and reset PF to defaults:

sudo pfctl -F all ; sudo pfctl -q -f /etc/pf.conf

NB: prior to 10.11.2, use of “pfctl -F all” might kernel panic your machine. This post was made public in December, but I wrote it back in August and filed the kernel panic bug then, which is fixed as of 10.11.2. Unfortunately, the bug goes back at least as far as 12F45…

To wrap up, keep in mind that this is a hack. What we’re really trying to accomplish requires both a privileged position on the network and also more expressive and fine-grained traffic controls than are provided by dummynet. As both of those things are not always available, doing dumb rate limiting on an individual host as documented above can still be useful.

While we’re here, let’s take a moment to illustrate the difference between dumb host-level rate limits (which I am derisively referring to as “traffic micro-management”) and proper traffic shaping at the network edge. Two primary goals of traffic shaping are: 1) avoid congestion at the bottleneck(s), which is typically your internet connection, and 2) maximize network utilization. Congestion happens when packets traversing the bottleneck get piled up (‘queued’) and have to wait a relatively long time to get through. This results in high ping times (known on Twitch as: “high ms”), in-game lag, and generally sluggish performance of any software that directly or indirectly uses the Internet. The way to avoid congestion at the bottleneck is to avoid sending more traffic than can pass through the bottleneck without incurring substantial additional delay. We also want to utilize as much of the bottleneck’s capacity as possible. These are somewhat opposed goals.

As an analogy for the two goals of avoiding congestion and maximizing throughput, imagine pouring stuff in a funnel as fast as possible without allowing any stuff to accumulate in the funnel.

Now imagine performing the above experiment again, but with multiple people pouring stuff in the funnel, each starting and stopping at random times with no coordination between them. Chances are good that the multi-user experiment won’t do as well at avoiding congestion and /or maximizing throughput. For the same reason, the only way to reliably do traffic shaping to avoid network congestion is to do it at the edge of the network (such as a router), where traffic shaping policies can account for and apply to all traffic that traverses the bottleneck.

Posted in OS X, scripts, The More You Know, tutorials | 3 Comments

Nuke and pave of pfSense on the SG-2440

I may not be the first to deal with the fallout of filesystem corruption on an SG-2240 running pfSense 2.2 due to sudden power loss, but I might be the first to put the cliff notes of the recovery process in one place.

The first obvious symptom of trouble is the web admin throwing http 500 and 503 errors. Research reveals that this problem is not completely rare, and is often caused by unclean shutdowns resulting in filesystem damage. If you’re feeling unsettled about why a tiny fan-less network appliance such as this would be so grumpy about power loss, be advised that pfSense has another mode where things can lose power safely because the ‘non-volatile’ file systems like / are mounted r/o, and volatile ones like /var/log, /tmp, /var/run are on ramdisk. The SG-2240 does not use this mode by default, it uses the ‘full install’ mode, which behaves much more like a standard FreeBSD system, so you’re supposed to shut it down like a nice person. Read up on the difference between the pfSense “full install” and “nanobsd” configurations.

Since the web admin is dead, to diagnose this further we’ll use the console port.

Accessing pfSense console port from OS X

Accessing pfSense console port from Windows

  • Connect mini-usb cable between pfsense console port and windows machine.
  • install USB to COM bridge driver found here: http://www.silabs.com/products/mcu/Pages/USBtoUARTBridgeVCPDrivers.aspx
  • open Device Manager -> Ports
  • locate Silicon Labs USB bridge COM listing. Note the number after COM, e.g. COM3
  • boot firewall
  • fire up Putty, make a new serial connection with a speed of 115200 using the COM port discovered previously
  • press enter. You should have a root shell.

Once consoled in, I ran /etc/rc.initial to use the ‘Restart PHP-FPM’ command to try to reboot the web stuff, as I read this worked for some folks. It emitted some nonsense about not knowing what the wheel group means. A cursory glance around town shows that /etc/group, /etc/passwd, and /etc/master.passwd are all munged. Not good.

Reinstall pfSense

  • Download a memstick image from pfsense. Choose the ‘netgate’ option from the Computer Architecture menu, since apparently the SG-2440 is a netgate.
  • Prepare a USB stick with install media.
    • Insert a USB stick (into your workstation, in this case a mac) that you don’t mind erasing.
    • If any filesystems on the usb stick are mounted, unmount them (but do not eject the device) – you can do this with Disk Utility by selecting the volumes and clicking “Unmount”.
    • Find the USB stick device number with: diskutil list
    • Wipe the partition table on the USB stick with dd:
      sudo dd if=/dev/zero of=/dev/disk3 bs=1m count=1

      (assuming the USB stick is /dev/disk3)

    • Copy the image to the device:
      gzcat pfSense-memstick-ADI-2.2.2-RELEASE-amd64.img.gz | \
      sudo dd of=/dev/disk3 bs=16k
  • Eject the USB stick and insert it into one of the pfSense USB ports
  • Boot the pfSense box.
  • Shortly after boot, you are prompted to press F12 if you want a boot menu. Do it.
  • You should now see a list of storage devices; select the USB stick.
  • Let the next menu pass you by; don’t choose anything:
  • After a bit more booting, you will be given the chance to press ‘i’ to run the installer. Do that.
  • From the next menu, accept the console settings.
  • Choose “Custom Install”
  • Select the Generic Ultra HS-Combo Disk as the target for the installation
  • Choose “Format this Disk”
  • Choose “Use this Geometery”
  • Format da1
  • Skip the custom partitioning step
  • Accept and install Bootblocks
  • Select the internal drive
  • Accept and Create
  • Watch the progress window
  • Embedded Kernel
  • Reboot
  • No VLANs
  • Name each of the four network interfaces igb0, igb1, igb2, igb3 for WAN, LAN, OPT1, OPT2 respectively.
  • Type ‘y’ to finish.
...
pfSense (pfSense) 2.2.2-RELEASE amd64 Mon Apr 13 20:10:22 CDT 2015
Bootup complete
FreeBSD/amd64 (pfSense.localdomain) (ttyu1)
*** Welcome to pfSense 2.2.2-RELEASE-pfSense (amd64) on pfSense **
 WAN (wan) -> igb0 -> 
 LAN (lan) -> igb1 -> v4: 192.168.1.1/24
 OPT1 (opt1) -> igb2 -> 
 OPT2 (opt2) -> igb3 -> 
 0) Logout (SSH only)               9) pfTop
 1) Assign Interfaces              10) Filter Logs
 2) Set interface(s) IP address    11) Restart webConfigurator
 3) Reset webConfigurator password 12) pfSense Developer Shell
 4) Reset to factory defaults      13) Upgrade from console
 5) Reboot system                  14) Enable Secure Shell (sshd)
 6) Halt system                   15) Restore recent configuration
 7) Ping host 16) Restart PHP-FPM
 8) Shell
 
Enter an option:
Posted in bit bucket | 1 Comment

PKTAP extensions to tcpdump in OS X

The tcpdump man page in OS X contains various references to something called PKTAP, such as in the documentation for the -k option:

 Control the display of packet metadata via an optional meta-
 data_arg argument. This is useful when displaying packet saved
 in the pcap-ng file format or with interfaces that support the
 PKTAP data link type.

 By default, when the metadata_arg optional argument is not spec-
 ified, any available packet metadata information is printed out.

 The metadata_arg argument controls the display of specific
 packet metadata information using a flag word, where each char-
 acter corresponds to a type of packet metadata as follows:

 I interface name (or interface ID)
 N process name
 P process ID
 S service class
 D direction
 C comment

 This is an Apple modification.

This sounds like fun, but my attempts to use this were foiled by the fact that none of my interfaces support the PKTAP data link type.

If I had searched the man page for other references to PKTAP, I would have learned that tcpdump can create a ‘virtual’ PKTAP interface that wraps a specified list of other interfaces. All those other interfaces are visible through this PKTAP interface, and all the associated metadata is available for viewing / filtering.

e.g. to view only packets sent or received from ssh processes, and also view the additional metadata (-k)

andre@flux [~] % sudo tcpdump -tknq -i pktap,en0 -Q "proc =ssh" 
tcpdump: data link type PKTAP
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pktap,en0, link-type PKTAP (Packet Tap), capture size 65535 bytes
(en0, proc ssh:44637, svc BE, in) IP 173.255.247.120.4200 > 192.168.5.82.65047: tcp 180
(en0, proc ssh:44637, svc CTL, out) IP 192.168.5.82.65047 > 173.255.247.120.4200: tcp 0

To simply view all of the PKTAP metadata on all packets, try something like the following (substituting en0 for your active interface(s)):

sudo tcpdump -q -n -i pktap,en0 -k

The PACKET METADATA FILTER section of the man page describes the various filtering controls.

It seems like this PKTAP stuff is used by default when doing packet captures on iOS using the provided tools. Wireshark also supports PKTAP, and had a few words about Apple’s implementation :)

Posted in OS X, Pro Tip, The More You Know | 1 Comment

Newpro is boss

2011 was drawing to a close, and I was uneasy at the lack of a Mac Pro refresh. My 2009 MacPro4,1 was still performing admirably, but video workflows were starting to feel sluggish as I incorporated more high-frame-rate content. Tasks like video encoding were almost as fast on laptops shipped earlier in 2011 as they were on my Big Bad MacPro.

time avexporter -dest ~/t -replace -preset AVAssetExportPreset1920x1080 -source ~/Movies/wow.mov

# MacBookPro8,2 Intel Core i7 2820QM @ 2.3 Ghz / 32nm “Sandy Bridge” / Early 2011 (thor)
326.51s user 11.64s system 447% cpu 1:15.55 total
325.68s user 11.63s system 448% cpu 1:15.29 total

# MacPro4,1 Intel Quad Core Xeon W5590 @ 3.3 Ghz / 45nm “Nehalem” / Early 2009 (rune)
324.57s user 10.70s system 451% cpu 1:14.24 total
323.17s user 10.56s system 451% cpu 1:13.97 total

Even in 2011, I figured the Mac Pro tower form factor was not long for this world. Thunderbolt had already arrived, and has turned out to be a pretty nice interconnect for storage, networking, and other high-bandwidth peripherals. Notably, video cards are not among the things that are typically worthwhile to use over thunderbolt (you’d need an external thunderbolt –> PCI chassis, and you’d have to live with sub-par graphics performance). Also keenly detecting a trend away from discrete GPU and towards ‘integrated’ GPUs in newer Macs, I wrote the following letter to a high-level apple executive:

We’ve never met, but I feel compelled to send this note. I know Apple does not comment on rumors, and I fully expect no reply.

*please* don’t further marginalize users who want the best possible GPU performance on ANY platform. Our current Mac Pro GPU options are already pretty bad compared to the rest of the desktop market, and all of our mobile devices still lag far behind high-end desktop GPU performance – even though our best desktop card (the ATI 5870) shipped over 3 years ago (!!), and our best mobile GPUs are much more recent.

While I realize it would probably be very easy to make the business case for not caring about the Mac Pro, please consider the possibility of maintaining it as sort of a loss-leader. The biggest, best Mac Pro attracts power users and power developers alike.

I’m no EE, but I do understand the basics of power / size / heat / performance ratios. Given similar technology, the larger card that uses more power will almost certainly be faster. That’s the one I want, and I’m not alone.

Now… if we can meet or exceed *high-end* desktop performance in a portable package, I’m all for it! Today’s portables aren’t that close, but … who knows what the future holds.

When a refreshed MacPro5,1 tower landed the following year in 2012, I was surprised – at least, until I looked at the specs, which were identical to the 2010 MacPro5,1 tower except for RAM and CPU. In other words, it seemed as though relatively little effort was put into this refresh, and it didn’t go very far in reassuring me that the Mac Pro had a future. “Not dead yet”, I thought.

I kept banging away, hope for the Mac Pro slowly fading over time, until October 2013 when the MacPro6,1 was announced. There were unanswered questions, but what we saw was pretty impressive, and would clearly keep the flame alive. I was extremely pleased to see such a head-on approach to the GPU problem, and it made me feel like my letter two years prior came at an interesting time for the people who built this thing. When I heard of the intended availability (Dec ’13), I probably put on a little sad face, because who would ever intentionally ship a product in December, unless there were larger constraints at play? I expected this date to slip, or for availability to be constrained at first. Which is all fine, because after waiting this long, a few more months seemed like nothing – and the worry was completely evaporated :)

They did manage to ship some new Mac Pros in 2013, but indeed availability was constrained for many months. It wasn’t until the second or third day of WWDC 2014 that the new mac pro was finally made available for purchase by employees at a discount (think: customers first). I placed my order within hours, and it shipped the following day!flux

It’s pretty much a dreamboat, even though single-core workloads are faster on a friggin iMac. The same avexporter test shown earlier clocks in at 1:03 on this mac, after logging about an ‘error loading GPU renderer’. When all the hardware resources are brought to bear (e.g. a stack of 19 effects rendering in real-time without any dropped frames in FCP X), the result is one you totally can’t achieve on an iMac – although you can get close by slapping some fast GPUs into a MacPro5,1.

There’s a fair amount of new architecture in this thing. I feel like MacPro6,1 is waaaay different from anything else Apple ships, and is decidedly ‘off the beaten path’. I have found a couple software oddities here and there that seem unique to this model, but nothing serious. In general, performance and reliability have been very good, and there’s nothing about the hardware that makes me uneasy. It even has a power light! Wow!

Shortly after I got foci, I also picked up a thunderbolt cable, even though the laptop was my first and only thunderbolt device. That cable hung in my closet until a couple hours ago, when I used it to benchmark a thunderbolt <–> thunderbolt network on the new pro.

iperf-tbolt

This leads me to conclude that the original thunderbolt cable is also a thunderbolt2 cable. Fancy!

Custodial note: doing the above thunderbolt test requires connecting the cable across two different thunderbolt controllers, so e.g. from port 1 to 2, or 3 to 4, but not 5 to 6. Use the following diagram:

newpro-tbolt

You’ll then need to create two thunderbolt bridges in the Network prefpane, and map each of the ports you’re using to a different bridge. Click the gear menu at the bottom of the interface list, then select “Manage Virtual Interfaces” to reveal the bridge editor.

tbridge-1

tbridge-2 tbridge-3

Let’s see, what else… ramdisk i/o seems to top out at about 4 GB/s.

The LuxMark OpenCL benchmark tells a good story. MacPro6,1 is shown first, followed by MacPro4,1.newpro-luxmark luxmark-gtx-680-rune

I also picked up a Promise Pegasus2 R6, which benches faster than the iStorage Pro setup it is replacing, even though it’s got 2 fewer spindles.

pegasus2-r6-bench

Posted in mac pro | Leave a comment

SSL enabled, registration disabled

… and now an announcement from the custodial engineering department: SSL has been enabled. Jihad nuclear al-qaeda infiltrate spy NSA snowden very much wow. I updated media links for recent posts (the ones that load on the front page), but not the older ones, so if you visit an old post and don’t see the SSL security indicator, check your browser console to reveal that it’s probably loading an image via http.

Also, user registration has been finally and permanently disabled. I tried various captcha systems over the years to keep spam bots from registering, but none work well enough, and I have tired of deleting spam accounts. To make sure to preserve the accounts for the handful of people who have commented in the past, I use this simple join query in the wordpress db:

SELECT DISTINCT user_login, 
                id 
FROM   wp_users 
       LEFT JOIN wp_comments 
              ON wp_users.id = wp_comments.user_id 
WHERE  wp_comments.user_id IS NOT NULL;

If anyone wants to comment on this or future posts, hit me up on twitter (the link is on your right), or just email me.

Posted in bit bucket | Leave a comment

Storage options for a SAS enclosure owner in a thunderbolt world

I posted the following at creativecow; below is a copy for my archives:

I’ve got an iStorage Pro 8 bay filled with 2 GB drives, and an ATTO R380 running on my 2009 Mac Pro in a RAID5 config (I have since realized that I should be using RAID6 for the more favorable reliability / performance tradeoff), and like many of you I’m eagerly awaiting the new mac pro. I’ve been researching various storage options, and would like to offer these options up for comment and criticism from the group.

My use case is single-user enthusiast-level media production, with the only firm constraint being the ability to sustain a modest 450 MB/s or so write to achieve reliable 120 fps capture of video game footage at 1920 x 1200. My current setup does 500 to 550 MB/s, which is enough, but is slower than it could be, possibly because the R380 is in slot 3 instead of slot 2, so it runs at 4x instead of 8x; I don’t remember why I set it up this way, and I should probably do something about it… Anyway :)

I see three general approaches for moving to a thunderbolt-based mac, two of which retain my existing enclosure and storage: 1) External Thunderbolt –> PCI chassis to house my R380, such as perhaps the mLogic mLink, keeping the iStorage Pro, or 2) Retire the R380 and replace it with something like the ATTO ThunderStream SC 3808D, which provides two SAS ports and operates at 6 Gb, still keeping the iStorage Pro, or 3) get an entirely new thunderbolt RAID + enclosure + disks, which would probably offer a performance boost (that I don’t *really* need).

1) I confirmed with ATTO and mLogic that the R380 with the newest drivers (4.0.1) is supported in thunderbolt topologies. An mLogic mLink is fairly inexpensive at about $400, and provides a single PCI slot that seems like it should be fast enough to let the R380 top out. However, I’m slightly concerned about inserting an additional device between the host and the platters; should I expect a significant latency / throughput hit as a result? This option is probably the least flexible and has the shortest useful lifespan of the three, but is also the cheapest by far.

2) An ATTO ThunderStream SC 3808D is more than twice the cost of an mLink at about $1000, and would replace the R380 and operate at 6 Gb instead of 3 Gb. This also leaves the door open for future SAS expansion via daisy chaining. I have no problem leaning on something like a ThunderStream for as long as it’s viable to do so, provided it’s reliable and performant enough.

3) Get a new thunderbolt RAID enclosure including new drives. At first I thought this was vastly more expensive than a SAS-connected solution, but maybe it’s not… To compare: on the one hand we have $3600 for a 3 TB Pegasus2 R8, and on the other hand, $1000 (ThunderStream SC 3808D) + $1400 (another iStorage T8 SAS enclosure) + ~$1200 (8 x 3 GB drives) = $3600. As the price is roughly a wash, going with the newer, faster solution with fewer discrete components is a no-brainer. There is also the fact that my existing drives have all just crossed the 3 year mark – no problems yet, but… check out the data from Backblaze that shows a marked increase in HDD failure rate after 3 years. Yet another reason for me to get a new enclosure + drives: I don’t have enough capacity elsewhere to back up or offload the 7.2 TB of data I currently have (like I said, enthusiast ;), which is somewhat unsettling given that it’s RAID5 and not RAID6.

So, those are the options I’ve been considering. Having written it all out, I think I’m leaning towards biting the bullet on a Pegasus2 24 TB R8, however I’d greatly appreciate any comments / pot-shots / hazing from the crowd.

Posted in mac pro | Leave a comment

irssi auto-op your friends

From the ‘short and sweet’ department, here’s a fast recipe for configuring irssi to automatically op users of your choosing.

cd ~/.irssi/scripts
curl -O https://scripts.irssi.org/scripts/friends_peder.pl
mkdir -p autorun ; cd autorun ; ln -s ../friends_peder.pl

If irssi is already running (and why wouldn’t it be ;), /run the friends_peder script. Otherwise just start irssi and the script will auto-load. Then add some friends.

/run friends_peder
/addfriend -flags o *!*@userfoo.blingding.chatnet.in #macosx undernet
/addfriend -flags o *!*@barman.wonton.blankets.nu #Quake4Life gamerznet

The addfriend syntax is:

-!- Friends: /ADDFRIEND [-mask full|normal|host|domain] [-flags 
          <[o][v][c]>] <nick|mask> [<channel> [<chatnet>]]]

You may have noticed the new window that appeared to show the state of your friends list:

friends_peder_window

There are some other commands, which you use by specifying an entry number to edit:

CHANNEL <num> <channel>    - set channel
    <channel> is either a channel nam
DELETE  <num>              - delete entr
FLAGS   <num> <flags>      - set flags
    <flags> is a list of c (color), o (give op), v (give voice)
MASK    <num> <mask>       - set 
    <mask> is in the usual nick!use
NET     <num> <net>        - set net
   <net> is one of your defined ircnets or * for all

 

Posted in bit bucket, scripts | Leave a comment

Chasing your tail, and winning! (or: ramdisk soothes latency bumps)

My 8 drive RAID array is a little weird. Throughput is decent, but latency is kinda bad. This causes World of Warcraft for mac to not be able to keep up when capturing in-game video at high data rates (200+ MB/s), which of course is the only way to get large frame size *and* high frame rate :) It took me literally years to finally think of this solution, and after a lot of fine tuning, it works great! The basic idea is to capture video files to a ramdisk and continuously move them to my RAID array during capture. See the script comments for the gory details.

https://gist.github.com/dreness/7293462

Here is an example use of this fancy fast capturing in action. These clips started off as 1920 x 1200 @ 100 fps, which is about 300 MB/s during capture. FCP X was used to re-time the clips and export for YouTube.

Posted in games, media, scripts | Leave a comment

SI File Transfer, Entity Capabilities, iChat, and Your Jabber Bot

What do these things have in common? This is the only place on the entire internet where you can read useful information about all of these things.

I’ve toyed with jabber bots before, mostly using the various XMPP stacks available for Python. This time, I wanted to find something even more high level, and I think I found it in Blather – the example echo bot weighs in at 9 lines of code, including the 2 require statements. Trying to send it a file, however, results in this:

unable-to-receive

I spent a while figuring out what’s required of a jabber bot so that iChat Messages will allow its user to send a file. Researching XMPP file transfer might not be such a mystical odyssey if you were implementing everything from scratch, but if instead you’re trying to (ostensibly) save some time and write as little code as possible, the path isn’t terribly clear. In this case, I ended up learning a fair amount about XMPP. ~20 hours of fervent web searching and reading yielded ~3 lines of code.

I found one solitary post from somebody writing a Blather bot who had this same problem. He was informed that the bot needs to advertise the correct set of capabilities for file transfer, however that guy wasn’t sure what they were. Somebody else chimed in, but that response was broken and also incomplete – but it was enough to put me on the right track. The minimum set of capabilities needed to be a XEP-0096 compliant file transfer recipient  are:

    <feature var="http://jabber.org/protocol/ibb"/>
    <feature var="http://jabber.org/protocol/bytestreams"/>
    <feature var="http://jabber.org/protocol/si"/>
    <feature var="http://jabber.org/protocol/si/profile/file-transfer"/>

Here is a working XMPP bot that can receive files sent by another client, using the XEP 0096 spec. Running it looks something like this:

csbot@botboy[~/cs-bot]ruby xmpp_receiver.rb
Connected to talkman@talk.example.com/1376830693085. Sent capabilities:
<iq type="result" id="blather0001">
  <query xmlns="http://jabber.org/protocol/disco#info" node="http://dreness.com/csbot#cc5+HwPwVxjmjK9bPjmuvv/Ehh8=">
    <identity name="csbot" type="bot" category="client"/>
    <feature var="http://jabber.org/protocol/ibb"/>
    <feature var="http://jabber.org/protocol/bytestreams"/>
    <feature var="http://jabber.org/protocol/si"/>
    <feature var="http://jabber.org/protocol/si/profile/file-transfer"/>
  </query>
</iq>

Receiving file from dre@xomg.example.com/foci
<file xmlns="http://jabber.org/protocol/si/profile/file-transfer"
  xmlns:ichat="apple:profile:transfer-extensions"
  name="btc-watcher.py"
  size="5387"
  posixflags="000001ED"/>

 

Posted in bit bucket | 7 Comments

Longcuts occasionally form when shortcuts conspire

Saw a tweet about one of those one-touch ‘build me a VM’ things for Oracle server + rails and decided to try it; this one is mostly driven by some software called vagrant, which seems cool enough. It definitely doesn’t seem like it’s supposed to be hard.

Everything was going fine until one of the child scripts reached for an RPM I was supposed to have downloaded and placed in specific spot – which I did. The error is:

err: /Stage[main]/Oracle::Xe/File[/home/vagrant/oracle-xe-11.2.0-1.0.x86_64.rpm.zip]:
Could not evaluate: Could not retrieve information from environment production
source(s) puppet:///modules/oracle/oracle-xe-11.2.0-1.0.x86_64.rpm.zip at
/tmp/vagrant-puppet/modules-0/oracle/manifests/init.pp:100

I looked, and sure enough the file is totally there – I copied it there! Ah ha, but Safari helpfully decompressed it for me, leaving a naked .rpm, which vagrant seemingly isn’t prepared to handle. Ok, whatever… zip it:

cd puppet/modules/oracle/files
zip -r oracle-xe-11.2.0-1.0.x86_64.rpm.zip oracle-xe-11.2.0-1.0.x86_64.rpm

Try again: vagrant destroy ; vagrant up. Faile. This time, the error is:

err: /Stage[main]/Oracle::Xe/Exec[alien xe]/returns: change from notrun to 0 
failed: /usr/bin/alien --to-deb --scripts Disk1/oracle-xe-11.2.0-1.0.x86_64.rpm 
returned 2 instead of one of [0] at 
/tmp/vagrant-puppet/modules-0/oracle/manifests/init.pp:129

Uh… what? two instead of one of zero. Ok… As I was reading the zip man page, I remembered that I totally don’t like zip, leading to the immediate assumption that even though zip exited zero, it didn’t produce output that vagrant was expecting. In a little rodent voice, my mind said “probably some kind of silly directory nesting crap”.

I revoked Safari’s archive expanding credentials and re-downloaded the rpm.zip, and copied that zip file into the puppet staging area. This time it all worked!

notice: Finished catalog run in 1401.84 seconds

Oracle is all ready to go!

{56} andre@foci [work/rails-dev-box-runs-oracle] % vagrant ssh
Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic x86_64)
 * Documentation:  https://help.ubuntu.com/
Welcome to your Vagrant-built virtual machine.
Last login: Fri Sep 14 02:23:18 2012 from 10.0.2.2
vagrant@rails-dev-box:~$ sqlplus
SQL*Plus: Release 11.2.0.2.0 Production on Tue Jul 30 00:12:34 2013
Copyright (c) 1982, 2011, Oracle.  All rights reserved.
Enter user-name: ^C
vagrant@rails-dev-box:~$ tnsping localhost
TNS Ping Utility for Linux: Version 11.2.0.2.0 - Production on 30-JUL-2013 00:16:24
Copyright (c) 1997, 2011, Oracle.  All rights reserved.
Used parameter files:
Used HOSTNAME adapter to resolve the alias
Attempting to contact (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=))(ADDRESS=(PROTOCOL=TCP)(HOST=127.0.0.1)(PORT=1521)))
OK (0 msec)

Looking back in Finder, I did eventually notice that the zip file expands to a directory called Disk1, from which I copied the rpm to the puppet staging area without realizing I was not copying from the downloads folder.

nested things...

I used “Show in Finder” from the safari downloads list to get there (which can be done successfully without reading anything, since the position is dependable), and somehow didn’t notice the extra nesting; and I used drag-and-drop to insert the file path into a waiting Terminal window to cp it into the puppet area, so I didn’t ever type “Disk1”. Drive safe!

Posted in Pro Tip, The More You Know | Leave a comment