the bits

As we consider a mac user’s renewed quest for GPU performance – this time for an ‘external’ GPU in a tbolt2 PCI chassis – we find similarities to other timeless quests. For instance, in our quest, the path is not clear in the beginning, and there is conflicting advice about how to proceed. The establishment is against us, and success may be temporary due to an ever-shifting and occasionally hostile landscape. The journey is fraught with peril, and you fully expect impasses surmountable only through deep soul searching and great courage. Also we have to assemble items from the marketplace and maybe perform a heroic deed.

My first thought was that tbolt2 wouldn’t have the bandwidth to let a fast GPU to shine. While it is true that a fast card would be limited by tb2, it’s still totally fast enough to outperform the MacPro6,1 d700s for many workloads. From the barefeats post:

Even when ‘hobbled’ by the limited bandwidth of Thunderbolt 2, the eGPU TITAN X ‘buries’ the AMD FirePro D700 on this OpenGL test.

The Diablo III results in that same post are even crazier. The new mac pro gets 78 fps with the internal d700 and 124 fps with a geforce titan x in an external tb2 chassis. Pretty not bad. Here’s the kicker: a six year old 2010 Mac Pro scores 167 fps (with the card installed in a legacy PCI slot. I mean a PCI slot. Heyo.)

My task is to pick a set of tradeoffs, optimizing in order for: performance, build simplicity and cleanliness, [ergo | cost]. The most commonly used thunderbolt2 chassis for hosting GPUs seems to be the Akitio unit, even though it’s a bit too small for many cards (but you can bend it and / or not close the back hatch), and the power supply is too weak to push a decent card. On the up side, the $200 – $300 price is comparatively low. The Akitio case doesn’t seem well equipped to power a fast GPU, and many of the builds I can find just have the components splayed out on the desk…

Others decide to transplant the Akitio board into a larger case with enough room for an ATX power supply and a full size card or two, and without leaving the back door open.

*

I once was told a very short story about how GPU drivers are all dens of mutual patent infringement, where everyone is guilty and they all just keep it ‘secret’ and carry on. Doing unsupported things with GPUs tends to require some negotiating with and gentle coercion of your computer system. Learning the secret handshakes in the first place is a mysterious business, and it’s easy to imagine all manner of unsavory behavior and sundry cut-throat affairs in this trade.

Our quest has produced one such tale already. There’s a person called netkas who did a lot of the groundwork in bootstrapping the “eGPU” scene… operated a forum, was responsive to people, helped them build their rigs, etc. Netkas then offers a service where if you provide diagnostic details from your system while your eGPU stuff is plugged in, you will be told whether your rig is viable. If it is viable, for the price of $20 you will be taught to sing the haunting melody that subdues OS X and brings your GPU to life. This paid service seems like a reasonable way for customers to support the ongoing work of playing cat and mouse with the vendors. People were grateful.

Very shortly after this service started and began yielding happy customers, it is said that a rival player known as goalque (seemingly well regarded by his side of the internet) inspected the work of netkas and generalized it into a rather burly shell script that now sits in goalque’s github repo. It may be executed by anyone for no fee, much to the continued frustration of the netkas camp. The feud lives on, with the scene’s two popular forums (netkas and techinfernal) appearing to come down on opposing sides, all of which is completely inconsequential to the users, who are all either stoked to pay $20 to netkas to light up their rig, or stoked to run a shell script from github to light up their rig.

2011 was drawing to a close, and I was uneasy at the lack of a Mac Pro refresh. My 2009 MacPro4,1 was still performing admirably, but video workflows were starting to feel sluggish as I incorporated more high-frame-rate content. Tasks like video encoding were almost as fast on laptops shipped earlier in 2011 as they were on my Big Bad MacPro.

time avexporter -dest ~/t -replace -preset AVAssetExportPreset1920x1080 -source ~/Movies/wow.mov

# MacBookPro8,2 Intel Core i7 2820QM @ 2.3 Ghz / 32nm “Sandy Bridge” / Early 2011 (thor)
326.51s user 11.64s system 447% cpu 1:15.55 total
325.68s user 11.63s system 448% cpu 1:15.29 total

# MacPro4,1 Intel Quad Core Xeon W5590 @ 3.3 Ghz / 45nm “Nehalem” / Early 2009 (rune)
324.57s user 10.70s system 451% cpu 1:14.24 total
323.17s user 10.56s system 451% cpu 1:13.97 total

Even in 2011, I figured the Mac Pro tower form factor was not long for this world. Thunderbolt had already arrived, and has turned out to be a pretty nice interconnect for storage, networking, and other high-bandwidth peripherals. Notably, video cards are not among the things that are typically worthwhile to use over thunderbolt (you’d need an external thunderbolt –> PCI chassis, and you’d have to live with sub-par graphics performance). Also keenly detecting a trend away from discrete GPU and towards ‘integrated’ GPUs in newer Macs, I wrote the following letter to a high-level apple executive:

We’ve never met, but I feel compelled to send this note. I know Apple does not comment on rumors, and I fully expect no reply.

*please* don’t further marginalize users who want the best possible GPU performance on ANY platform. Our current Mac Pro GPU options are already pretty bad compared to the rest of the desktop market, and all of our mobile devices still lag far behind high-end desktop GPU performance – even though our best desktop card (the ATI 5870) shipped over 3 years ago (!!), and our best mobile GPUs are much more recent.

While I realize it would probably be very easy to make the business case for not caring about the Mac Pro, please consider the possibility of maintaining it as sort of a loss-leader. The biggest, best Mac Pro attracts power users and power developers alike.

I’m no EE, but I do understand the basics of power / size / heat / performance ratios. Given similar technology, the larger card that uses more power will almost certainly be faster. That’s the one I want, and I’m not alone.

Now… if we can meet or exceed *high-end* desktop performance in a portable package, I’m all for it! Today’s portables aren’t that close, but … who knows what the future holds.

When a refreshed MacPro5,1 tower landed the following year in 2012, I was surprised – at least, until I looked at the specs, which were identical to the 2010 MacPro5,1 tower except for RAM and CPU. In other words, it seemed as though relatively little effort was put into this refresh, and it didn’t go very far in reassuring me that the Mac Pro had a future. “Not dead yet”, I thought.

I kept banging away, hope for the Mac Pro slowly fading over time, until October 2013 when the MacPro6,1 was announced. There were unanswered questions, but what we saw was pretty impressive, and would clearly keep the flame alive. I was extremely pleased to see such a head-on approach to the GPU problem, and it made me feel like my letter two years prior came at an interesting time for the people who built this thing. When I heard of the intended availability (Dec ’13), I probably put on a little sad face, because who would ever intentionally ship a product in December, unless there were larger constraints at play? I expected this date to slip, or for availability to be constrained at first. Which is all fine, because after waiting this long, a few more months seemed like nothing – and the worry was completely evaporated :)

They did manage to ship some new Mac Pros in 2013, but indeed availability was constrained for many months. It wasn’t until the second or third day of WWDC 2014 that the new mac pro was finally made available for purchase by employees at a discount (think: customers first). I placed my order within hours, and it shipped the following day!

It’s pretty much a dreamboat, even though single-core workloads are faster on a friggin iMac. The same avexporter test shown earlier clocks in at 1:03 on this mac, after logging about an ‘error loading GPU renderer’. When all the hardware resources are brought to bear (e.g. a stack of 19 effects rendering in real-time without any dropped frames in FCP X), the result is one you totally can’t achieve on an iMac – although you can get close by slapping some fast GPUs into a MacPro5,1.

There’s a fair amount of new architecture in this thing. I feel like MacPro6,1 is waaaay different from anything else Apple ships, and is decidedly ‘off the beaten path’. I have found a couple software oddities here and there that seem unique to this model, but nothing serious. In general, performance and reliability have been very good, and there’s nothing about the hardware that makes me uneasy. It even has a power light! Wow!

Shortly after I got foci, I also picked up a thunderbolt cable, even though the laptop was my first and only thunderbolt device. That cable hung in my closet until a couple hours ago, when I used it to benchmark a thunderbolt <–> thunderbolt network on the new pro.

This leads me to conclude that the original thunderbolt cable is also a thunderbolt2 cable. Fancy!

Custodial note: doing the above thunderbolt test requires connecting the cable across two different thunderbolt controllers, so e.g. from port 1 to 2, or 3 to 4, but not 5 to 6. Use the following diagram:

You’ll then need to create two thunderbolt bridges in the Network prefpane, and map each of the ports you’re using to a different bridge. Click the gear menu at the bottom of the interface list, then select “Manage Virtual Interfaces” to reveal the bridge editor.

Let’s see, what else… ramdisk i/o seems to top out at about 4 GB/s.

The LuxMark OpenCL benchmark tells a good story. MacPro6,1 is shown first, followed by MacPro4,1.

I also picked up a Promise Pegasus2 R6, which benches faster than the iStorage Pro setup it is replacing, even though it’s got 2 fewer spindles.

I posted the following at creativecow; below is a copy for my archives:

—

I’ve got an iStorage Pro 8 bay filled with 2 GB drives, and an ATTO R380 running on my 2009 Mac Pro in a RAID5 config (I have since realized that I should be using RAID6 for the more favorable reliability / performance tradeoff), and like many of you I’m eagerly awaiting the new mac pro. I’ve been researching various storage options, and would like to offer these options up for comment and criticism from the group.

My use case is single-user enthusiast-level media production, with the only firm constraint being the ability to sustain a modest 450 MB/s or so write to achieve reliable 120 fps capture of video game footage at 1920 x 1200. My current setup does 500 to 550 MB/s, which is enough, but is slower than it could be, possibly because the R380 is in slot 3 instead of slot 2, so it runs at 4x instead of 8x; I don’t remember why I set it up this way, and I should probably do something about it… Anyway :)

I see three general approaches for moving to a thunderbolt-based mac, two of which retain my existing enclosure and storage: 1) External Thunderbolt –> PCI chassis to house my R380, such as perhaps the mLogic mLink, keeping the iStorage Pro, or 2) Retire the R380 and replace it with something like the ATTO ThunderStream SC 3808D, which provides two SAS ports and operates at 6 Gb, still keeping the iStorage Pro, or 3) get an entirely new thunderbolt RAID + enclosure + disks, which would probably offer a performance boost (that I don’t *really* need).

1) I confirmed with ATTO and mLogic that the R380 with the newest drivers (4.0.1) is supported in thunderbolt topologies. An mLogic mLink is fairly inexpensive at about $400, and provides a single PCI slot that seems like it should be fast enough to let the R380 top out. However, I’m slightly concerned about inserting an additional device between the host and the platters; should I expect a significant latency / throughput hit as a result? This option is probably the least flexible and has the shortest useful lifespan of the three, but is also the cheapest by far.

2) An ATTO ThunderStream SC 3808D is more than twice the cost of an mLink at about $1000, and would replace the R380 and operate at 6 Gb instead of 3 Gb. This also leaves the door open for future SAS expansion via daisy chaining. I have no problem leaning on something like a ThunderStream for as long as it’s viable to do so, provided it’s reliable and performant enough.

3) Get a new thunderbolt RAID enclosure including new drives. At first I thought this was vastly more expensive than a SAS-connected solution, but maybe it’s not… To compare: on the one hand we have $3600 for a 3 TB Pegasus2 R8, and on the other hand, $1000 (ThunderStream SC 3808D) + $1400 (another iStorage T8 SAS enclosure) + ~$1200 (8 x 3 GB drives) = $3600. As the price is roughly a wash, going with the newer, faster solution with fewer discrete components is a no-brainer. There is also the fact that my existing drives have all just crossed the 3 year mark – no problems yet, but… check out the data from Backblaze that shows a marked increase in HDD failure rate after 3 years. Yet another reason for me to get a new enclosure + drives: I don’t have enough capacity elsewhere to back up or offload the 7.2 TB of data I currently have (like I said, enthusiast ;), which is somewhat unsettling given that it’s RAID5 and not RAID6.

So, those are the options I’ve been considering. Having written it all out, I think I’m leaning towards biting the bullet on a Pegasus2 24 TB R8, however I’d greatly appreciate any comments / pot-shots / hazing from the crowd.