Category Archives: Etc.

2014-03-08 10.02.59

Fadecandy on the Making Embedded Systems podcast

I recently had the opportunity to speak with Elecia White on the Making Embedded Systems podcast:

Micah Elizabeth Scott (@scanlime) came to talk about Fadecandy, a really neat way to control smart LEDs (NeoPixel, AdaFruit’s term for the WS2812). The conversation ranged from beautiful LED control algorithms and open source embedded projects to triangle tessellations, art, and identity.

You can get the full podcast on, iTunes, or download the MP3.


Sifteo and San Francisco

If you’ve been following me on Twitter you’ve probably noticed I’m not dead yet, despite the extreme silence on this blog lately. Well, possibly I just turned into a really inanely chatty zombie. But either way, I haven’t been cranking out awesome projects like mad. What gives??

First big event: this past July, I moved to San Francisco. Having grown up in the suburbs all my life, I’ve always wanted to live in a big city. So, I decided to pull up roots a bit and make that happen. So far, I feel like it’s one of the best decisions I could have made. If you also happen to live in SF, drop me a line. I’d love to meet, perhaps for an IPA or Trippel at Toronado or a fantastic soy latte at Ritual.

Then, a bit of a career shift. For the past six years, I’ve done a variety of operating systems, graphics, USB, and Bluetooth work at VMware. It’s been awesome, and I had the chance to build some great things there that I still get to use every day when I run VMware Fusion. But it was time for a change. I really wanted to work at a startup, and do something more hardware-focused.

So, very shortly after moving to SF, I got a job with Sifteo, makers of the adorable little gaming cubes that everybody loves. I arrived at the company shortly before their cubes went to market, so I didn’t have much time to influence that product much, but I’ve been having a fantastic time doing future R&D work for them lately. We have some seriously fantastic stuff in store, and I can’t wait until I can share it with everyone.

And finally, as if that were too few New Things, I’m now one week in to a month-long trip to Thailand! I’m loving it so far. I discovered that I totally suck at trip logs, but luckily my travelling companion is a rather awesome amateur photographer, and he’s been keeping a blog at

With any luck, after I get back to the states I’ll have some more time for personal projects. Perhaps I’ll even get a chance to finish something I started.


Another Reason to Beware Bargain Basement Bluetooth

I was debugging a Bluetooth-related problem on a Windows 7 machine recently, and I found another great example of why sometimes you get what you pay for, even when buying something as nominally standardized and homogeneous as a Bluetooth adapter. It so happens that this machine was using one of these $5 adapters with the fake antenna:

That was the first thing I noticed about these adapters. The second was that each and every one of them had the same address, 11:11:11:11:11:11. Someone didn’t feel like paying for a MAC OUI.. or an EEPROM.

Anyway, back to the present… I had been noticing that sometimes when sending a file to my cell phone from a Windows 7 machine, it would time out. It didn’t happen every time, but it seemed to time out more often than not. I dig a little deeper, and it looks like the file transfer application (fsquirt.exe) is never even getting a chance to open a connection to the phone! It is timing out far before that even happens. What’s the deal?

To make sense of this, it helps to understand a bit more about how the Bluetooth stack works in Windows 7. First a big caveat: I do not work at Microsoft, nor do I have any confidential information about the inner workings of Windows. All I have, really, are informed guesses based on my observations and perhaps a little reverse engineering to satisfy my curiosity.

The main actors here are:

  • fsquirt.exe — The GUI wizard which walks you through sending a file to a Bluetooth device. It lets you choose a device, then it sends the file via the standard OBEX protocol.
  • bthprops.cpl — Technically this is a control panel, but for historical reasons it also implements the public API for Microsoft’s Bluetooth stack. This includes entry points for enumerating attached radios, inquiring for nearby devices, and setting up service discovery records for the local machine. Applications can link to this as a DLL.
  • fdbth.dll — Microsoft’s applications, like fsquirt, actually don’t use the public APIs from bthprops.cpl to inquire for nearby Bluetooth devices. Those public APIs are synchronous: they don’t return until the Bluetooth inquiry is totally finished. It is of course much more user-friendly to display Bluetooth devices as they are discovered, and Microsoft’s apps do this by using an undocumented Function Discovery provider.
  • bthserv.dll — Some operations in bthprops.cpl go directly to the bthport driver using device ioctls, but there is a system-wide service which maintains a database with the state of all known Bluetooth devices. It’s also responsible for maintaining local SDP records. The APIs in bthprops.cpl communicate with bthserv.dll via local RPC calls.
  • bthport.sys — This is the kernel-mode driver which does most of the work. It receives remote name requests, inquiries, and many other operations in the form of ioctls from user-space code.

The important fact I learned while investigating this bug: Inside bthport.sys, there is a global FIFO queue of actions which contact remote devices. If anyone asks to read a device’s name, to perform an inquiry, or to establish an ACL connection with a remote device, it goes in this queue. From Microsoft’s perspective this probably simplifies handling unreliable Bluetooth radios quite a lot, since we never rely on the radio to handle more than one of these operations at a time. Additionally, Inquiry operations are rather heavy-weight, and depending on your link manager settings, you may not be able to do anything else while they’re in progress.

So, back to the hang. When I traced the communications between the USB dongle and the Windows 7 Bluetooth stack, I found this:

The setting: fsquirt has just finished the wizard pane where we’re looking for nearby devices. It notifies fdbth that it no longer needs to keep looking. But in its haste to prettify the UI as much as possible, it has already started asking for human-readable names for all of the Bluetooth devices within earshot. Those requests go into bthport’s queue. The fsquirt app wants to establish an ACL connection to my phone already, but it’s waiting in line behind one of these name requests.

In case you don’t have the Bluetooth spec memorized (I sure don’t), here’s a play-by-play of the USB log:

  • OUT 05 04… — The PC asks the Bluetooth dongle to get a human-readable device name for one of the nearby devices.
  • IN 0F… — Command is in-progress.
  • (Over 20 seconds elapse)
  • IN 03 0B 08… — Connection request finished, error code 08 (Timed out)

But wait! We never asked the device to create a connection! We asked it to read the other device’s name! This looks like a pretty grievous firmware bug in this decrepit little dongle. As we see below, Windows is pretty confused too. My “waiting” and “timed out” notes indicate the state of the fsquirt wizard. The timestamps aren’t saved in this log, but fsquirt is only willing to wait 10 seconds or so before giving up. It’s waiting on bthport, and bthport is waiting on any response, even a timeout, from the Bluetooth adapter.

And it keeps waiting, for 70 seconds. Why so long? It should never take that long for a Bluetooth device that’s actually present to respond. Well, actually it isn’t waiting on the remote device. It’s waiting on the radio itself. The radio is programmed to time out after a much shorter interval specified by Windows, in this case about 20 seconds. Windows is relying on the radio to implement this timeout. What we’re seeing here is an even longer failsafe timeout, to catch errors with the local radio. Like the one we just experienced.

When Windows finally gives up on waiting for our buggy radio to respond correctly to the remote name timeout, we can see it issue the next command. This time it’s the Create Connection command that fsquirt had been waiting on. But fsquirt is already long gone, and the user is already frustrated.

If we’re lucky, the user throws the buggy Bluetooth dongle out the window, not their PC.


Espresso Machine Sous-vide

This project is a bit different from what I normally do. No firmware to write, just several trips to the hardware store and a lot of quality time with the drill, files, and nibbler. But this one has a higher purpose: Food.

It was actually a commission by a friend of mine. He wanted to try sous-vide cooking, but neither of us were crazy about the idea of paying for something lab-grade, and even the consumer-focused SousVide Supreme seemed kind of lame with its $450 price tag and no circulation pump.

The interwebs are brimming with do-it-yourself solutions for sous-vide cooking, ranging from the incredibly simple and cheap, through a middle-ground of modified rice cookers and crockpots up to other DIY immersion circulators like this project by Scott at Seattle Food Geek.

The easier solutions (like a modified crock pot) didn’t seem to meet my friend’s requirements. He wants the flexibility to use large water baths sometimes, and he wants it to be stable for long durations. So, a real immersion circulator starts to sound awesome for this. I was really close to building the Seattle Food Geek DIY circulator above, when I noticed one of his ideas for improving the design:

“After burning out my first set of heating coils, I realized that there must be a better method of heating the water. The coils are very effective and heat the water very quickly. However, I’m pretty paranoid about burning them out again, and they’re a pain to replace. I’ve found some commercial immersion heating elements, but they’re about $100, which inflates the budget for this project by quite a bit. I may try using the heating element and pump system from an old espresso machine, the kind that makes steam. Since it already has a self-contained heater and an pump, it might even be cheaper than the heating coils and aquarium pump.”

Well, this is what I ended up doing. I’m not sure it ended up being cheaper (those heating coils he used were really cheap!) but it was definitely fun to build, and the end result is working pretty well:

Top and side panels, in use

DIY circulator in action

Sous-vide Chicken

Sous-vide salmon, carrots, and beets

Rough bill of materials:

  • Kalorik EXP-20737 Aqua Line 1250-Watt 15-Bar-Pump Espresso Maker“, was $59.99 on Amazon
    • Pump
    • Heating element
    • Assorted hoses, fittings, thermostats, and wiring
  • Enclosure: Sheet metal cash box, $10 at Fry’s
    • Cut holes with a power drill and nibbler tool.
  • JLD612 PID Controller, $32.50 from Lightobject
  • PT-100 Temperature Probe, $19 from Lightobject
    • These probes are NOT waterproof, and the wiring between the cable and the sensor can be a bit flaky.
    • I ended up redoing the wiring, and insulating the probe with PVC hose and silicone kitchen/bathroom caulk
  • 25A zero-crossing solid state relay, $13.22 from Digi-key
  • Heavy duty 2 foot 15A power cord, $9.85 from Amazon
  • Fiberglass automotive exhaust wrap, $13.95 from Amazon
  • Local homebrewing supply store:
    • 2x male/female 1/4″ quick-disconnects. (Price: way too much. If you’re on a budget, skip these.)
  • Local hardware store:
    • Reinforced 1/4″ PVC hose, rated for potable water. 10 feet for $10.
    • Non-reinforced 1/4″ PVC hose, for the temperature sensor. 10 feet for $10.
    • Ceiling fan speed control (for adjusting pump power), $10
    • Hose clamps, 6 for about $1 each
    • Silicone kitchen/bathroom caulk
  • Junk drawer:
    • Heavy duty power switch from an old UPS
    • IEC receptacle from a broken power supply
    • Very old Radio Shack panel mount neon indicator light
    • A nicer knob for the pump speed control
    • Heavy-duty wire
    • Lighter gauge wire, for control signals
    • Wire nuts
    • Kapton tape
    • Hot glue
    • Nuts and bolts
    • 1/8″ Audio plug/jack for temperature probe

This turned out to be a really interesting design, because you can use it with nearly any water bath vessel. Even the lab-grade immersion circulators need a place for the circulator to stick out on top, and this often means making a custom lid. I usually use this machine with a small icechest, but I’ve also heated up water in plain soup mugs and mixing bowls. You just run three hoses in (inlet, outlet, and temperature probe) and you’re set. Of course, the more insulation you have, the easier it will be to keep the temperature steady.

On the downside, this design can take a while to heat the water. I think it took maybe 30 minutes to heat about a gallon of room-temperature water up to 140°F. This is certainly not limited by the heating element- the espresso machine heater is 1200 watts. However, I have a mechanical thermostat on it that keeps the heating element from rising too far above 120°C. It’s okay if we make a little bit of steam, but it’d be bad if the steam got too hot. So what ends up happening, is that while you’re initially heating the water, the PID is telling the heating element to run at full power, but the mechanical thermostat is still keeping it from overheating. So when you’re initially heating the water, our limiting factor is the pump’s flow rate. If it’s too slow, the inlet water isn’t coming through fast enough to cool off the heater.

I spent what seems like a week shopping for alternative pumps. I looked through McMaster-Carr, eBay, even tried a CPU water-cooling pump. That one was nicely high flow-rate… until you give it any sort of load, that is. And to be practical, this design really needs a self-priming pump. So, I stuck with the one that came with the espresso machine. It’s not actually that bad. The only tweak I had to make is to limit the pump’s power using a ceiling fan speed controller. Since it’s a solenoid type pump, this only limits the strength with which the solenoid slams from side to side. The actual flow rate is mostly fixed, since the solenoid is oscillating at your mains frequency. The reduced power helps the pump run much quieter and cooler, which is rather necessary when you’re running something continuously that wasn’t meant to.

Since assembling this thing, my boyfriend and I have cooked several meals in it. As long as we’re patient enough to let the PID loop fully settle before cooking anything, it’s remarkably accurate. I can routinely hold the water bath within 0.4 °F or so. Of course, sometimes the PID goes nuts and overshoots/undershoots too. I’m sure I’ll get more feedback on this once I deliver it to my friend 🙂


Announcing vusb-analyzer 1.1

The Virtual USB Analyzer is a graphical tool for analyzing USB sniffer logs. It can do some basic protocol decoding, and it has a graphical timeline view which helps to visualize the latency and concurrency characteristics of USB traffic.

This release only adds a single feature, but it’s one that many people have probably been waiting for: support for Linux’s built-in usbmon sniffer tool.

With usbmon, tracing all USB traffic on your system is as simple as cat’ing text from a sysfs node to a file. It doesn’t have quite as many features as VMware’s low-level capture logs, but for most users (especially users debugging Linux USB drivers) this should be more than sufficient.

If you aren’t familiar with usbmon, the first thing to read is Documentation/usb/usbmon.txt from your kernel sources. That will tell you the basics, including how to capture raw data from the kernel. There are a few other simple front-ends for making this data more human-readable. If you’re using vusb-analyzer, those tools aren’t required. Just save the raw data to a file with a .mon extension, and hand that to vusb-analyzer.

This feature was a patch contributed by Christoph Zimmermann. Without his contribution, I no doubt would have procrastinated this release for many more years 🙂


November Update

I guess I don’t have a complete blog entry to write on any one thing.. but there are several unrelated things that I feel like writing down. I recently joined Twitter, but I must suck at using it because none of these things ended up there.


Tragic. Introspective. Complicated. This isn’t really a place where I would feel it appropriate to go into detail, but it would be unjust to omit given that the other things here are significantly less important.


I don’t have a huge amount of flying time on it yet, but I’ve been enjoying the Blade mSR so far. I’ve been flying it mostly in smallish spaces, so I haven’t had the guts to really see how maneuverable it is- but I’m impressed with its stability and responsiveness so far. Also, either I’m a totally awesome pilot, or (more likely) it’s actually pretty easy to fly.


I haven’t had much time to work on the DSi lately. I’ve let Bushing borrow my hardware, and he’s been working on producing a small quantity of additional RAM tracing/injecting setups. This isn’t the kind of thing we can make many of (I estimate it takes about 6 hours to solder one together) but it’d be helpful for the community if more than one of these rigs existed in the world.


What time I have spent on DSi-related stuff has been on Temporal Hex Dump. It’s actually coming along okay- I took a bit of a development detour to improve it’s Mac OS support after I bought a MacBook Pro… but the next step is to add the part of it that is actually a hex dump. So far, it has the timeline navigation widget and a correlated list of memory transactions:

Temporal Hex Dump

Robot Odyssey

So, way back in April, I posted the first screenshots of Robot Odyssey on the Nintendo DS. This was I guess what I’d call a “binary port”. It runs the original Robot Odyssey binaries, via static binary translation, and with many modifications and UI enhancements to adapt the game for the DS console.

I never got around to posting pictures, but I actually did make significantly more progress on this project before getting distracted. I ended up implementing a pretty sophisticated system for patching the game and dynamically manipulating its world state, as well as “forking” additional copies of the running game very quickly for the purposes of having those forked-off copies render additional sprites. What is this good for? Well, I could have the game proper running on the DS’s top screen, while the bottom screen shows status icons for each robot, where the icons accurately depict the state of the robot’s bumpers and thrusters, and even what item the robot is holding at the time. I also had a mostly complete game load/save UI implemented, including a live preview that shows you where you saved while picking a game to load. My next step was going to be a UI for editing circuits inside your robots using the stylus.

Anyway, I know I’m being cruel by describing all these great features and not posting a video or a ROM image. Sorry. I promise I’ll make a video one of these days when I have a bit more time. As for a ROM image, right now the best I have is the source code for a tool which will generate a ROM given the original Robot Odyssey files. You can get the source with:

svn co

To compile it, you’ll need devkitARM and Python. You’ll also need to put your original Robot Odyssey files (the DOS version, not Apple!) in the “original” directory in the source tree. You can leave them in a zip file, or strewn about the directory in random places. The translator knows what files it needs, and it will find them by SHA-1 hash.

I give no guarantees as to the current state of the source tree- I haven’t worked on it in a few months, and I think I last left it in a state where loading and saving were partially but not usably done. If you aren’t a developer, the source tree isn’t likely to be useful to you yet.

Homework assignment: If anyone in the audience wants to see this Robot Odyssey DS port turn into a real homebrew ROM image that anyone can download without compiling it themselves, we need to have redistribution approval from the game’s current copyright holders. Problem is, I have no idea who that is. I did email Warren Robinett about this project a while back, but he couldn’t tell me who currently owned the copyright. Following the trail of acquisitions, I think it might be owned by Houghton Mifflin Harcourt. Yeah, the textbook company. I did email them about the Robot Odyssey copyright multiple times, but I never got a reply. So, what now?

One approach is always to release it then look for the return address on the cease & desist letter. But, in all seriousness, I would like to be able to release a legal port of a classic game which the current copyright holder clearly has no commercial interest in. It seems like there should be some way to do this.


New MacBook Pro + Fyre

Back when I bought my last Mac: USB and the Playstation 2 were new, Altivec and shiny plastic were in, and fans were almost starting to line up for the first Harry Potter movie. Oh, and warrantless wiretapping was still illegal. So, Apple, I guess it’s time to put all those hardware failures behind us and see if this relationship can work.

I’ve actually had a lot of fun hacking on Mac OS at work while developing VMware Fusion. Working on such a low-level software project, you see a lot of the ugliest parts of any OS. And there’s certainly a lot inside Mac OS that scares me. But there’s also a lot of cool technology in there. And best of all, I don’t have to futz with my ALSA settings for an hour if I want to watch a YouTube video. Win!

I’d been thinking about upgrading my aging Thinkpad to a 15″ MacBook Pro when it finally kicks the bucket. But darnit, those things are too reliable. So I needed a different excuse. I think it finally boiled down to:

  1. Faster CPU (and 6MB L2 cache.. drool…)
  2. More memory (4GB standard in a laptop. We truly do live in the future.)
  3. Runs Mac OS, Linux, and Windows- so I can develop software for all three
  4. Ooh, shiny!

It arrived this morning, and indeed it is quite shiny. I really do like the new Unibody design- and the large trackpad surface with pseudo-button is actually quite a lot nicer than the standard touchpad on my Thinkpad. (I’ll see how long I can resist the urge to do some low-level poking around in the touchpad, to see how its multi-touch protocol works…)

I think I’m a bad Mac user though. Just about the first two things I did were to install the Ubuntu 9.10 RC, and Windows 7. Then I had to continue my tradition and render a new background image:

Rendering some wallpaper for my MacBook Pro

This is using Fyre‘s cluster rendering support. The GUI and one rendering node are running in a 2-VCPU 64-bit virtual machine on my MacBook Pro. I have two more nodes running on a desktop Athlon64 machine running Linux natively. The total speeds here are pretty impressive- comparable to what David was able to get from running cluster nodes on every machine in one of the CS labs back in college. Isn’t Moore’s Law great? (And I’m sure the 6MB L2 cache helps :))


Graphics in VMware Fusion 3 and Workstation 7

I work on the graphics virtualization team at VMware. The company is about to release two new desktop virtualization products: Fusion 3.0 is in beta, with a release coming tomorrow. Workstation 7.0 has a public release candidate available.

There are a lot of exciting features in these releases, and my team has been working really hard on making the graphics virtualization in these releases the best we can. Our focus with these releases has been introducing a few large architectural changes in our graphics stack:

  • A brand new WDDM graphics driver for Windows 7 and Vista.

This is a complete rewrite, sharing no code with our Windows 2000/XP driver. This has been a monumental undertaking for the team, but the results have been pretty shiny. The architecture is clean and extensible, it’s much easier to understand and debug the driver’s performance, and it gives us a good platform to build on for future guest driver releases. Oh, and it can run 3D games with Aero enabled.

  • A brand new OpenGL driver for Windows guests.

Yes, this release has not one but two brand new drivers in it! The new OpenGL driver is based on Gallium, and it finally brings both major graphics APIs into VMware virtual machines. Both of our new drivers really deserve a blog post all to themselves.

  • We’ve been progressively rearchitecting the host-side infrastructure with a project we call “Better Unity and Multi-mon”.

Whereas previous releases of VMware’s products have implemented multi-monitor support and Unity mode on top of an architecture that was fundamentally designed for a single framebuffer, Fusion’s entire graphics stack has been overhauled in order to natively handle multiple screens, windows, and framebuffers. This architecture is common to all of our products, but so far only Fusion 3.0 has had its entire stack updated. Workstation and ESX still use a blend of new and legacy components.

As an end-user, the effects of our Better Unity and Multi-mon project are relatively subtle. Unity mode now renders much more efficiently, and applications like games and media players will run just as fast in Unity mode as they will in single window or fullscreen mode. Window resizing is smoother. We also have a high-quality GPU accelerated scaler which you can see in action on your VM’s live thumbnails in the VM Library window.

Screen Object

Back in April, I announced the VMware SVGA Device Developer Kit. This is an Open Source (MIT-style license) project which documents the virtual hardware interface that our graphics stack uses, and provides a simple reference driver and some examples. If you’re interested in operating system development, graphics, or virtualization, it can be a handy platform to experiment with, and it’s the best starting point if you’re trying to write a VMware graphics driver for a new operating system.

I’ve been trying to maintain the DDK with an open development process. Instead of periodically dumping code, all development takes place directly in the open repository. The one exception to this policy is for code which doesn’t work on released VMware products, either due to serious bugs or unreleased features.

As part of our architectural improvements for Fusion 3 and Workstation 7, we added a pretty big new feature to the SVGA device: “Screen Object”. Without Screen Object, the device has a single framebuffer with a geometry controlled by a set of registers. Multi-monitor support was faked by using an additional set of registers to carve up a single large rectangular framebuffer into individual monitors. With Screen Object, the guest can create “screens” dynamically, and they don’t necessarily need to be backed by any specific kind of memory.

A Screen is really an opaque non-guest-addressable memory object, just like an SVGA3D surface. If a Surface is analogous to a texture or a backbuffer, a Screen is analogous to a frontbuffer. You can blit between system memory and a Screen, and you can blit from Surface to Screen.

This is a simple concept, but it’s a powerful tool that can be used to implement a variety of different memory management and asynchronous rendering strategies. Last week I checked in a pretty big update to the DDK which includes refdriver support for Screen Objects, and 10 new Screen Object-only examples.

To run the examples, you’ll need Fusion 3 or Workstation 7. You’ll also need to manually enable Screen Object. Like many new features, we haven’t enabled Screen Object by default right away. It is enabled automatically for Vista, Windows Server 2008, Windows 7, and Mac OS X guests, since our drivers on those platforms may take advantage of the new features. For improved portability, it is disabled by default on other guest OS types. You can enable it manually by adding a line to your VMX file:

svga.enableScreenObject = TRUE

As always, the DDK is provided as-is, with no official support from VMware. So please don’t bug our support folks 😉 But I’ll try to answer questions as time permits if you send them to micah at Thanks!