Interfacing with The Energy Detective

I recently bought The Energy Detective (TED), a pretty inexpensive and friendly way to keep tabs on your whole house’s electricity usage. It’s a lot like having a more featureful version of your utility company’s power meter, sitting on your kitchen counter. It can estimate your utility bill, and tell you how much electricity and money you’re using in real-time. The resolution is pretty good- 10 watts, 1 second.

As a product, I’ve been pretty happy with it. It does what it claims to, and the measurements seem to be fast and accurate. Of course, being the crazy hacker I am, I wanted to interface TED with other things. I don’t have a home automation system as such, but I did want to get real-time graphs of my electricity usage over various time periods. I also wanted the possibility to use TED as an information feed for various other display devices around the house. (But that’s a topic for another blog post…)

The stock TED package consists of two pieces: A measurement/transmit unit (MTU) and receive/display unit (RDU). The MTU has a pair of current transformers, and it installs entirely in your house’s breaker panel. It takes the power readings every second, and transmits them over the power lines to the RDU, which is just a little LCD panel that plugs into the wall. The RDU has a USB port, which you can use with TED’s “Footprints” software.

So, hoping that the USB port would do what I want, I bought the Footprints software for $45. The TED system itself is, in my opinion, a really good value. The Footprints software is not. As a consumer, I was disappointed by two main things:First of all- the UI lacks any polish whatsoever. It looks like a bad web page from the 90’s. Second of all, the data collection is not done in hardware, it’s implemented by a Windows service. This means you can’t collect data to graph unless you have a Windows PC running. Not exactly a power-efficient way to get detailed power usage graphs.

As a hobbyist, a few more things frustrated me about Footprints. The implementation was pretty amateurish. Their Windows service runs a minimal HTTP server which serves up data from an sqlite database in XML. The front end is actually just a Flash applet masquerading as a full application. Energy Inc, the company behind TED, has an API for Footprints: but you have to sign a legal agreement to get access to it, and I wasn’t able to get any details on what the API does and doesn’t include without signing the agreement. So, I opted not to. It would be much more fun to do a little reverse engineering…

So, I did. The end result is that I now have two ways of getting data out of my TED system.

Using the USB port

The TED RDU’s USB port is actually just a common FTDI usb-to-serial adapter. The RDU sends a binary packet every second, which includes all of the data accessible from the Footprints UI. This includes current power usage, current AC line voltage, utility rates, month-to-date totals, and anything else you’ve programmed into your RDU.

There has been some prior work on reverse engineering this protocol. The Misterhouse open source home automation project has a Perl module which can decode the TED messages.

Unfortunately, the Perl module in Misterhouse won’t work with more recent versions of the RDU like mine. The recent RDUs have a different packet length, and they require a polling command to be sent before they’ll reply with any data.

I found the correct polling command by snooping on Footprints’ serial traffic with Portmon. I also noticed a packet framing/escaping scheme, which explains some of the length variations that would have broken the module in Misterhouse.

The result is a Python module for receiving data from a TED RDU. It isn’t terribly featureful, but it should be pretty robust.

Direct from the wall socket

Now for the more exciting method: What about reading data directly from the power line, without using the TED receive/display unit at all? This could provide some exciting opportunities to embed a small and cheap TED receiver inside of other devices, and it would provide some insight on what exactly is being transmitted by the box in my breaker panel.

The TED RDU is pretty simple internally: A Dallas real-time clock, PIC18 microcontroller, chip-on-glass LCD, some buttons, and the TDA5051A, a single-chip home automation modem from Philips. This chip can receive and transmit ASK modulated signals at 1200 baud, with a carrier frequency of around 132 kHz.

Digi-key carries the TDA5051A, but I figured it would be more educational (and more hobbyist-friendly) to try and build a simpler receiver from scratch using only commonly available parts. The result is the following design, with an 8-pin AVR microcontroller and three op-amps:

Update: There is now a Revision 2 of the schematic, which uses only a single power supply and one op-amp.

  1. The power line is sampled via a 9V AC wall wart. With this design, it needs to be a separate isolated power supply. So far, I haven’t had any luck with using this same transformer to power the circuit. Any rectifier introduces high-frequency harmonics which drastically degrade the signal-to-noise ratio.
  2. The first stage is a 10x amplifier and 138kHz band-pass filter. This is from Texas Instrument’s “Filter Design in Thirty Seconds” cheat-sheet.
  3. The next stage is an amplifier and high-pass filter, to remove the last remnants of 60 Hz ripple.
  4. The third stage is just for amplification. In a design which used higher quality op-amps, this stage may not be necessary.
  5. After the third op-amp, the signal passes through an RC network which AC couples it, filters out high frequency noise which could cause glitches in the microcontroller’s I/O pin, and limits the current into the micro’s clamping diodes.
  6. At this point, we have an amplified digital signal which is still ASK-modulated:

The rest of the work happens in software. The ATtiny85 keeps itself quite busy:

  1. The AVR first applies a narrow digital band-pass filter, to strip out any ringing or other noise that remains after the analog band-pass filter.
  2. Next, a digital low-pass filter. This passes the 1200 baud serial data, but rejects higher frequency glitches.
  3. A threshold is applied, with hysteresis, to convert this data into a stream of ones and zeros.
  4. Next is a relatively typical software serial decoder, with majority-detect. The ASK modulated data has 8 data bits, 1 start bit, and 2 stop bits. Polarity is slightly different than typical RS-232. “1” is the presence of an ASK pulse, “0” is the absence of a pulse. Start bits are “1”, stop bits are both “0”.
  5. Using this serial engine, we receive an 11-byte packet.
  6. The packet is converted from TED’s raw format into a more human-readable (but still machine-friendly) serial format.
  7. The reformatted data is finally output via a software serial port at 9600 baud.

The end result, as viewed from a PC connected to the serial output pin:

HC=245 KW=001.324 V=121.138 CNT=023
HC=245 KW=001.335 V=121.135 CNT=024
HC=245 KW=001.348 V=121.072 CNT=025
HC=245 KW=001.345 V=121.021 CNT=026
HC=245 KW=001.345 V=121.044 CNT=027
HC=245 KW=001.324 V=121.152 CNT=028
HC=245 KW=001.314 V=121.280 CNT=029
HC=245 KW=001.314 V=121.306 CNT=030
HC=245 KW=001.311 V=121.297 CNT=031
HC=245 KW=001.349 V=121.232 CNT=032
HC=245 KW=001.347 V=121.228 CNT=033

The “KW” and “V” columns are self-explanatory. “HC” is my TED’s house code, and “CNT” is a packet counter. Normally in increments by one. If it skips any numbers, we missed a packet.

But wait, what’s wrong with this picture? The power and voltage readings have too much precision. The standard TED display unit will give you a resolution of 10 watts for power, and 0.1 volt. As my data above shows, the TED measurement unit is actually collecting data with far more precision. I can only guess why TED doesn’t give users more precision normally. I suspect they removed it because the extra precision may imply extra accuracy that may not exist.

So, what is TED actually sending? Once a second, it broadcasts an 11-byte packet over your power lines at 1200 baud:

  • Byte 0: Header (always 0x55)
  • Byte 1: House code
  • Byte 2: Packet counter
  • Byte 3: Raw power, bits 7-0
  • Byte 4: Raw power, bits 15-8
  • Byte 5: Raw power, bits 23-16
  • Byte 6: Raw voltage, bits 7-0
  • Byte 7: Raw voltage, bits 15-8
  • Byte 8: Raw voltage, bits 23-16
  • Byte 9: Unknown (Flags?)
  • Byte 10: Checksum

Pretty straightforward. I don’t know the actual A/D converter precision in the measurement/transmit unit, but both the power and voltage readings are sent on the wire as 24-bit raw numbers. I’m still not sure what byte 9 is. In my measurements, it hovered around 250, sometimes jumping up or down by one. It may be some kind of flag byte, or maybe it measures power line frequency? The checksum is dead simple: Add every byte in the packet (modulo 256), and the checksum byte ensures the result is zero.

To figure out what to do with these raw measurements, I collected data simultaneously with my circuit and with the TED RDU’s USB interface. Both sets of results went into a spreadsheet. After removing outliers, I did a linear regression. The resulting linear function is what you’ll find in the current firmware for my homebrew receiver. The following plots from the spreadsheet are a pretty striking illustration of the additional precision available via my raw interface:

The top graph shows power line voltage as recorded by the TED RDU. The second graph shows the raw values I’m receiving from the TED MTU. The bottom graph shows the correlation between the two, in blue, and my linear regression, in red.

I plan to keep improving the circuit. Hopefully there’s a way to get both data and power from a single supply, without dealing with any annoying high-voltage circuitry. If there is any interest, I might make a kit available. If you’re interested, post a comment and let me know what features you’d like. USB port? Serial port? Display?

Using an AVR as an RFID tag

Experiments in RFID, continued…

Last time, I posted an ultra-simple “from scratch” RFID reader, which uses no application-specific components: just a Propeller microcontroller and a few passive components. This time, I tried the opposite: building an RFID tag using no application-specific parts.

Well, my solution is full of dirty tricks, but the results aren’t half bad. I used an Atmel AVR microcontroller (the ATtiny85) and a coil. That’s it. You can optionally add a couple of capacitors to improve performance with some types of coils, but with this method it’s possible to build a working RFID tag just by soldering a small inductor to an AVR chip:

image

image

The above prototype emulates an EM4102-style tag- a very popular style of low-frequency RFID tag which stores a 40-bit unique ID. I can read my bogus ID value (0x12345678AB) using Parallax’s RFID reader. Below is another prototype, with a larger coil and a couple of capacitors for added range and stability. It is programmed to emulate a HID prox card, a simple FSK-modulated tag with a 44-bit payload. I can read this card successfully with my garage door opener. This one is a little large to conveniently carry around, but a smaller AVR package should help.

image

AVRFID Tag Prototype

So, the shiny electrical tape is beautiful, but how does this thing even work? The power pins on the microcontroller aren’t even connected!

As I said, this makes use of several dirty tricks:

  • The coil actually powers the AVR through two of its I/O pins. Nearly every chip out there has clamping diodes on its I/O pins, which prevent voltages on that pin from rising above the chip’s supply voltage or sinking below ground. These diodes are useful for arresting static discharge.When you first hold the RFID tag up to a reader, the chip has no power- the supply voltage is zero. When the coil starts to pick up power from the RFID reader, these two I/O pins are presented with a sine wave, a few volts in amplitude. Anywhere that sine wave exceeds the supply voltage, some energy is diverted from the coil to the chip’s supply rails, via the clamping diode. The end result is that the chip is powered, and the coil’s sine wave is truncated. The top and bottom of the sine have been chopped off, and it looks a lot more like a square wave now.
  • Power filtering using the AVR’s die capacitance. In the smaller prototype, there is no power filtering capacitor at all. In fact, the power is filtered by the internal capacitance of the power planes in the AVR’s silicon die. This isn’t much, but it makes the power supply stable enough that we can execute code even though the supply is pulsing at 125 kHz.
  • Very low voltage operation. This particular ATtiny85 chip is specified for operation at voltages as low as 2.5v. The extended voltage range version (I didn’t have any of these handy) is specified down to 1.8v. But I’m running these AVRs at barely over 1 volt. At these voltages, the normal AVR clock oscillators don’t work- but I can get away with this because of the next hack…
  • The coil is the AVR’s clock source. The inductor isn’t just hooked up to any I/O pin: it’s actually connected to the AVR’s clock input. Remember the square-ish wave we’re left with after the clamping diodes suck away some power? That waveform is now our clock input. The microcontroller is executing code at 125 kHz, in lockstep with the RFID reader’s carrier wave.
  • Firmware? What firmware? At such low speeds, the chip’s firmware looks less like a program, and more like a sequence of I/O operations to perform in sync with each carrier clock cycle. There aren’t a lot of cycles to spare. In the EM4102 protocol, you could potentially do some useful work with the 32 clock cycles you have between each bit. With the HID protocol, though, you need to output an FSK edge as often as once every 4 clock cycles. As a result, the firmware on the RFID tag is extremely dumb. The “source code” is really just a set of fancy assembler macros which convert an RFID tag code into a long sequence of I/O instructions.

The fact that this thing works at all is quite a testament to the robust design of the AVR. The latest AVRFID source is in Subversion, as usual.

Playstation 2 Dual Shock protocol, revisited

Last summer I did some hard-core reverse engineering of the Playstation 2 Dual Shock controller protocol, so I could build a fully-featured Playstation controller emulator/extender device. I wrote up a fairly minimal protocol document, and published source code for an emulator that could impersonate a Dual Shock 2 controller well enough to fool every game in Paul’s comprehensive library 😉

Anyway, this device has been working really well for Paul and I, and we’ve made good use of it. I wasn’t sure if anyone else had noticed my docs and code, though, or found them useful.

Well, recently I heard from Julian Bleecker, who is using my emulator for a pretty nifty project. He also pointed out a very well-organized guide to the Playstation controller protocol, which prominently sites my reverse-engineering work. Cool.

If you’re interested in interfacing anything with a Playstation or a Playstation controller, I highly recommend this guide. It includes information about all of the advanced configuration commands I reverse-engineered, plus it includes all of the basic electrical information that my little document totally glossed over. It sites multiple sources, and they have independently verified everything and provided their own protocol traces.

Playstation controller extender

The little hardware project I started almost 2 months ago is finally done. Completely finished. Bug free! Well, almost. It is, however, in a fully assembled state with firmware that is actually pretty usable.

P8030108

The Unicone2 is the result of my mini-quest to extend Playstation 2 controllers over long lengths of cat5 cable. A while back I did something similar, extending Game Cube and N64 controllers over ethernet. That was the original Universal Controller Emulator, or Unicone. It used an FPGA, and a pair of Linux boxes. This time the design focus is low-latency, no PC required, and full support for Playstation 2 Dual Shock controllers and the Guitar Hero controller.

The Unicone2 went through a few redesigns before I settled on something I liked. The final version is based on a pair of microcontrollers and an asymmetric RS-422 link over cat5 cable. The remote end, where the physical controllers live, runs a PIC16F877A microcontroller. The base station, where all the consoles sit, uses the awesome Parallax Propeller, an 8-core microcontroller.

The remote unit’s job is pretty simple. It initializes and polls its two controllers, attempting to maintain a constant polling rate. The controller data is streamed over a very low-latency 250kbps RS-422 link. In its spare time, the PIC reads lower-bandwidth data (LED status, force feedback state) over an incoming 19200-baud RS-422 link.

The base station has a much harder job. It needs to emulate four controllers and receive data from two remote units simultaneously. These are all high-speed (250 to 500 kbps) asynchronous serial streams, with no flow control. It’s pretty much an intractable workload for most microcontrollers. I spent a long time trying to solve this problem with an FPGA, just like I did with the original Unicone project. This was working okay, but it was tedious. Playstation 2 controllers are even more complex than N64 controllers, and the problem doesn’t map well to the hardware domain.

When I first read about the Propeller, it seemed like a perfect solution to this problem. Just write one controller emulator, and run four copies of it simultaneously on different processors. The controller emulators and the RS-422 receivers communicate over shared memory, making the whole device quite low-latency. This also leaves plenty of CPU power and memory to emulate other flavours of video game controller, as well as leaving space to implement special effects like controller mixing and synchronized record/replay of controller events.

P8030070

I’ll end the hardware rambling for today. Paul or I might have to post soon with some more info about the gameplay implications of the Unicone2. Besides acting as an extender, it can already act as a controller crossbar switch and mixer, which will let us experiment with improvised multiplayer action on single-player games.

Firmware and schematics are in Subversion.

Hardware hackery and the Dual Shock protocol

The hardware-tinkering mood I’ve been in lately shows no sign of subsiding. The theme this time: Everything over cat5.

Backing up a bit… It’s summer! Hooray. So, how does one make the most of the season without actually doing something crazy like going outdoors? Of course the answer is to play video games in a different room which has more fresh air and a better view of the day-star.

Some people would pick up the Playstation and move it to the other room, but my solution is to run the video over inexpensive cat5 cable. This isn’t a new idea, or even a complicated one, but component-video-over-cat5 baluns still sell for around $80 each! I recently made my own pair out of a couple more altoids tins and some parts from the junk drawer.

These work really well. There is a little bit of ghosting visible on a black screen, but during normal use the quality looks perfect, or at least no worse than the quality of the TV it’s connected to. Unfortunately, and I started testing it with a little Megaman 2 and quickly noticed an unrelated problem: the distance is a little too much for our wireless controllers. Time to build a Playstation-controller-over-cat5 extender.

I can think of three different approaches for building such an extender:

  1. Purely electrical: Each PSX signal gets an RS-422 driver and a separate twisted pair. This is very simple, but it would require two cat5 cables per controller! I want at least two controllers running over a single cable.
  2. Low-level multiplexing: Compress multiple signals onto each twisted pair by multiplexing below the protocol level. This would be bit-for-bit identical to plugging a controller in locally, but it has a few big disadvantages. It requires quite a fast clock on the twisted pair, since it requires several bits of information to be sent for every bit in the original protocol. It also doesn’t add any flexibility. You couldn’t, for example, add a crossbar switch for controllers which could let you switch one controller between multiple video game consoles via a button combination.
  3. High-level multiplexing: The console-end of the link includes an FPGA which fully emulates one or more Dual Shock controllers. The controller-end consists of a microcontroller that polls a bank of controller ports and sends updates over an RS-485 link to the FPGA. This is very powerful, and it’s the method used by my old Unicone (Universal Controller Emulator) project. Unfortunately, it’s also relatively complex and it introduces a few milliseconds of latency.

I’m currently working on (3), but I haven’t ruled out (2) completely. First step, I need to know how to emulate a Dual Shock controller. The internet is full of pages that describe the very basics of the Playstation controller protocol, but I couldn’t find a single document that described the protocol in anywhere near enough detail to emulate a controller properly. Time for some reverse engineering…

This was quite a lot like the job I did a while ago to reverse engineer the Nintendo 64 controller protocol enough to build a microcontroller that could emulate a controller. Instead of an HP logic analyzer borrowed from work, this time I have a Saxo board from fpga4fun acting as a real-time Playstation packet sniffer. I should be able to use the same packet sniffer setup to finish reverse-engineering the Playstation 2 memory card protocol so that I can add PS2 support to my psxcard USB adaptor. 🙂

A Verilog core to emulate the Dual Shock controller will hopefully be coming up soon. In the mean time, I published my Notes on the Playstation 2 Dual Shock controller protocol. It’s actually surprisingly more complex than I was expecting. There are several backward-compatibility features, two rumble motors (one with 8-bit speed control), analog buttons… The protocol has several features that the Dual Shock controller doesn’t take advantage of, like support for 254 different force-feedback actuators that you can arbitrarily map into your polling packets. Crazy.