Interfacing with The Energy Detective
I recently bought The Energy Detective (TED), a pretty inexpensive and friendly way to keep tabs on your whole house’s electricity usage. It’s a lot like having a more featureful version of your utility company’s power meter, sitting on your kitchen counter. It can estimate your utility bill, and tell you how much electricity and money you’re using in real-time. The resolution is pretty good- 10 watts, 1 second.
As a product, I’ve been pretty happy with it. It does what it claims to, and the measurements seem to be fast and accurate. Of course, being the crazy hacker I am, I wanted to interface TED with other things. I don’t have a home automation system as such, but I did want to get real-time graphs of my electricity usage over various time periods. I also wanted the possibility to use TED as an information feed for various other display devices around the house. (But that’s a topic for another blog post…)
The stock TED package consists of two pieces: A measurement/transmit unit (MTU) and receive/display unit (RDU). The MTU has a pair of current transformers, and it installs entirely in your house’s breaker panel. It takes the power readings every second, and transmits them over the power lines to the RDU, which is just a little LCD panel that plugs into the wall. The RDU has a USB port, which you can use with TED’s “Footprints” software.
So, hoping that the USB port would do what I want, I bought the Footprints software for $45. The TED system itself is, in my opinion, a really good value. The Footprints software is not. As a consumer, I was disappointed by two main things:First of all- the UI lacks any polish whatsoever. It looks like a bad web page from the 90′s. Second of all, the data collection is not done in hardware, it’s implemented by a Windows service. This means you can’t collect data to graph unless you have a Windows PC running. Not exactly a power-efficient way to get detailed power usage graphs.
As a hobbyist, a few more things frustrated me about Footprints. The implementation was pretty amateurish. Their Windows service runs a minimal HTTP server which serves up data from an sqlite database in XML. The front end is actually just a Flash applet masquerading as a full application. Energy Inc, the company behind TED, has an API for Footprints: but you have to sign a legal agreement to get access to it, and I wasn’t able to get any details on what the API does and doesn’t include without signing the agreement. So, I opted not to. It would be much more fun to do a little reverse engineering…
So, I did. The end result is that I now have two ways of getting data out of my TED system.
Using the USB port
The TED RDU’s USB port is actually just a common FTDI usb-to-serial adapter. The RDU sends a binary packet every second, which includes all of the data accessible from the Footprints UI. This includes current power usage, current AC line voltage, utility rates, month-to-date totals, and anything else you’ve programmed into your RDU.
There has been some prior work on reverse engineering this protocol. The Misterhouse open source home automation project has a Perl module which can decode the TED messages.
Unfortunately, the Perl module in Misterhouse won’t work with more recent versions of the RDU like mine. The recent RDUs have a different packet length, and they require a polling command to be sent before they’ll reply with any data.
I found the correct polling command by snooping on Footprints’ serial traffic with Portmon. I also noticed a packet framing/escaping scheme, which explains some of the length variations that would have broken the module in Misterhouse.
The result is a Python module for receiving data from a TED RDU. It isn’t terribly featureful, but it should be pretty robust.
- Get the latest TED Python Module from my Subversion repository.
Direct from the wall socket
Now for the more exciting method: What about reading data directly from the power line, without using the TED receive/display unit at all? This could provide some exciting opportunities to embed a small and cheap TED receiver inside of other devices, and it would provide some insight on what exactly is being transmitted by the box in my breaker panel.
The TED RDU is pretty simple internally: A Dallas real-time clock, PIC18 microcontroller, chip-on-glass LCD, some buttons, and the TDA5051A, a single-chip home automation modem from Philips. This chip can receive and transmit ASK modulated signals at 1200 baud, with a carrier frequency of around 132 kHz.
Digi-key carries the TDA5051A, but I figured it would be more educational (and more hobbyist-friendly) to try and build a simpler receiver from scratch using only commonly available parts. The result is the following design, with an 8-pin AVR microcontroller and three op-amps:
Update: There is now a Revision 2 of the schematic, which uses only a single power supply and one op-amp.
- The power line is sampled via a 9V AC wall wart. With this design, it needs to be a separate isolated power supply. So far, I haven’t had any luck with using this same transformer to power the circuit. Any rectifier introduces high-frequency harmonics which drastically degrade the signal-to-noise ratio.
- The first stage is a 10x amplifier and 138kHz band-pass filter. This is from Texas Instrument’s “Filter Design in Thirty Seconds” cheat-sheet.
- The next stage is an amplifier and high-pass filter, to remove the last remnants of 60 Hz ripple.
- The third stage is just for amplification. In a design which used higher quality op-amps, this stage may not be necessary.
- After the third op-amp, the signal passes through an RC network which AC couples it, filters out high frequency noise which could cause glitches in the microcontroller’s I/O pin, and limits the current into the micro’s clamping diodes.
- At this point, we have an amplified digital signal which is still ASK-modulated:
The rest of the work happens in software. The ATtiny85 keeps itself quite busy:
- The AVR first applies a narrow digital band-pass filter, to strip out any ringing or other noise that remains after the analog band-pass filter.
- Next, a digital low-pass filter. This passes the 1200 baud serial data, but rejects higher frequency glitches.
- A threshold is applied, with hysteresis, to convert this data into a stream of ones and zeros.
- Next is a relatively typical software serial decoder, with majority-detect. The ASK modulated data has 8 data bits, 1 start bit, and 2 stop bits. Polarity is slightly different than typical RS-232. “1″ is the presence of an ASK pulse, “0″ is the absence of a pulse. Start bits are “1″, stop bits are both “0″.
- Using this serial engine, we receive an 11-byte packet.
- The packet is converted from TED’s raw format into a more human-readable (but still machine-friendly) serial format.
- The reformatted data is finally output via a software serial port at 9600 baud.
The end result, as viewed from a PC connected to the serial output pin:
HC=245 KW=001.324 V=121.138 CNT=023 HC=245 KW=001.335 V=121.135 CNT=024 HC=245 KW=001.348 V=121.072 CNT=025 HC=245 KW=001.345 V=121.021 CNT=026 HC=245 KW=001.345 V=121.044 CNT=027 HC=245 KW=001.324 V=121.152 CNT=028 HC=245 KW=001.314 V=121.280 CNT=029 HC=245 KW=001.314 V=121.306 CNT=030 HC=245 KW=001.311 V=121.297 CNT=031 HC=245 KW=001.349 V=121.232 CNT=032 HC=245 KW=001.347 V=121.228 CNT=033
The “KW” and “V” columns are self-explanatory. “HC” is my TED’s house code, and “CNT” is a packet counter. Normally in increments by one. If it skips any numbers, we missed a packet.
But wait, what’s wrong with this picture? The power and voltage readings have too much precision. The standard TED display unit will give you a resolution of 10 watts for power, and 0.1 volt. As my data above shows, the TED measurement unit is actually collecting data with far more precision. I can only guess why TED doesn’t give users more precision normally. I suspect they removed it because the extra precision may imply extra accuracy that may not exist.
So, what is TED actually sending? Once a second, it broadcasts an 11-byte packet over your power lines at 1200 baud:
- Byte 0: Header (always 0×55)
- Byte 1: House code
- Byte 2: Packet counter
- Byte 3: Raw power, bits 7-0
- Byte 4: Raw power, bits 15-8
- Byte 5: Raw power, bits 23-16
- Byte 6: Raw voltage, bits 7-0
- Byte 7: Raw voltage, bits 15-8
- Byte 8: Raw voltage, bits 23-16
- Byte 9: Unknown (Flags?)
- Byte 10: Checksum
Pretty straightforward. I don’t know the actual A/D converter precision in the measurement/transmit unit, but both the power and voltage readings are sent on the wire as 24-bit raw numbers. I’m still not sure what byte 9 is. In my measurements, it hovered around 250, sometimes jumping up or down by one. It may be some kind of flag byte, or maybe it measures power line frequency? The checksum is dead simple: Add every byte in the packet (modulo 256), and the checksum byte ensures the result is zero.
To figure out what to do with these raw measurements, I collected data simultaneously with my circuit and with the TED RDU’s USB interface. Both sets of results went into a spreadsheet. After removing outliers, I did a linear regression. The resulting linear function is what you’ll find in the current firmware for my homebrew receiver. The following plots from the spreadsheet are a pretty striking illustration of the additional precision available via my raw interface:
The top graph shows power line voltage as recorded by the TED RDU. The second graph shows the raw values I’m receiving from the TED MTU. The bottom graph shows the correlation between the two, in blue, and my linear regression, in red.
I plan to keep improving the circuit. Hopefully there’s a way to get both data and power from a single supply, without dealing with any annoying high-voltage circuitry. If there is any interest, I might make a kit available. If you’re interested, post a comment and let me know what features you’d like. USB port? Serial port? Display?
- You can get the latest schematics and firmware from my Subversion repository.
Very cool, Micah. I’d love to set something like this up at home. Maybe people would understand why I’m such a pest about turning things off when they’re not in the room.
script error
I would love to run your script but I don’t know python. I tired to follow direction online to run your script but this is the error output that I get:
Traceback (most recent call last):
File “/Users/costa/Desktop/ted.py”, line 42, in
import serial
ImportError: No module named serial
Do you have any advice for me?
Re: script error
Hi,
You need to install the pySerial module from:
http://pyserial.wiki.sourceforge.net/pySerial
After that, you should be able to run the script. It takes one command line argument: the path to the serial port that your TED unit is on.
Good luck!
data logger
Could you add some storage, like an SD card? I would love to be able to log a months worth of data and then plug the card into a pc and play with the data.
Maybe add a clock to the design so that timestamps can be added to the logged data. Not sure about an easy way to set the clock.
Onscreen TED RDU
A suggestion: a alternative TED RDU that outputs standard video
Most televisions have extra video-in connections and you could turn your television into a second TED display. Maybe leverage something like http://www.decadenet.com/bob4/bob4.html
If the unit could accept IR commands it could be interactive like the desktop RDU as well.
Re: Onscreen TED RDU
That’s quite a good idea. I’d probably use a Parallax Propeller chip (a multi-core microcontroller which is especially good at generating video), and an SD card for long-term data storage.
I’ll have to try this out on some rainy day
Thanks!
Re: Onscreen TED RDU
I contribute some funding through paypal if you will send me one shortly after that rainy day.
Also came across party interested in T.E.D. integration at http://chuck-wright.com/ His device has a couple serial inputs as well as some counter inputs. His main user interface is an embedded web server on an ethernet port.
monitoring sub-panels
If you have multiple MTUs from TED, each has a different house code. This device would allow you to seperatly measure sub-panels and give you the choice of aggregating them or netting them if you have on-site generation such as a solar array, wind turbine, or genset.
another feature suggestion
Interface back onto the power line and send X-10 commands upon several conditions like:
excessive kw
low voltage
this would make it easier to cycle off loads during peaks and cycle off motors and other electronics that don’t like low voltage situations.
maybe a “plix” chip would do the job.
http://www.nomad.ee/micros/x10faq.html
Help
I’m new to MAC and have loaded python. I can run this script in python, but I get errors on line 188 and 175. I have installed the serial module, but do not know how to run the script and point to the serial port that TED is connected to??
I do not want to go back to windows, but this scripting is killing me.
DR
Re: Help
What are the errors you’re getting, and what is the exact command you’re using to run the script?
It expects a command line argument with the name of the serial port. I think on Mac OS you’ll get a file like /dev/tty.usbserial-FOO, where FOO is some kind of device identifier. Locate this file, and give its path to the script on the command line.
–Micah
Re: Help
I have all .py scripts associated with python. I click on the script (I named it TED.py)…. Python IDLE opens, the script is in the window and I click on “RUN Module”… I get the errors in the Python Shell window…
Running it from terminal, I get the same errors, plus a few more serial communication errors.
Traceback (most recent call last):
File “/Users/user/Desktop/TED.py”, line 188, in
main()
File “/Users/user/Desktop/TED.py”, line 175, in main
t = TED(sys.argv[1])
IndexError: list index out of range
Re: Help
I got it working! Thanks!
So it is running in the terminal window and reading the output of my TED (from the USB/serial port driver) every second.
The result is a display of the raw packet and the 4 decimal values below it… This scrolls up the window as long as I will let it run.
Now, to figure out how to decode all values and write to a file or something so that I can get a graphical dashboard on my mac. Forget waiting for them to write a MAC version of footprints.
Thanks for the research and help!
DR
Re: Help
So what was the fix? Inquiring minds want to know..
I would be interested in a kit
If you are still thinking about making one available.
I did not understand that you had to buy multiple copies of Footprints if you had multiple RDUs
and now I am looking at spending a bunch of money just to enable the USB ports. Since I wrote
my own code to monitor the system I will not be using even one copy of Footprints.
Nice job on the reverse engineering!
No luck on mac…
When I plug my TED 1001 into my mac, two devices appear:
/dev/tty.usbserial-A900a2fu
/dev/cu.usbserial-A900a2fu
But neither one seems to produce any output with ted.py.
I tried running python from the command line:
>> p=serial.Serial(‘/dev/tty.usbserial-A900a2fu’,19200,timeout=0)
>> p.write(‘\xAA’)
>> p.read(4096)
”
Nothing. The ‘\xAA’ command just doesn’t seem to work. Is there some special trick to this? What version of the pyserial module do I need?
>>> serial.VERSION
’2.4′
Has anyone gotten this to work on max osx 10.5?
Re: No luck on mac…
Reading around on other forums…
Call (800) 959-5833 and pay $45 to activate the TED data port. You will need to run a windows program to activate the port or have your TED activated before they ship it to you;
- Is this really necessary? Does the TED device come with the data port disabled? Do I really need to pay $45 and run their software on a windows machine before the TED will talk over the serial port?
Kit or Completed Device
I would be really interested in a kit of the V2 design, either that or a completed device. Even just a printed circuit board could be enough.
Re: Kit or Completed Device
OK, a minimum would be a programmed ATtiny85, most people probably don’t have the stuff to program one… A circuit board would be nice but not necessary.
Re: Kit or Completed Device
Hey yall.
Since this seems to be a small hub of activity when it comes to TED and python…
I’ve heavily modified the ted.py script and added the ability for it to store information locally in a sqlite database as well as remotely in a mySQL database.
You can see the result work-in-progress here:
http://www.murkyview.com
you can download my version of ted.py from http://www.murkyview.com/ted.py
It’s archiving every second.
I’m a decent web scripter, but the whole serial communication thing i don’t all understand so my question is about how the protocol table works in ted.py
I’ve used the TED API serial spec to fill in as much data as I can, but some of it doesnt seem to be decode correctly, especially the totals.
I’ve pasted the table in here. I think the “fmt” for the total values is wrong but I can’t find a reference on what to change it to.
# Offset, name, fmt, scale
(82, ‘rateArray’, “
(108, ‘MTUHourCode’,”
(132, ‘lowVoltsDay’, “
(136, ‘highVoltsDay’, “
(140, ‘lowVoltsMonth’, “
(143, ‘highVoltsMonth’, “
(146, ‘peakKWHDay’, “
(150, ‘peakKWHMonthToDate’, “
(158, ‘totalKWHDay’, “
(148, ‘peakDollarDay’, “
(152, ‘peakDollarMonthToDate’, “
(154, ‘totalDollarDay’, “
(166, ‘totalKWHMonthtoDate’, “
(170, ‘totalDollarMonthtoDate’, “
Any help greatly appreciated.
Chris
chrisale@gmail.com
py newbie
I have been enjoying reading everyone’s posts and thought I would try out some python. If anyone has the patience to help out a newbie that would be great. I have Ted with the viewer and power monitor installed, but wanted to try out some other data logging techniques. Thanks
Here is my error message.
Traceback (most recent call last):
File “C:\Documents and Settings\deb\Desktop\ted.py”, line 42, in
import serial
File “C:\Python26\Lib\site-packages\serial\__init__.py”, line 18, in
from serialwin32 import *
File “C:\Python26\Lib\site-packages\serial\serialwin32.py”, line 9, in
import win32file # The base COM port and file IO functions.
ImportError: No module named win32file
>>>
FTDI USB Driver
The device would not mount for me until I installed the FTDI USB Driver.
Now I see tty.usbserial-A900a211 .
I am using OS X 10.5.6
Re: FTDI USB Driver
I needed this driver for both my MBP and Intel Mac Mini.
The script is running nicely with python 3 for me.
Thanks
Re: No luck on mac…
I haven’t bought a TED unit yet. If I ask when I purchase it, will they activate the data port without also selling me the Footprints software?
Alternatively, has anyone found a way to activate the TED unit without buying the Footprints software?
Thanks for the awesome post!
ted transmitter ?
Excellant work on this….
I’m looking forward to making one of these next weekend.
Any chance of getting a version of the transmitter (MTU) designed ?
Re: ted transmitter ?
Cool. Good luck building your receiver, and let me know how it turns out!
I don’t have any interest in building my own transmitter, since Energy Inc.’s transmitter is affordable and well-designed, and I already have one
If you wanted to build a transmitter, I know I’ve seen a couple of homebrew energy meters online that might provide some inspiration. They’re basically just a current transformer (either storebought or homebrew), some amplifiers and A/D converters, and a microcontroller that can sample the current and voltage waveforms pretty fast, and multiply/integrate them to calculate power.
If you want to send the data in a TED-compatible format, you’d probably want to just use the same powerline modem chip that the TED modules use. It’s the TDA5051A, if I remember correctly:
http://www.nxp.com/acrobat_download/datasheets/TDA5051A_2.pdf
You’d also need to figure out that unknown byte 9 in the packet format
Good luck!
–Micah
I’ve only tested it with the new ones, so I can’t say for sure.. I don’t know if they changed the packet format at all.
But I suspect it would work. I’d be surprised if the older TEDs did anything with the polling byte other than ignore it.
–Micah
Great Work…!
Great python script. I’ve had been trying to get the serial data out of the TED to log to my Linux box for quite some time, and this script made it all easy. It is much appreciated.
I did however flesh out the struct table of TED values decoded from the packet using the SDK so I can get all the TED generated values. I also added the ability for it to log the data to MySql so I could get historical data. The link to that file for those interested is here:
http://www.bananabend.net/energy_detective/ted5.py
This project though makes it much more interesting, bypassing the MTU altogether. I think it would make an interesting Arduino project! With all the shields available for it, the opportunities are endless!
And thanks again for the great work with the python script!!
Thanks for your work on this Micah. I have Ted 1001 with firmware version 8 and I had to change the packet length from 278 to 276 to get the script to work. I plan to start on a web interface for this now. It’d be nice if google opened up their powermeter software to 3rd parties so we can import our data into it. Looks like they already support Ted 5000 but us Ted 1000 users are on our own.
Hi Micah, Thanks for the great work!
Bandon, i would be interested in following your work on the web interface for TED1001.
I am looking forward to finally getting the Mysql server up and running on my diskless Damn Small Linux embedded PC
Hiya Micah,
I was wondering if you could help me on some Python scripts?
Basically I need a python sript to read an output from force sensors which I have made…I have got a breadbaord with PIC16F877a, and i want the python to put the readings from that variable resistors…
TED is a tad bit expensive in comparison to alot of the newer energy monitors out there right now. As a developer my personal favorite is the Envi. Not necessarily because it is cheaper than the TED but because it uses XML instead of python. The usability of XML is fantastic especially when it comes to blending it into web application.
Looks like it has been awhile since anyone has commented, but hope someone is listening. I am experiencing the same problem Anonymous (above) had
t = TED(sys.argv[1])
IndexError: list index out of range
Darn, sure would have been nice if they would have shared the cause.
If anyone can help me, it would be greatly appreciated.