Searching for a Calorie Graph

Kenneth did some academic paper research and found that our EE column = Energy Expenditure which he thought was the crude calorie calculation on the device.

I've plotted EE for an ~12 hr period in the top half of the graph above, and in the lower half is the calorie graph from the GoWearFit web service. I'd say Kenneth is right!

The web service obviously does slightly different processing. I'm also not sure of the units for EE - the values range from ~113 - 1050, whereas the calorie chart range from ~1.5 - 7.5 calories-per-minute. Maybe it is just a divide-by-100 and the more sophisticated web service processing is simply that much more accurate, but I'm not sure. A summation over EE/100 gives a much larger calorie count than the web service.

Memory Clear Ability

I was surprised to hear that if you have a wristband display for your GoWearFit armband, and don't subscribe to the web service, that you can no longer use the armband once its memory is full.

If true, this seems rather unfair. Also poor business practice - the person in question wanted to buy a second device+wristband to replace a lower quality competitor unit, but was unwilling to pay for two unused web subscriptions. I seem to be driving multiple sales here...

I tracked down the command to clear the device memory of sensor data the same way the BodyMedia app does it.

A new bmhack update (bmhack-z717a) is available with a command line '--clear' command.
See the updated "release post" for the download link.

This feature, like everything else, is highly experimental. I have not yet received a single end user report - please provide them if you've used bmhack.

UPDATE 7/17: New version z717a. TESTED UNDER WINDOWS. More robust serial I/O.

UPDATE 7/18: New version z718a. New data record type added, thanks to Freak. Enhanced device memory dump capabilities for improved debugging.

UPDATE 7/18: New version z718b. Fixed limited memory bug (thanks again Roy!). Software can theoretically now handle a completely full device.

A Trivial Activity Classifier (that works!)

With low expectations, I thought I'd throw a simple classifier at the data records to see if it could differentiate between sleep, mundane activity, and strenuous activity.

The results surprised me!

Top: GoWearFit activity graph, Middle: k-means classifier, Bottom, GoWearFit Sleep Graph. Click graphic for full resolution.

I used a simple k-means clustering function out of the Open Source "R" statistical package (via the Python RPy interface). I did _NO_ normalization on the data at all. I just read in the tab-delimited data, stripped off the date fields, and fed it to the classifier.

Each 1-minute data record became a point in 27-dimensional space, and I told the algorithm to divide them into 3 classes (hoping they would end up being "sleep", "vigorous", and "moderate"). And it simply worked!

The black-grey-and-white graph above is the output of the classifier for each minute in a ~18 hour period. The blue graphs are the graphs provided by the GoWearFit web report, aligned manually.
The peaks at 10:45am and Noon were bike rides/walks. I took an afternoon nap, and went to sleep around 11:30pm.

Getting the Sensor Data Right

Over the past couple days I've made considerable progress on the GoWearFit command line extraction tool. Be sure to get the latest version in my edited release post below.

The current version (z712a) 100% parses the device data memory, and there are only a few fields I still don't understand. It also adds extracts date and time for the samples, and exports everything into one nice tab-delimited spreadsheet table.

I added an example tab-delimited output file into the release post as well.

Finally, I discovered that FileDropper hosting is not quite as free and forever as they claim. So a lot of earlier data postings have disappeared. Most are obsolete, but I thought I'd mention it.

This is still very untested. Let me know if works or not!

Command Line Extraction Tool

PRE-ALPHA: bmhack is a command line extraction tool which will extract and parse data from your (new model) BodyMedia ArmBand device into a variety of formats, including a tab-delimited spreadsheet-compatible file, raw packets in ASCII HEX, and a full Python cPickle binary dump.

THIS TOOL WILL NOT HELP YOU if you are looking for something to give you calorie counts or activity graphs. It is in a very early state, and is only useful for technical people who wish to attempt to experiment with and analyze the raw data or learn more about their BodyMedia device.

You must install Python 2.6, and the Python numpy libraries.
Windows: Python 2.6.2 from http://www.python.org/download/releases/
Windows: Numpy for 2.6.2 from http://sourceforge.net/projects/numpy/files/

Download bmhack VERSION z718b [June 18 update] from:
http://www.mediafire.com/?sharekey=26cf7e726ccc34f061d4646c62b381cbfe370e2ca4df676ac95965eaa7bc68bc

Windows: Launch bmhack with C:\python26\python.exe bmhack.py

There is also an example of several days of CSV output in that MediaFire directory, filename z705a.csv

It is verified to work under Linux, and should work under Windows (let me know?).

If you want to experiment with the less understood parts of the protocol, download the known_requests.pickle from the same site and move into your current directory when you run bmhack so you can use the --fromSerialFull command.

MISSING
-[DONE VERSION z713a] Timestamps - Pretty sure I know where time offsets live before each data table. I output them to CSV, but can't intepret them yet. I expect a global time field somewhere?

-"Register" data - user attributes like height, serial number, etc are probably stored outside of the main data memory area. I haven't really looked.

-Other

In addition to being a command line program, you can also import bmhack into a Python interpreter and manipulate the data or the device directly. There are an array of analysis and display python functions.

The bmhack.py code and comments also contain my (incomplete) documentation of the ArmBand data structures. Look at ReadAllStruct() to start.

Linux: How to force the ftdi_sci driver to recognize the BodyMedia Vendor/Product IDs:

sudo modprobe ftdi_sio vendor=0x11f8 product=0x0007

Verify that a new '/dev/ttyUSB*' device has been created.

./bmhack.py -h
THIS CODE IS DECLARED BY THE AUTHOR TO BE IN THE PUBLIC DOMAIN.
NO WARRANTY EXPRESSED OR IMPLIED OF ANY KIND IS PROVIDED.
USAGE: ./bmhack.py [SOURCE] [TARGET] [TARGET] ...
Retrieve and convert data from a BodyMedia armband device
Convert from a packet source to one or more target formats
A packet source can be a live BodyMedia USB device, a cPickle
dump file of packets, or a capture file form a serial port sniffer.

SOURCES - specify only one
--fromSerial=  Extract data by quering a live USB device on the specified serial port
--fromDump=  Read packets from a cPickle dump file saved previously
--fromFSPM=  Parse packets from a 'Free Serial Port Monitor' by HDD Software.   is an export of the RequestView window.

TARGETS - specify one or more
--toDump=  Write cPickle dump of all packet data
--toCsv=  Write a Spreadsheet-compatible tab delimited file of most of the data
--toPackets=  Write parsed packets in human-readable HEX format

NOTE: Specify '-' as a filename to toCsv and toPackets commands in order to write to stdout instead of a file

HTTP Sniffing

I've tried USB sniffing, serial snooping, but some how neglected network sniffing of the HTTP traffic. Back to my old friend Wireshark nee Ethereal.

VERY promising results which hopefully will help solve the record field alignment puzzle.

This is obviously the technique that Henry at BodyBuggHacks used, at least for his initial postings (his later partial Java code posting indicated direct use of the serial port or a BodyMedia Java API). The protocol has changed a bit - field names are different etc, but the structure is the same.

Here is an edited sampling of the response to an HTTP POST during a sync:

....sr.6com.bodymedia.common.shared.armband.comm.UploadRequestO...r.0....L..seri
alPortBeant.,Lcom/bodymedia/device/serial/SerialPortBean;xr.,com.bodymedia.commo
n.shared.comm.DataRequest...).z.n...L..handlert..Ljava/lang/String;L..mySessionI
Dq.~..xpppsr.*com.bodymedia.device.serial.SerialPortBean.\.qBI.K...Z..cradle_mod
eL..ageq.~..L..batteryq.~..L..birthdateq.~..L..boardq.~..L..boardSeriesq.~..L..c
alibrationq.~..L..channelListq.~..[..channelst..[BL..epochq.~..L..gsrThresholdq.
~..L..handq.~..L..heartTargetq.~..L..heightq.~..L..memoryq.~..L..productCodeq.~.
.L..recordsq.~..L..serialq.~..L..sexq.~..L..smokerq.~..L..subjectq.~..L.
systemTimeq.~..L..uptimeq.~..L..versionq.~..L..volumeq.~..L..weightq.~..xp.t..Ag
e: 99 years ^M
t.;Battery voltage: 4.160  Max voltage: 4.200 Charged to 94%^M
t..Series: 16^M
t./ 0: RAWACCTR INUSE   input: 698 output: 1763^M
1: RAWACCLO INUSE   input: 707 output: 2343^M
2: RAWACCTR INUSE   input: 2094 output: 1212^M
3: RAWACCLO INUSE   input:
2095 output: 2930^M
4: RAWACCFW INUSE   input: 988 output: 2410^M
5: RAWACCFW INUSE   input: 1964 output: 2818^M
6:   RAWGSR INUSE   input: 0 output: 1366^M
7:    RAWON INUSE   input: 0 output: 1367^M
8: RAWTSKIN INUSE   input: 40000 output: 1416^M
9:  RAWTCOV INUSE   input: 40000 output: 1416^M
10: RAWTSKIN INUSE   input: 25002 output: 2048^M
11:  RAWTCOV
INUSE   input: 25002 output: 2048^M
t..Chan#    Name ^M
-----    ---- ^M
  0    RAWACCFW^M
  1    RAWTSKIN^M
  2    RAWGSR^M
  3    RAWACCTR^M
  4    RAWACCLO^M
  5    RAWVBAT^M
  6    RAWTCOV^M
  7    RAWON^M
  8    EE^M
  9    MOVTSKIN^M
 10    MOVGSR^M
 11    MOVACCTR^M
 12    MOVACCLO^M
 13    MOVVBAT^M
 14    MOVTCOV^M
 15    MOVON^M
 16    MADACCTR^M
 17    MADACCLO^M
 18    F0CROSS^M
 19    HRATE^M
 20    PEDO3^M
 21    PLATEAU^M
 22    MADECG^M
 23    TRPEAKS^M
 24    MOVTHETA^M
 25    FWPEAKS^M
 26    FCOUNT^M
 27    MOVACCFW^M
 28    MADACCFW^M
 29    TCOUNT^M
 30    LCOUNT^M
 31    PEDO3TOE^M
 32    TIMESTMP^M
 33    HEARTBT^M
 34    T0CROSS^M
 35    L0CROSS^M
 36    RAWECG^M
 37    LOGSWEEP^M
 38    MADTHETA^M
 39    LOPEAKS^M
 40    COMPGSR^M
 41    RAWCGSR^M

[...SNIP...]

SESSION-BEGIN0502Firefly2_00000000af1bc00e04a54d7ca00000000090c0000046500200780MOV
TSKIN^M
93494295095895d96396696996c96f97898098498598798998798598899199d9a29a099f9a29a29a
19a39a29a39a39a099d99b99c99f9a19a49a69a59a59a79a99ab9ad9b09b09b19b39b69b89b99b89
b99bb9bd9bf9c09bd9bd9c09bf9be9be9be9be9bf9c19c39c59c79c89ca9ca9ca9ca9ca9c89c69c5
9c59c69c79c99c99c99c99c99c69c39c09c19c59c89cc9ce9d09d29d39d29d29d39d49d79db9dd9d
d9df9de9dd9de9dc9d99d89d99da9de9e09db9d29ca9c59be9b99b59b19ae9ab9aa9aa9aa9ae9b29
b39b49b49b39b39b29b29b29b39b39b39b49b49b49b49b59b59b69b69b79b89b99ba9bb9bb9bb9bb
9bc9bc9bd9bd9bd9bf9bf9c09c09c19c29c19c19c29c29c39c39c49c59c59c29be9bc9bb9ba9bb9b
b9bb9ba9ba9ba9ba9bb9bb9bc9bd9bd9be9be9be9c09c29c49c59c69c59c49c59c69c69c69c79c79
c69c69c69c69c69c59c59c59c59c49c49c39c39c39c29c29c29c39c49c59c69c79c89c99ca9ca9ca
9cb9cc9cc9cc9cc9cd9cd9cd9cc9cc9ca9c79c59c49c59c
79c79c89c89c99ca9cb9cb9cb9cb9cb9cb9cb9cb9cb9cb9cb9ca9c99c79c69c59c69c89ca9cc9cd9
cd9cd9cd9cd9cd9cd9cc9cc9cc9cc9cb9cb9cb9cb9cb9cb9ca9ca9ca9ca9ca9ca9ca9ca9ca9cb9cb
9cc9cc9cd9cd9ce9ce9cf9cf9d19d29d49d59d79d89d89d99da9da9db9dc9dd9de9df9df9df9dd9d
c9dc9dd9dd9de9de9df9df9e09e09e09e19e09dd9dc9dd9de9df9e09e09e19e19e19e09e09e19e29
e39e59e69e69e79e89e99e99ea9e99e99e79

[...SNIP....]

t.DThere have been 1247284976 seconds since the epoch.  Time zone -9.^M
t.&gthresh  offbody: 1376  onbody: 1371^M
t..Handed: Left ^M
t..heartrat
e targets lo: 0 hi: 0t..Height: 67 inches^M
t..FILE: 78339 bytes of 1048576, 7 percent used^M
t..Product code: 173^M
t..#  Type      Name     Div               Channels            Bytes^M
-  ----  --------     ---  -------------------------------  -----^M
0   16     V6RES1    1920    9  11  12  27  14  16  17  40    13^M
1   17     V6RES2    1920   20  21  23
24  38  29  37 254    12^M
2   18     V6RES3    1920   30  34  35  31  39  13  28 254    12^M
3   19     V6RES4    1920   26  18  25  10   8 254 254 254     9^M
4   20     UNUSED       0  254 254 254 254 254 254 254 254     0^M

I am very excited by that table of record types (16-19) to channel numbers. I had been parsing that table, but couldn't interpret the numbers before. I believe this gives me the numbered field name to record mapping I've needed.

I am willing to send you a full sample if you request via e-mail centibenzo at gmail.

EDIT (7/11): BREAKTHROUGH:
1. The device stores data as 12-bit packed fields, not byte-aligned.
2. The "table headers" with apparent column labels do NOT correlate with the columns (baffling).
3. The actual record layout is obtained from the record type table (seen above in HTTP, but also in device mem) indexed into ANOTHER set of field names retrieved by individual request packets. Weird.

I should have full data extracts soon.

20+ Hours of Sample Data

Full day activity image (click image for clear view)

Here is an almost full day (~20hrs?) of activity, starting in mid-morning.

It includes (in order) a ~10 min bike ride, a walk and ~5 min bike ride an hour later, an early afternoon nap (clearly visible in full-size image approx 1/3rd from left), and an eventual ~6-8hrs of broken sleep (dark right side).

Here is the tab-delimited parsed data:
http://www.filedropper.com/z709a-full-serial

Here is a gzipped raw binary dump of the 200kbyte data memory (that I know of). You can try to parse all my unknown data structure fields from this if you like Sudoku.
http://www.filedropper.com/z709a-mem-dump

OBSERVATION: This dump had 1121 data records. I wore the armband something like 20 hours. That comes to 56 records per hour, which adds some evidence to my 1 record/minute estimate.

OBSERVATION: this data dump came in one continuous data record table. I believe this is because I put the armband on and kept it on for the entire day. I think a new data record table is started whenever the armband is put on.

OBSERVATION: this one day of data took ~50kbyte of device memory. The device must have more than the 200kbyte memory I know about.