Thursday, July 16, 2009

Memory Clear Ability

I was surprised to hear that if you have a wristband display for your GoWearFit armband, and don't subscribe to the web service, that you can no longer use the armband once its memory is full.

If true, this seems rather unfair. Also poor business practice - the person in question wanted to buy a second device+wristband to replace a lower quality competitor unit, but was unwilling to pay for two unused web subscriptions. I seem to be driving multiple sales here...

I tracked down the command to clear the device memory of sensor data the same way the BodyMedia app does it.

A new bmhack update (bmhack-z717a) is available with a command line '--clear' command.
See the updated "release post" for the download link.

This feature, like everything else, is highly experimental. I have not yet received a single end user report - please provide them if you've used bmhack.

UPDATE 7/17: New version z717a. TESTED UNDER WINDOWS. More robust serial I/O.

UPDATE 7/18: New version z718a. New data record type added, thanks to Freak. Enhanced device memory dump capabilities for improved debugging.

UPDATE 7/18: New version z718b. Fixed limited memory bug (thanks again Roy!). Software can theoretically now handle a completely full device.

46 comments:

  1. I have been trying to get this working under windows. I got all the Python stuff installed that I believe I need. When I run the program it works for a bit then hits an exception because it has a zero length packet in ParsePacket2(). I think this is because the serial is working slightly differently, but I really don't know.
    What do I have to do to get it to read from the serial port and just dump the raw data so I can send that to you?

    ReplyDelete
  2. Yes, sounds like a serial timing issue. I'll try to make the serial reading a bit more robust. Thanks for the report!

    You can probably get a quick fix by increasing the "setTimeout" in the OpenSerial() function.

    ReplyDelete
  3. I just posted a new version, z717a, which I have tested (with minor fixes) under Windows.

    Also features must more robust serial I/O (I expect that was your problem).

    Let me know how it works!

    ReplyDelete
  4. That works much better. I was able to read my armband data using the --toPackets option and got a file with the hex dump of packets.
    However, I tried the --toCsv option and it goes for a bit and then get's this assert:
    File "C:\...\bmhack-z717a.py", line 805, in ReadRecord
    assert RecPack.has_key(id), "Unknown record ID %i at offset %i" % (id, offset)
    AssertionError: Unknown record ID 48 at offset 4352

    ReplyDelete
  5. Here's the data from the dump around that offset, and it looks like it's not reading enough data for the record that starts near here (an 0E one):
    A 02 10d8 2c 93 94 __ 5E 03 8C 58 11 __ __ __ 13 17 DC 3B A1 38 48 CF 12 07 40 C3 04 10 __ 0E BA FB 06 7F 13 15 E1 11 15 AF F8 4E 8F 30 11 40 07
    A 02 1104 2c __ 10 98 87 90 82 79 91 95 A0 C7 0B D8 28 11 __ 60 __ 12 98 3F 3F F2 85 78 6F 12 23 E1 01 12 70 __ 13 BA F3 0A 6F 13 1E B0 FD 14 9D

    ReplyDelete
  6. After looking at the data a bit more, I made a small modification to the code, adding the following to the RecPack stuff:
    RecPack[48] = '*I'

    (* above is a lessthan sign, like in RecPack[2], the comment won't let me put one in)

    This allowed it to read my data into a csv file and the result looked reasonable.

    ReplyDelete
  7. Thanks Freak!
    Looks like you've encountered a new record type. I wonder what it's for. I hope there aren't too many more floating around.

    I added your patch to release z718a.

    I also added a few command line options for debugging.

    The "offset" in your original error message refers to the offset in the device memory (which is assembled from the raw packets).

    You can now:
    --toMemDump=binary mem dump - use hex ed of choice
    --toMemHex=text mem dump with offset labels
    --toMemHexColor=ANSI mem dump with offset labels

    In addition, be aware that I have it wired so you can call the command line "main" from within ipython to use the debugger upon an exception, or make direct calls.

    import bmhack
    bmhack.main(["bmhack","--fromDump", "dumps/z711a-full-serial.cpickle", "--toMemHexColor", "-"])

    ReplyDelete
  8. Excellent. So I'm attempting to read out the data from my device this morning and I think somehow it's got more data than is being read out by the code. Previously when I would use the toPackets option it would end with a bunch of FF's, but not I see data going right up to the end. Also, when I use the toCsv option it reads passed the end of the data and asserts on the zeros in the memory after...

    Also, my name is Roy. Freak is my blog name.

    ReplyDelete
  9. Ok, I modified the MemoryDump() function to read more memory from the armband and got all my data.
    There must be some way to query the armband for how much memory needs to be read, then make MemoryDump read however much it says.
    Either that or just keep reading until you get a string of FF's.

    ReplyDelete
  10. Here is my modification to the MemoryDump() pacs data to make it read more memory.

    pacs = [
    '\x82\x01\x00\x00\x00\x00\x60\x22\x00',
    '\x82\x02\x60\x22\x00\x00\x60\x22\x00',
    '\x82\x03\xC0\x44\x00\x00\x60\x22\x00',
    '\x82\x04\x20\x67\x00\x00\x60\x22\x00']

    ReplyDelete
  11. Hey, great! I've long suspected there had to be more memory, but have never collected more than a couple days before clearing.

    When serial sniffing, I've always seen the GoWearFit software read the exact same locations regardless of how much data I've had (up to a couple days).

    It is painless enough to just figure out the max and read it all. I was guestimating that it must have about 1 Mbyte for claimed 14 days of data (200kbyte of which I was reading, IIRC).

    ReplyDelete
  12. I cleared my armband last night, so I only had about 10 hours of data. About 2/3rds of it was from when I was sleeping.

    Also, 0x2260 is only 8800 bytes, and you were only reading a little more than 2 of those (around 20k). So, no where near 200k. My change above made it read 4 banks of 0x2260, so about 34-35k. Looking at the "toPackets" output my data ended at about 28920 bytes.

    With 10 hours equalling 28920 bytes, that would mean about 69427 bytes per day or 971980 bytes for 14 days, so you are probably right, the device likely has 1MB of memory.

    ReplyDelete
    Replies
    1. Monstruo. Te puedo pagar para que me des accesoria? No se nada de programación, pero me interesa leer mi equipo. Cuanto me contarás?

      Delete
  13. Ah ha. You found a regression. Fixing it now...

    ReplyDelete
  14. Fixed. bmhack-z718b.

    I had broken the memory dump a couple days ago when moving from a straight packet replay to crafting my own memory request packets.

    The MemoryDump functionality is now significantly more robust. I'm also taking Roy's suggestion and by default reading memory up until I see the first full packet payload of 0xFFFFFFFF... The function will now in theory read up to 2 MBytes of address space if it doesn't find the terminating 0xFF 46 times.

    Some experimental findings:

    -The device appears to have 2 MBytes of address space. 2*2^20 starts to repeats the data at 0x0.

    -Reading above 1 MByte address space and the device seems to get VERY SLOW. Strange.

    -All unused memory returns as 0xFF (even though we suspect the device only has 1 MByte of physical memory).

    -Roy gets nominated as pre-Alpha Testing of the Year award. :)

    ReplyDelete
  15. Awesome. Seems to be working great now.
    Thanks.

    ReplyDelete
  16. Thank you sooo much for this guys! I am now going to purchase another gowear fit device. You guys are so awesome!

    I will let you know how it works out for me.

    Thank you again,
    G

    ReplyDelete
  17. Hi NurseGnet,
    By all means test it on your first device!

    This is all experimental, I'd hate to see you put in an investment and it not work for you.

    Clear memory works well for me, so far.

    ReplyDelete
  18. Running b.py --fromSerial com7 --toDump (I just renamed bmhack.py to b.py) worked fine. Ran it twice without problems.

    Running it with --toCsv failed both times I tried. Both times gave this traceback:

    C:\Documents and Settings\Administrator\My Documents\Downloads\bodybugg>python b
    .py --fromSerial com7 --toCsv=bb.csv
    ..........................................................................Traceback (most recent call last):
    File "b.py", line 1180, in *module*
    sys.exit(main())
    File "b.py", line 1121, in main
    SaveStructTabDelim3(packets, dopts['--toCsv'])
    File "b.py", line 984, in SaveStructTabDelim3
    fields, records = RecordTable(packets)
    File "b.py", line 951, in RecordTable
    recs=ReadAllRecords(mem)
    File "b.py", line 853, in ReadAllRecords
    r = ReadRecord(mem[offset:], offset)
    File "b.py", line 826, in ReadRecord
    assert RecPack.has_key(id), "Unknown record ID %i at offset %i" % (id, offset)
    AssertionError: Unknown record ID 0 at offset 675

    ReplyDelete
  19. I have a bodybugg device, and a watch that remotely displays the number of calories, time and other vitals. I was wondering if it would be possible to change your weight and time on the bodybugg device so things would be displayed correctly on the watch. There's no way to do this without their windows/java interface which is very problematic since I do not have windows...

    ReplyDelete
  20. Finnerty, this is great! I am going to buy a armband today. Do you think you could write a .NET library instead of the Python script for a donation ;) If so, drop me an email.

    ReplyDelete
  21. Any thoughts on using this program with a Bodybugg v1 and mac? I couldn't find any specific references to the older version of hardware in terms of compatibility. Also I know there is no support for MAC and BBv1, but could never understand this unless they were using some sort of non-standard USB Driver...

    ReplyDelete
  22. @Anonymous I don't know anything about Bodybugg v1. I suspect you'll have to do some work.

    @LOGIKonline I'm hoping I never have to write anything in .NET. I avoid Windows as much as possible - that's why I wrote this! :)

    ReplyDelete
  23. Hello, I'm new to your blog, lots of interesting stuff here. So does this Data dump work with bodybug? I bought one for my wife, she's wanted one for a while. I have the digital display watch, so if I could just occasionally dump the data, I could keep it going indefinitely. Also, is your program Intel-based Mac compatible?

    ReplyDelete
  24. Has anyone been able to use just the armband and display. I would love some help. Every time I try I get a serial exception, unable to open port com

    ReplyDelete
  25. Joanna - check out my post from a few days ago on CB's most recent post... it's got a walkthrough for both Linux & Windows.

    Anyone have their *entire* device wiped by the clear command? i.e. user data gone, too? we used --clear on my wife's band and now it doesn't want to sync with the watch since it claims the band isn't configured.

    Thoughts?

    Thanks!

    ReplyDelete
  26. you probably are seeing the "NOT SET UP" message, this means it doesnt have all its configuration setup, usually that means the age/sex/weight settings or the time on the unit has been reset.

    I have a tool that works under windows that allows you to reprogram your bugg.

    ReplyDelete
  27. How can I get the utility to reprogram the bugg?

    ReplyDelete
  28. deluge01 said...
    Joanna - check out my post from a few days ago on CB's most recent post... it's got a walkthrough for both Linux & Windows.

    I can't find this walkthrough. Could you provide a link?

    ReplyDelete
  29. Sorry for the delay -- here's the stuff I posted elsewhere:

    Fantastic code... works great under Ubuntu 9.04's live CD. 9.10 is available here (and I'm sure if you hunt, you'll find 9.04, too):
    http://www.ubuntu.com/GetUbuntu/download

    For those who are linux n00bs (like me) here's the set of commands I ran from the live CD to get the bodybugg data - and then clear it:
    operations:
    become root
    get python support libraries (numpy, imaging, serial)
    get linux to ID the bodybugg (make sure it's connected first!)
    dump the bodybugg data to dump.csv
    clear the bodybugg memory

    commands:
    sudo -i
    apt-get install python-numpy
    apt-get install python-imaging
    apt-get install python-serial
    modprobe ftdi_sio vendor=0x11f8 product=0x0007
    python bmhack-z718b.py --fromSerial=/dev/ttyUSB0 --toCsv=dump.csv
    python bmhack-z718b.py --fromSerial=/dev/ttyUSB0 --clear


    For those working under windows and getting "Access Denied" when trying to attach to a COM port, CB's code is missing a Windows-specific line. No fault against CB -- he did a *phenomenal* job with this!!

    in bmhack-x718b.py:
    add a line before line 88 from:
    def OpenSerial(fname="/dev/ttyUSB0"):
    ser=serial.Serial(fname,baudrate=921600,timeout=.01)
    ser.open()

    to:
    def OpenSerial(fname="/dev/ttyUSB0"):
    ser=serial.Serial(fname,baudrate=921600,timeout=.01)
    ser.close()
    ser.open()

    All I did was add ser.close() right before the ser.open(). It's a Windows quirk but it makes the script work like a charm!

    so then all you need to do is put bmhack-z718b.py in the c:\python26 folder and (from a command prompt) run:
    python bmhack-z718b.py --fromSerial=COM3 --clear

    Note that you'll need to try other COM ports if your system has more than 1 comm (serial) port. Check the device manager (run 'devmgmt.msc') under 'ports' to see what COM port your bodybugg is hooked up to. Could be COM3, COM4, COM5, etc.

    In windows, you'll have to download & install python 2.6.4 (http://www.python.org/download/), PIL - python image library (http://www.pythonware.com/products/pil/), numpy (http://sourceforge.net/projects/numpy/files/) and pyserial - python serial (http://pyserial.sourceforge.net/pyserial.html#installation).

    ReplyDelete
  30. Anonymous -- would love a link to the Windows app that reprograms the bugg. Please hook us up!

    ReplyDelete
  31. Could anyone help me with how to do this on my macbook pro?

    ReplyDelete
  32. I had the same error as Freak with ID 51, could not export to csv
    so I did the same thing he did
    RecPack[51] = '<I'
    and now it works.

    ReplyDelete
  33. Went through the whole process and got the following but now the watch just says "ARM BAND NOT SET UP". I think this thing just cleared EVERYTHING. How do I get the user data back on it like age/weight/sex/time...? Where is the cool reprogrammer mentioned above?

    C:\Python26>python bmhack-z718b.py --fromSerial=COM4 --clear
    PIL library not found. ToImage() func unavailable THIS CODE IS DECLARED BY THE AUTHOR TO BE IN THE PUBLIC DOMAIN.
    NO WARRANTY EXPRESSED OR IMPLIED OF ANY KIND IS PROVIDED.
    ...............................................................................................................................................................................................................................................Cleared logged sensor data from device

    ReplyDelete
  34. FAT-Guy44 here!

    THANK YOU for those of you who made this hack available to the public! The clear command worked successfully for me, but does anyone know how to change the following?:

    Time (on display watch)
    User settings

    I'd be happy to paypal someone a few bucks for that windows application the previous poster mentioned.

    ReplyDelete
  35. ok i am lost here and everyone's comments are running together. i have installed Python 2.7(the only version 2 on the website close to 2.6) nubby's for 2.6 the pyserial and the Bmhack.py

    my bodybugg location port is Port_#0002.Hub_#0003 via my device manager

    what is the command i use to clear the body bugg memory please someone help!

    thanks

    ReplyDelete
  36. I have the same problem, bodybugg cleared, however it says armband not setup. could you please direct us to the windows program that we can use to reprogram our bodybugg?

    ReplyDelete
  37. Tried to clear wifes bodybugg and got the timeout exception. Can anyone tell me how to clear properly and maybe reset armband data. Got the bugg second hand and cannot afford to pay $80 for 6 month account. Please contact me at sieverst7@gmail.com.

    Microsoft Windows [Version 6.1.7600]
    Copyright (c) 2009 Microsoft Corporation. All rights reserved.

    C:\Python27\bmhack-z718b.py --fromSerial=COM5 --clear
    THIS CODE IS DECLARED BY THE AUTHOR TO BE IN THE PUBLIC DOMAIN.
    NO WARRANTY EXPRESSED OR IMPLIED OF ANY KIND IS PROVIDED.
    .....Traceback (most recent call last):
    File "C:\Python27\bmhack-z718b.py", line 1180, in
    sys.exit(main())
    File "C:\Python27\bmhack-z718b.py", line 1102, in main
    packets, mem = MemoryDump(ser)
    File "C:\Python27\bmhack-z718b.py", line 147, in MemoryDump
    packets.append(WriteAndReadSerialPacket(ser, pp))
    File "C:\Python27\bmhack-z718b.py", line 128, in WriteAndReadSerialPacket
    return ReadSerial(ser, minLen=n*66, maxLen=n*66)
    File "C:\Python27\bmhack-z718b.py", line 116, in ReadSerial
    raise Exception("ReadSerial timeout after %f sec, expected %i bytes, got %i
    bytes: %s" % (time.time() - t0, minLen, len(s), s))
    Exception: ReadSerial timeout after 3.003000 sec, expected 13200 bytes, got 1319
    ♥e¿ ♫☻ 0è ,ƒ?Å►ûτ]òH╘ì╨┴☼↕G◄ ↕h½4#⌂b▼↕EíO↑0 §*∩ ╧‼+ ║♂╟╫║½♦< <
    ♥e¿ ♫☻ ≡î ,☺⌡◄ ↕╕J0C├]▀↕;qZ‼ ↨ ⌂‼-

    Deleted rest for size!!!

    ReplyDelete
  38. Hi, I had the same issue as july25 poster.
    I tried to create the csv file then, and that worked, Then tried again to clear, and it worked then. computers do amaze me :)

    so after I got the error, i did the following:

    python.exe bmhack-z718b.py --fromSerial=COM5: --toCsv=testal290810.csv
    (this created a csv file that I still have to figure out what to do with)

    then I did:
    python.exe bmhack-z718b.py --fromSerial=COM5: --clear
    (and it cleared, No idea why it gives me the error from july25 poster first, and then minutes later does the right thing, the only thing different is that I ran the csv-forming script first)
    Savvy

    ReplyDelete
  39. Oh, and mine has its memory status back in the green now, but still gives me the same data on my watch as before clearing (still have yesterday's numbers there....
    Pretty perfect for me!

    Savvy

    ReplyDelete
  40. # ./bmhack-z718b.py --fromSerial=/dev/ttyUSB0 --toCsv=dump2.csv
    THIS CODE IS DECLARED BY THE AUTHOR TO BE IN THE PUBLIC DOMAIN.
    NO WARRANTY EXPRESSED OR IMPLIED OF ANY KIND IS PROVIDED.
    ..Traceback (most recent call last):
    File "./bmhack-z718b.py", line 1179, in
    sys.exit(main())
    File "./bmhack-z718b.py", line 1120, in main
    SaveStructTabDelim3(packets, dopts['--toCsv'])
    File "./bmhack-z718b.py", line 983, in SaveStructTabDelim3
    fields, records = RecordTable(packets)
    File "./bmhack-z718b.py", line 950, in RecordTable
    recs=ReadAllRecords(mem)
    File "./bmhack-z718b.py", line 852, in ReadAllRecords
    r = ReadRecord(mem[offset:], offset)
    File "./bmhack-z718b.py", line 825, in ReadRecord
    assert RecPack.has_key(id), "Unknown record ID %i at offset %i" % (id, offset)
    AssertionError: Unknown record ID 168 at offset 675

    ??

    ReplyDelete
  41. Someone mentioned a program to reload user data on the device after a wipe. Several people have asked, but no one has responded with an answer. I guess this program does not exist?

    ReplyDelete
    Replies
    1. I would like to be able to use the watch display but refuse to pay for the sub on a device that I own. I know a little python and would be willing to attempt to create code to do this but has anyone identified where the memory locations for this data is?

      Delete
    2. I'm not sure how accurate this stuff is as far as good coding, but it seems to work

      def calculate_age(born):
      today = date.today()
      try: # raised when birth date is February 29 and the current year is not a leap year
      birthday = born.replace(year=today.year)
      except ValueError:
      birthday = born.replace(year=today.year, day=born.day-1)
      if birthday > today:
      return today.year - born.year - 1
      else:
      return today.year - born.year
      return 0

      ########################
      def SetDeviceName(ser,name="BodyBugg"):
      """Set the name of the device for bluetooth """
      pac = '\x86\xa4' + name
      ser.write(CreateSimpleRequest(pac,"\x00\x78"))
      return ReadSerial(ser, 66)


      ###############

      def SetWeight(ser,weight=150):
      """Set the weight of the person...need to change the chr to pack probably due to > 255 weights likely """
      pac = '\x86\x8c' + chr(weight)
      ser.write(CreateSimpleRequest(pac,"\x00\x24"))
      return ReadSerial(ser, 66)

      #######################

      def SetTime(ser):
      """Set the EPOCH Time and Time Zone based on localtime """
      pac = '\x86\x88' + struct.pack("<i",calendar.timegm(time.gmtime())) + struct.pack("b",(time.timezone/3600)-time.localtime().tm_isdst)
      ser.write(CreateSimpleRequest(pac,"\x00\x10"))
      #sys.stderr.write(struct.pack("<b",(time.timezone/3600)-time.localtime().tm_isdst).encode('hex'))
      return ReadSerial(ser, 66)
      ######################

      def SetBirthday(ser,mm,dd,yyyy):
      """Set the BDay and age of the person """
      pac = '\x86\x8f' + struct.pack("<i",mm) + struct.pack("<i",dd) + struct.pack("<i",yyyy)
      ser.write(CreateSimpleRequest(pac,"\x00\x41"))
      rtn = ReadSerial(ser, 66)
      userage=calculate_age(datetime.date(yyyy,mm,dd))
      pac = '\x86\x90' + chr(userage)
      ser.write(CreateSimpleRequest(pac,"\x00\x23"))
      return ReadSerial(ser, 66)
      ############################

      modified the createsimple request also since some pacs seem to have a byte that i have no clue what does at the end (always the same for the same pac)

      def CreateSimpleRequest(addr,lastword="\x00\x00"):
      [.....]
      # Zero pad the rest of payload
      pk += ListToByteString([0]*(61-len(pk)))
      pk += lastword
      checksum = Checksum(pk[1:]) # sync byte not included
      pk += "\x00" + chr(checksum % 256)
      pk += '\xba\xba\xba\xba'
      return pk



      hope it works for someone (tested on my SP)

      Delete
  42. Any clue on what this could mean?
    This is from OS X 10..4
    Python 2.7.1
    I copied an pasted the commands from bhmhack-z718-1.py

    THIS CODE IS DECLARED BY THE AUTHOR TO BE IN THE PUBLIC DOMAIN.
    NO WARRANTY EXPRESSED OR IMPLIED OF ANY KIND IS PROVIDED.
    Traceback (most recent call last):
    File "", line 2, in
    File "", line 95, in main
    NameError: global name 'Usage' is not defined

    ReplyDelete
    Replies
    1. you didnt copy this?

      class Usage(Exception):
      def __init__(self, msg):
      self.msg = msg

      Delete