Controlling Glass via Bluetooth with a Bluegiga WT12

Here we are, one step closer to my ultimate goal of using the Keyglove as a wireless input device to control a wearable computer. For this installment of progress, the milestone is Bluetooth control of Glass, using a Bluegiga WT12 module and the custom HID descriptor that I wrote about earlier. I won’t go into detail about what that descriptor does, since it’s all documented in the other post, but the short version is that it provides a keyboard, consumer page control (e.g. media), mouse, and raw bidirectional 16-byte packet transference.

First, I wanted to see what I could easily find out about the Glass Bluetooth subsystem. So, I plugged in my WT12 breakout and an FTDI board for simple control from a host PC and opened Realterm. The Bluegiga iWRAP firmware running on the WT12 has a couple of commands that are useful for discovery, namely INQUIRY (find other Bluetooth devices) and SDP (service discovery protocol for finding out what a given device can actually do via Bluetooth).

In order to get glass into “discoverable” mode, all you have to do is go to the normal Glass Home Settings card, and swipe to the Bluetooth card. Glass will then be discoverable for a short while (probably 60 or 120 seconds, but I didn’t measure it).

Once it is discoverable, we can run an inquiry and SDP discovery from iWRAP. Here’s the basic inquiry/name response, including the Class of Device reported by Glass (running XE7 firmware):

in 10 name
INQUIRY_PARTIAL f8:8f:ca:24:5b:84 200714
INQUIRY 1
INQUIRY f8:8f:ca:24:5b:84 200714
NAME f8:8f:ca:24:5b:84 "Jeff Rowberg's Glass"

Pretty basic stuff. The MAC is unique to each device, and the name reported is based (apparently) on the Google account it’s set up with. The Class of Device can be reverse-engineered with the CoD Generator. 0x200714 works out to the following:

• Major service class: Audio (0x200000)
• Major device class: Wearable (0x700)
• Minor device class: Helmet (0x10), Glasses (0x4)

This makes sense based on what Glass is, and what functionality it provides. It’s a wearable device, and it acts as a phone headset. That’s about it from a Bluetooth perspective, as far as I know.

Now we can try an SDP search for all reported services:

sdp f8:8f:ca:24:5b:84 1002 all
SDP f8:8f:ca:24:5b:84 < I 0 I 10000 > < I 1 < U 111e U 1203 > > < I PROTOCOLDESCRIPTORLIST < < U L2CAP > < U RFCOMM I 0d > > > < I 5 < U BROWSE > > < I 9 < < U 111e I 105 > > > < I SERVICENAME S "Handsfree" > < I 311 I 1d >
SDP f8:8f:ca:24:5b:84 < I 0 I 10001 > < I 1 < U f96647cf-7f25-4277-843d-f407b4192f8b > > < I PROTOCOLDESCRIPTORLIST < < U L2CAP > < U RFCOMM I 0e > > > < I 5 < U BROWSE > > < I SERVICENAME S "Glass Identity" >
SDP

This is a pretty sparse SDP list, but not too surprising based on the CoD above. They have the hands-free profile (HFP), a custom Glass Identity service (not sure how this works yet), and that’s it. Note that the SDP entries don’t specify everything the device supports, but only what it provides itself.

The next step is to pair with the device. However, this is where I ran into my first snag, and that is that Glass wouldn’t accept my pairing request when I tried from the iWRAP side:

pair f8:8f:ca:24:5b:84
SSP COMPLETE f8:8f:ca:24:5b:84 HCI_ERROR_AUTH_FAIL
PAIR f8:8f:ca:24:5b:84 FAIL

I thought maybe this was because it had an active link with my Nexus 4, but when I turned my phone’s Bluetooth system off to test this, the same thing happened. It might have worked if I completely unpaired my phone first, but I didn’t want to try that since it’s not a good solution anyway, and there are better alternatives with a bit of work. Apparently, the only thing you can easily do with it out of the box when it comes to Bluetooth is to pair it to a phone using the Hands-Free Profile and PAN profile (for Bluetooth tethering). Therefore, I had to follow the instructions in the Voiding Your Warranty: Hacking Glass session from Google I/O 2013. Fortunately, this only requires a couple of basic steps that did not in fact require voiding my warranty, or even rooting the device.

Here’s the process:

1. Install and set up the Android Developer Tools bundle (a good Glass-oriented guide is here)
2. Install the ADB drivers modified to work with Glass (a bit of a pain on Windows 8, since the modification makes them unsigned and a special restart is required)
3. Install settings.apk and launcher2.apk using ADB

For the purpose of documenting this process, I also installed the excellent and free Droid @ Screen to capture Glass screenshots over ADB, as shown here and above:

Once the ADT/ADB setup is done and the new .apk files are installed, swiping all the way left on the Glass timeline and tapping to access the Settings card results in a new prompt, asking whether you want to use Glass Home or Launcher to complete the action.

Using Glass Home will take you to the normal timeline-based settings area, but selecting Launcher takes you a generic normal-looking Android home screen, with no icons.

Swipe and tap on the app tray icon to bring up the list of apps, of which on my device there are now three:

Selecting the new “Settings” app gives us the classic Android Settings view. Many of these items will generate an “Unexpected error” message followed by dumping you back in the app tray. However, the all-important Bluetooth section works, and that’s good because that is where we need to go.

From here, we can see currently paired devices (e.g. my Nexus 4), and we can discover and pair with other devices, which is what we need to do now.

Select and activate the “Search for devices” option to discover any visible Bluetooth devices nearby. In order for this to work, the Keyglove’s WT12 module has to be in discoverable mode. In iWRAP terms, this means PAGEMODE 3, or else PAGEMODE 4 with no active connections. Fortunately for us, this is how I have it set up.

Since the Keyglove does show up as it should, all we have to do is select it and tap to pair. I have also configured it to use “just works” pairing mode, which doesn’t have man-in-the-middle protection enabled, but as a result means that neither side has to deal with any PIN codes.

Aaand, voila! We’re paired and connected with my combo HID descriptor and ready to have some fun.

At this point, now we issue a “SELECT 1” command to iWRAP running in the serial terminal to select the HID input link, then send raw reports or, for basic ASCII alphanumeric characters and space/backspace/enter, simply type them into the serial terminal (iWRAP translates these for you). The raw HID reports give you more control over exactly what is sent to the device, so I like to use those instead of the automatically “wrapped” ASCII characters.

While you can certainly send your everyday typed alphanumeric characters and symbols and things to Glass, the Glass Home interface makes it not all that useful at the moment. What I am mainly interested in is how to actually control Glass, to navigate through the timeline, select things, and trigger special actions like changing the volume, taking a picture, turning the projector on and off, etc. Through a lot of experimentation, I found that at least most of what I want to do is possible. I couldn’t find a HID report that would instantly snap a photo, but it is still quite possible that one exists—or that a specific combination would work. If anyone knows, I would love to hear from you.

The main reports that I wanted to test were non-alphanumeric keyboard codes and all of the Consumer Page reports I could try. I am sure that I didn’t manage to find all of the functionality possible, but I’ve got a good start. The keyboard reports are simple enough; each one is a single code, triggered by sending a report with that code in one of the six available key slots followed shortly (I used a 20ms delay) by a report with 0x00 in that same key slot. This effectively simulates a press and release. Remember from the last post that in iWRAP, the keyboard raw report format is this:

0x9F 0x0A 0xA1 0x01 [modifier] 0x00 [key1] [key2] [key3] [key4] [key5] [key6]

So, sending the “Enter” key (code = 0x28) looks like this:

0x9F 0x0A 0xA1 0x01 0x00 0x00 0x28 0x00 0x00 0x00 0x00 0x00

0x9F 0x0A 0xA1 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00

Here are the relevant keyboard codes that I found to be useful in working with the timeline UI:

• 0x28 – Enter (select)
• 0x29 – Escape (cancel/back)
• 0x4f – D-pad Right
• 0x50 – D-pad Left
• 0x51 – D-pad Down
• 0x52 – D-pad Up
• 0x65 – Menu

There aren’t that many, but with these actions, you can replace the vast majority tapping or swiping the side of Glass. (The new double-tap/hold that you can do in the XE7 web browser is one that I haven’t figured out yet.) The Menu report gives you access to the app’s menu, assuming it has one. There’s no other way to do this with Glass as far as I am aware, so there’s an added bonus: new functionality! Here’s an example of using the Menu report to access the wallpaper setting on the Launcher app, followed by basic D-pad selection:

So what about other shortcuts? For that, we look to Consumer Page reports. These have a different structure, if you recall:

0x9F 0x05 0xA1 0x02 [field1] [field2] [field3]

It’s set up for a 3-byte-wide set of bitfields instead of distinct key slots. The same concept for press/release cycles applies here though. Here are the relevant consumer page reports that I found to be useful for Glass:

• 0x00 0x00 0x01 – Volume Up
• 0x00 0x00 0x02 – Volume Down
• 0x00 0x00 0x08 – Toggle projector on/off (no sounds generated)
• 0x00 0x08 0x00 – Home
• 0x00 0x16 0x00 – Projector off (w/sound, like escape/swipe down)

These actions combined with the D-pad navigation and selection above give you the power to do just about anything with Glass remotely. More detail on Android-specific HID input can be found here, though I haven’t fully grasped everything there myself yet. Here’s an example of volume control with the above reports:

But what about actual tapping, for controlling touch-based apps or accessing settings that you can’t get to with the D-pad? This is mainly useful for development/porting purposes, since an app designed with Glass in mind will by definition not rely much on touch in the way that apps designed for touchscreens are. Even so, having this ability would be handy.

This is where the 3rd report in our custom HID descriptor comes in: we can actually use the mouse in Android! Who knew? Yes, it might look a little odd, but it works just fine. You can send typical “click” actions (though I could not see any different behavior between different button values, e.g. left vs. right vs. middle), and you can send three axes of motion data for X/Y movement as well as scrolling. Here’s an example of accessing the detailed volume settings after clicking on the button on the right

There you have it, everyone. Complete control of Glass via Bluetooth HID is actually quite easy. Other than the initial pairing hurdle, it’s no more complicated than any other Bluetooth HID implementation. Now I just need to integrate the proof of concept code with the Keyglove firmware. This may be a bit of a challenge because there are a few different ways to do it—should I use pure touch controls (e.g. tap to “swipe” left or right), or touch + motion (e.g. tap to activate, move to navigate)? Or both? It’s a flexible platform, so it won’t be locked down in any case. I will try to configure it both ways and see what feels more natural.

On July 6, 2013   /   Development, Hardware, Uncategorized   /   5 Comments