The Emperor's Smart Clothes...

Ever since reading Rainbow’s End by Vernor Vinge in 2007, I’ve been fascinated by the idea of wearable computing and alternate forms of computer user interfaces. While the book came out in 2006, the story is set in 2025 and it envisions amazing leaps in how technology can and will transform our lives. While much of it may seem obvious now, the book was released prior to the first iPhone when the RAZR flip phone was all the rage.

When I got my first iPhone in 2008, I was captivated by its potential but immediately felt constrained by it’s on screen keyboard. Now I could suddenly get online and do all these great things, but I had to fumble with the onscreen keyboard to type anything. 

In Rainbow’s End, children could chat with friends in class, and search online, etc; all without the teacher even really noticing. I desperately wanted the smart clothing from the book that provided custom gesture recognition, and with enough practice/experience would make text entry effortless whether sitting at a desk or walking down the street. 

The basic goals were simple;

1. Support an interface for wearable computing

2. Have it be built around a learning system to best personalize itself for each user.

3. Have it work without touching or moving against anything.

4. Make it always readily available.

5. No vision required.

In researching, the closest I could find was a product called the Twiddler which was a one handed chording keyboard for mobile entry. I wanted a solution that didn’t require holding something in your hand though, to cover points #3 and #4 above.

Unfortunately nothing yet existed, but I thought if something even half as good as what was in the book could be built it would still be a huge improvement. The sensitivity of the iPhone’s accelerometer and gyroscope demonstrated what was already available, so what I thought were the key building blocks already existed.

Surface mount accelerometer/gyroscope chips were only about 1 square cm, so the idea was to mount a number of the sensors on rings and/or a glove to capture small hand movements/gestures without having to actually hold anything. As the user could be swinging their arms as they walked down the street, additional sensors above the palm/wrists/forearms could be used to help normalize the movement of individual fingers. Like the Twiddler, cording could be used to expand the range of possible inputs. 

As I experimented more and more with the sensors of my iPhone, I realized the gestures didn’t need to be rigidly defined. Maybe to enable/disable keyboard mode it could be as simple as wiggling both hands up and down simultaneously. Each use of `backspace` would be a valuable input to the ML model to not only identify that something was handled incorrectly, but the user’s final intent could be captured as well. (Ex. gesture "A"  was not → note).

Beyond just text, scrolling could be as simple as moving the pointer finger in a circle, `alt-tab` could be a flick of the wrist, etc. The gyroscope would capture the direction of gravity, so the direction or angle of the hang up/down could be used to differentiate meaning.

By toggling different input/interface modes, the inadvertent movements of a train/car/elevator/etc could be prevented from producing false input.

To support blind entry, simple haptic feedback could be used to register the recognition of a gesture. Like the little click a key makes on a keyboard.

While the ideas and simple experiments were abundant, I unfortunately lacked the CE or EE background to actually connect a number of small sensors together with a microcontroller. The Arduino Uno wasn’t released until late 2010, and I didn’t begin to develop against one until years later. Ten plus years ago, I had no idea how to solder well, or even have a microcontroller communicate with a surface mount MEMS sensor.

I figured the iPhone had only been out a few years by that point, so if I just gave it time someone would bring something to market. I mean Rainbow’s End won the Hugo award in 2007, so I figured whole rooms of CE undergrads working on wearable computing would be likely building something better than I could even imagine. 

Unfortunately it’s 2021 now, the iPhone has been out ~14 years; and the closest I’ve seen to Mr. Vinge’s vision was Google Glass. 

Now surrounded in a room littered with various Arduino hardware, it looks like I may have to build one myself after all. Luckily I’ve learned a lot in the past ten years, and the RP2040 looks like the perfect platform to get started with.

Wish me luck!

$12 USB Pedals = Excellent

Spending many many hours in front of the computer, both for work and leisure; I've invested heavily in setting up the best human-computer interface I could. Even tiny improvements in productivity add up quickly when multiplied by the hours/month/years I'll be able to take advantage of them.

In that spirit I don't shy away from tools or systems that have steep learning curves if I believe they offer some efficiency improvement once mastered. This has pushed me towards things like Linux, the dvorak key layout, GNU Emacs, and the Kinesis keyboard.

[Here is a picture of my desk at work. I've attached a large touchpad to the top of my keyboard so I can put the entire keyboard in my lap and not have to move my arms to use the mouse.]

Obviously having the above sitting on your desk at your office, makes you stick out a little (especially in conformist Japan). During one discussion with a coworker I was joking that the Kinesis even offers an optional foot pedal for those interested.

Now at least the old Kinesis foot pedals simply generated a key press to help with something like holding down the shift key to reduce strain on your hands or arms. Thinking further though, I wondered if someone hadn't come out with a generic USB foot pedal that could do something more.

While still chatting with my coworkers a quick search turned up exactly what I was looking for. Amazon now sells a $12 USB foot pedal!

Some Googling around quickly determined that these pedals have some driver software that's leaves you a little wanting, but which updates the hardware so it knows which key press to transmit when stepped on. A number of issues here, yada yada yada; but the hardware will connect to anything using the USB HID keyboard driver!

I didn't want to mess with the hardware, but I figured if it used a standard USB keyboard driver then I could code something up and make it do whatever I so desired ($ sudo make me a sandwich).

Talking to a friend about my idea, he later pointed me to Aleksandr Levchuk's VIM Clutch which was the exact kind of thing I wanted to do. Still wanting a bit more flexibility to define pedal actions, and to prevent burning myself while soldering; I just tossed together a tiny ruby script/driver to handle the pedal.



Now after getting everything working, using ruby and libusb; I've re-created the VIM clutch in software using the xautomation package. You can drop in your own onpress and onrelease functions though to enable the above script and $12 pedal to do anything you want.

I still haven't decided how to best use my new pedal yet, but here are a few ideas;

• Play/Pause for music
• Meta-key for Emacs
• Bring my chat window to center screen when pressed, then return when released
• Record keyboard macro for Emacs while held down, playback macro when pressed quickly.
• Next tab in Firefox

One of the nice things of implementing the driver in Ruby, is that I can easily have it do all of the above just by detected what my active window is.

Anyways, hope this helps!

Tips & Tricks with Clojure Reducers

After noticing a few others encounter the same issues I ran into when first playing with Clojure's reducers, I thought I should collect some of what I discovered into a quick blog post.

First off, reducers are *AWESOME* and fully realize the power of parallel sequences but as they are still alpha have a few rough edges to work around. I'm sure in the near future it will become much more polished but until then be aware of the follow.


Note, clojure.core.reducers will be abbreviated as "r" below;

• In many cases r/map and r/reduce are faster than their core counterparts, but not in all cases. At this point in time I only use the r/ variants for larger collections (> 1000 elements) and that has worked well.
• The parallel part of reducers doesn't kick in until you use r/fold instead of reduce. For r/fold to be faster than r/reduce though, it needs to be able to "divide-n-conquer", and hence needs to be able to quickly divide up the collection.
  Summary: r/fold is only faster than r/reduce when operating on a vector.
• (into [] (r/map inc (vec (range 10)))) works, but uses r/reduce instead of r/fold.
  Instead it's faster to use the following to take advantage of fold;
  
  
  so,
  (fold-into-vec (r/map inc (vec (range 2000))))
• Reducing into map is many times slower than reducing into a single value or vector. Unfortunately merge and merge-with become the bottlenecks here, and this is the biggest rock in my sandal at the moment. :(
• Don't forget mapv, which was introduced in Clojure 1.4 I believe. It provides a nice middle ground between core/map and core.reducers/map.

The function I'd specifically like to tune further with reducers is the following;