The awesome power of the Toby eye tracker

My apologies for having such a long radio silence. As you might've guessed, life is becoming a bit complicated and keeping me busy. I've also been wrapped up with a side project (which will be featured later, but you can see the early prototype at the beginning and end of the video) for using my voice bank.

Anyway, I promised to get back to posting. But just to get something up quickly, here is an example video that my rehab facility asked me to produce for them. The is a demonstration of how I use my Toby tracker to assist me in my day to day life. The video is edited and as was provided to my rehab facility. Nonetheless, I think it does quite a good and detailed demonstration of how it works.

Eye Tracking, Windows Surface & a Broken Arm

If you've been following this blog for some time, you note that have been anticipating the lack of use of my hands and have been looking for solutions to allow me to continue to access my computer. For some time now I've been keeping an eye on the PC eye go by Tobii (see what I did there?) And I've been pushing my rehab facility to look into it.

Now, my rehab facility had access to one of the earlier Tobii eye tracking computer solutions but it was prohibitively expensive and not to the usual computer standards I am used to. Ringing in at about C$10,000. So once I knew about the Windows surface and the PC eye go, for total of about $3000 I was pushing my rehab facility pretty hard to get one for trial, which they recently did. And fortunately, I was the test case for it. As a matter of fact, I'm using it right now in conjunction with Dragon NaturallySpeaking. I did try Windows speech recognition which is functional, but frustrating as this comprehension was not nearly as good.

yeah, that hurt

The goal of the evaluation was to find out how easy it is to use for my day to day computer needs and how easy it is to keep calibrated. Also evaluating any physical demands it places on me. And unfortunately, my evaluation turned into a far more realistic experience than expected as about two weeks ago I fell and broke my right arm in four places. Three and the rest, one in the shoulder. Eliminating the use of my hands and any ability I had remaining to access my desktop computer. I immediately became dependent on the eye tracking.

That said, broken arm and all I have to say I'm extremely pleased with the eye tracking experience. In particular I'm glad I had it on hand after the fall (see where they did there?) As without it I would've had zero computer access. Once I figured it out, it is very easy to keep calibrated. However I do have to point out it is very sensitive to changing lighting conditions. For example in the evening when the sun is setting and the lights in the house are going on. This can really wreck calibration. So as lighting conditions change you may have to frequently recalibrate. But, if you do this often enough as you start noticing changes in the precision. It's not that big of a deal. However if you wait too long, functionality can be completely lost, in my case I have to ask for help to push the on-screen buttons.

After calibration, there are two primary ways you use the eye tracking to interact. Via mouse emulation which I don't particularly like and gaze selection, which is awesome.

Mouse emulation actively tracks ways you working on the screen and moves the mouse cursor accordingly. This can be quite frustrating as they can be a little "drift" between where you are looking at where the cursor is. This results in chasing the mouse cursor across the screen. Needless to say, annoying. Secondly, to send a click you have to dwell where you want the click that happened. A.k.a. stare at it. Now as your eyes are always making slight movements, even when you think you are fixed on a particular location this can be just enough to prevent selection from happening. And then added to it the tendency to look at the cursor from problem number one… Again annoying.

That said, it does work and it does have its place.

Now the gaze the selection is really great. There is a bar overlaid on one side of the screen which you look at to select the desired mouse function, and then you look where you want that action to happen. It also has its own on-screen keyboard which takes up about half of the screen. This may sound like a lot of screen real estate, put on the tablet it makes the buttons not that huge. But the are extremely easy to look at and type with.

I had a few issues with keyboard layout, just for my habits of typing and the tendency to want to do some programming/scripting but all in all it's very functional and very responsive. It is of course slower but if you're using it in conjunction with Dragon naturally speaking. Very pleasant experience. The eye tracking keyboard by itself is perfectly functional for shorter replies to e-mails, Facebook and twitter updates and so forth.

Using VNC & eyetrack to remote
connect to desktop to use Dragon

One of the things that have appreciated the most, is using VNC to remote connect from the tablet to my desktop. This is been extremely handy (see, did it again). Using the eye tracking on the tablet as essentially the mouse to access some specific programs and functionality I had on my desktop.

I would have to say there's only two things about the gaze selection I had issue with. The first off, is there is no auto completion reword expansion built into the on-screen keyboard. Even the default Microsoft on-screen keyboard has this as an option. As you're essentially hunting and pecking one letter at a time to type. This would've been an extremely useful addition. It is however easy enough to install a third party piece of software such as wordQ to add this functionality. And then add a little auto hotkey script on top of that for easy access (just so you don't have to turn the keyboard page to access the number pad)

The second thing has to do with the built-in scroll tool. While it is perfectly functional, it is a little frustrating to use when scrolling through say and longer webpage. Use for example have to look at the bottom of the screen while, at the corner of your vision keeping an eye on the top of the screen to know when to stop scrolling. He really needs a page up/page down button. This is something that's also easy to solve with auto hotkey scripts, as a matter of fact, already written and tested.

I also tried plugging the tablet into my windows of seven desktop with a 22 inch monitor. The functionality was nearly identical with some changes due to Windows 8 Metro interface not being present. It was again extremely easy to set up and calibrate, and of course had been the much larger screen made selections extremely easy as I had much larger look targets.

All in all, I'm extremely pleased with the Windows surface to Pro tablet and a Toby eye tracking functionality. And, seeing as the fall has pretty much put an end to the use of my arms and hands in any functional sense for computer access. I am in the process of ordering one. I am however a bit undecided as to whether or not I want to order the Windows surface Pro three.

One thing to consider however is that on battery power the Windows surface 2 Pro with the eye tracking plugged in is only given me about 2 1/2 hours of power. Not that big of a deal, but you do have to keep it in mind.

Also, if you're anticipating the loss of the function of your hands. I would suggest it's worth acquiring the eye tracker (perhaps not the tablet right away) and becoming familiar with it earlier rather than being pushed in at the deep end. Before I broke my arm, I had a few occasions where it needed to interact with it manually. Particularly when I was learning about the calibration.

Looks like Siri Proxy Is Dead & Bit of an Update

So where have I been?

There's been quite a bit of chaos going around recently. For a couple reasons, in no particular order:

The family has been heavily wrapped up with moving house as I'm not able to navigate stairs safely anymore. As always, moving house is always stressful and annoying. But what is really unfair about it, all the work is falling on the shoulders of my wife.

Discovered a nice apartment that would work for us and be rather accessible right from the start. Nonetheless, we will be sad to be leaving this home and neighborhood. We love it, and our neighbors.

Please join me when I say "I hate moving"

I lost a considerable amount of use of arms hands since our last post. Making everything, difficult. This is necessitated an increase in volume of people coming through the house to help me out, leaving a very limited amount of time to get to the computer. When I do, those two or three hours are often taken up with correspondence, reading or World of Warcraft hearthstone…

Ah crap... more wasted time with watercraft

It's nice that I found a video game I can play again, but damn you World of Warcraft! I thought I was free of your corruptive influence. Damn you back to the depths of hell!

The lack of use of my hands has also greatly limiting my creative problem solving for day--to-day issues as I can't really use any of the solutions anyway. This means I'm going back to more work in front of the computer, which has been slow for the above-mentioned reasons ;-)

On the flip side, my rehab facility has given me access to a Windows surface tablet with a Tobii eye tracker. That has been quite interesting and we are laying the groundwork for when I will need one.

The death of siri proxy

About a month ago at WWDC Apple announced the release of IOS 8 and at about the same time appears to have patched their guzzoni service on which Siri proxy
relied. My Siri proxy stopped working, I tried on and off for about three weeks to get back up and running with no luck. It's dead.

Such a shame, it was awesome while it lasted. However, IOS8 is promising to open up the SDK for Siri with home kit so that may be used for various home automation's solutions.
(Crosses fingers)

So I upgraded my phone to IOS7 and started to look at alternatives.

Obviously, there is nothing as good out there right now but I needed a way to verbally, remotely lock and unlock my door. The easiest solution was to use my always on raspberry pi to check my Gmail account for messages from a specific center with the specific catchphrases.

With a bit of research I managed to write a Ruby script. I Chose Ruby because of the convenience of regex matching and the familiarity I got writing scripts for my Siri proxy.

The script is not fancy at all. But it does get the job done. It works with the following logic.

1. Every 30 seconds checked my Gmail account
2. is there an unread message from a specific e-mail address
3. if yes, does this message contained trigger phrases?
4. If yes, do specified action.

If you like to give it a try, you can find the Ruby script here my Gmail checker

You will however have to install the Gmail gem http://rubygems.org/gems/ruby-gmail

The way this script is presently written, it runs just fine on raspberry pi. but not on Windows. However, it should not take much to get it running on the Windows machine. I just did not figure out the syntax to send the string to Wget.exe on Windows. For your convenience, I included it in the zip.

Status report

You've probably noticed that I've not been posting much recently… My apologies for that, I do have a backlog of topics I want to get around to posting, but it's becoming increasingly difficult for me to do so.

I seem to be crossing a number of tipping points which is making things much more difficult for me. In particular, my balances rather severely affected. I can still walk, but am prone to falling. This is probably the biggest reason I've not been posting much, as I spend my days on the ground floor and my computer is in the basement. Stairs are scary!

The bandage they gave me
was a bit oversized.

I don't have much use of my arms and hands at all anymore which I'm going to say is probably the biggest reason why my balance is shot. No arms, no balance. As a matter of fact, I took quite the fall the other night. I was standing behind my son while he was brushing his teeth for bed. For some reason for right knee buckled and I fell backwards clipping the edge of the toilet with my right eyebrow. Leaving quite the boxers scar About 1/2 inch long. My four-year-old was very good about it and just asked why I had a hole and was leaking. Got lucky on that one though. Another inch or two towards the toilet seat and it could've been much worse. Ending up with more than just a "sexy scar" for the ladies.

My speech is just starting to be affected, just enough to start messing with my voice recognition. I'm also finding I'm speaking more softly, mumbling and having to repeat myself more. Fortunately my voice bank is all ready to go. Hopefully I won't need it for quite some time.

Steve Gleason Super Bowl ad for eye tracking
with Microsoft surface

Between these two issues, I'm starting to seriously consider alternative computer options For you zero, constant access. I've been monitoring Tobi eye tracking for some time, which is the solution I think I will go. They have a nice package for sale with the Windows Surface2 & Eyetracker for about $3000. Expensive, but the prices came way down from where It was.

That being said Tobi has a partnership steel series gaming trying to bring eye tracking mainstream. These are due out this summer,  should drive the price down even further. Going to try to hold on until then…

But! Still rolling!

With everything that's going on, and the balance issues the wife and I have been looking to move. We find a place that should work out for us. I ground floor apartment with elevator access from the garage. This should at least give me more in the way of computer access. Took a long time to find a good apartment that would be good for the family, accessibility and my mental health. But finally found one. We just put our home up for sale.

We both hate to move. We love the neighbors and the neighborhood fortunately however we are only moving a few blocks.

So until the dust settles sometime in July, I don't expect to be able to spend much time in front of the computer. But please, feel free to follow my daily adventures on twitter until then.