Video Games Are All Greek to Eelke Folmer

The word video derives from the Latin “to see.” It should logically follow, then, that people who are blind or visually impaired would not play video games very well. Add the word “haptic,” however, from the Greek haptikos pertaining to sense of touch, and now video games take on a whole new meaning.

Eelke Folmer, an Assistant Professor in the Department of Computer Science and Engineering at the University of Nevada in Reno, is doing just that… reinventing the video game, that is.

“Haptic feedback is not a prevalent modality,” Folmer said. “Yet your skin is the largest organ on your body, so you can communicate a lot of information through touch.”

Two boys who are blind play Pet-n-Punch an adapted videogame version of Whack-a-Mole. They use Nintendo Wii Remotes, which perform haptic feedback.

Haptic feedback may seem like a foreign concept, but chances are you’ve encountered it before. It’s the same technology that causes your cell phone to vibrate when receiving an incoming call. It also causes videogame controllers to oscillate in response to something happening on the screen.

It’s the latter that Folmer has utilized to bridge the gap between video games, which were previously designed for those with sight, and people who are blind or visually impaired. His controller of choice is Nintendo’s Wii Remote, because it’s cheap (between $15 and $25) and “does everything we need it to do,” he said. One thing he needs it to do: get kids who are blind or visually impaired to exercise.

“The Wii gets people off the couch and moving around,” he said. “It’s an example of a video game that actually has health benefits.”

Childhood obesity is prevalent among kids who are blind or visually impaired. Research shows that these kids are less active because they have less opportunity—they don’t know how to play sports, it may be unsafe, etc. The Wii Fit offers an elegant solution. The accelerometer, within the remote, measures motion. Couple that with its haptic response, and Folmer found he could convey a whole spectrum of information through this device. And it’s more convenient than traditional exercise.

“You can play against the computer or with friends so you don’t need a sighted guide,” he said. “You’re not running around, so it reduces the risk of injury.” He aptly named his version of Wii Fit “VI Fit.” VI stands for visually impaired. People can download the games directly from vifit.org for free. All games are designed by one of Folmer’s grad students, Tony Morelli. His original project was an accessible version of Guitar Hero, called “Blind Hero.” The player wore a glove that contained pager engines in each finger, which would buzz when notes to the corresponding phalange should be played. The prototype cost $2,000, but Morelli and mentor Folmer knew they were on to something.

(left) A man who is visually impaired plays Blind Hero. (right) A young boy who is blind plays VI Tennis at Camp Abilities.

The first game Folmer and Morelli developed for VI Fit was tennis, because the Wii version yielded a high level of physical activity, he said.

Exercise can be categorized into three classes: light; moderate; and vigorous. Kids are recommended to engage in 60 minutes of moderate to vigorous exercise a day (40 minutes moderate; 20 minutes vigorous). An example of moderate exercise would be walking; running or sprinting would be vigorous. With this in mind, Folmer set out to reach his own version of moderate to vigorous activity.

Tactile dowsing: the player moves the Wii remote in the horizontal plane (left); the closer the Wii remote points to the target direction, the more continuous the perception of vibrotactile feedback will feel (right).

VI Tennis works like echolocation. The player can hear his/her opponent serve and hear the ball bounce on their side of the court. In addition to audio cues, the player can also feel vibration (through the Wii Remote in his/her hand) when the ball hits the court. The distance of the ball is indicated through haptic feedback. “As soon as the ball bounces on your side, you feel a short burst,” Folmer said. “Then as the ball gets closer, you feel a longer pulse, which indicates the timeframe in which you hit the ball.”

He tested his game at Camp Abilities at the College in SUNY Brockport, founded by Dr. Lauren Lieberman, PhD. Lieberman’s camp focuses on skills in accessible sports for kids who are visually impaired, thus the perfect arena to test VI Fit. Upon her contact with Folmer, Lieberman connected him with John Foley, who had done similar research on how kids with low vision could play “exergames” (video games involving exercise) like Dance Dance Revolution.

Together, this team tested two versions of the tennis game; one with just audio cues and another with both audio and “vibrotactile” cues (haptic feedback). The kids performed higher with audio and haptic feedback, which supports studies that have shown presenting feedback in multiple modalities (e.g. audio combined with haptics) reduces percentage for error.¹

The team also applied heart monitors to the virtual athletes to measure activity. The tennis game yielded a moderate level, equal to that of jogging, Folmer said. This was enough to be considered healthy, but still not vigorous enough.

Figure 1: A man performs dowsing in VI Bowling. Figure 2 shows him throwing the ball.

One reason for this was that VI Tennis, only incorporated a temporal component. Folmer separates haptic feedback into two components: spatial (in space) and temporal (in time). For the tennis game, it didn’t matter where you hit the ball (spatial component); it only mattered when you hit it (temporal component). Yet Folmer realized much of physical activity requires the former. After creating a VI Bowling² game, involving space, not time, Folmer attempted to marry the two in a virtual version of Whack-a-Mole, called “Pet-n-Punch.”

Unlike its predecessors, Pet-n-Punch³ uses both hands (a Wii Remote in each). When one of the controllers vibrates, the player must “punch” the rodent (just don’t hit the kitties or you lose points). The game incorporates both spatial and temporal components, since you have to swing either your right or left hand (spatial) when that certain remote vibrates (temporal).

Pet-n-Punch yielded much higher energy expenditure—requiring a more comprehensive upper-body workout in the use of both arms—than its parent VI games. Folmer’s still not satisfied, however. He’s looking into developing a VI Hurdles game, which would require use of the entire body as players jump over virtual hurdles (a controller would vibrate within the timeframe to jump).

When people think of “extreme gamers,” they may picture a pale-faced teenager, glued to the dull glow of a computer screen, rounding out a 24-hour session on World of Warcraft or Call of Duty. Folmer says the real extreme gamers are those individuals with severe disabilities, who are very fond of gaming. Creating videogame interfaces for people who can’t see the screen or can’t use conventional controllers feels like “putting a man on the Moon,” he said. “And it’s an opportunity to develop all sorts of solutions that can be beneficial for everyone.”

Society is already moving away from the traditional desktop setting that ushered computers into the mainstream. Smart phones are one example of this phenomenon. In fact, an interface that didn’t require a screen at all could make the danger caused by driving while texting—200,000 car accidents in the US alone resulted from driving while “intexticated” in 2010, according to the National Safety Council—a thing of the past.

But first, Folmer’s focused on offering an alternative way to exercise for children who are blind in this country—nearly 60,000 kids are legally blind in the US, according to a 2009 report conducted by the American Printing House for the Blind.

As an added bonus, parents can play these VI Fit games with their kids who are blind or visually impaired, which increases socialization among families. Studies have shown that social families work out more and are thus more fit. So perhaps the VI Fit can help break down the stigma that kids who are blind or visually impaired can’t play sports.

“Parents are sometimes too overprotective of their kids for fear of injury,” he said. “But if you play with your kid and understand that s/he can actually exercise, there’s the potential for parents to take their kids to the gym or buy a tandem bike.”

References:

1. Hershenson, M. Reaction time as a measure of intersensory facilitation. Journal of Experimental Psychology 63 (Mar 1962), 28{93.
2. VI Bowling Study
3. Pet-n-Punch Study
4. Vifit.org