Google Glass App Captions Real-Life Conversations

[Sarah LeBoeuf]

Google Glass App Captions Real-Life Conversations


Hearing-impaired Google Glass wearers can get closed captioning in real life thanks to Georgia Tech's new app.

There's been no shortage of negativity surrounding the wearable computer Google Glass, from the dangers of the manufacturer's own urging not to be a "glasshole." [http://www.escapistmagazine.com/news/view/137723-Driving-With-Google-Glass-Almost-As-Unsafe-As-With-Smartphone] Researchers at the Georgia Institute of Technology, on the other hand, have found a positive use for Google Glass: displaying closed captioning for real-life conversations in order to help the hearing impaired.

As shown in the video, adding closed captioning requires the use of a separate smartphone app. This companion Android app, called Captioning on Glass, converts the speech to text and sends it to the Glass, where it appears on the display. According to Jim Foley, a professor at Georgia Tech's School of Interactive Computing, the app lets the wearer focus on the speaker while filling in any blanks with a text transcription. "If hard-of-hearing people understand the speech, the conversation can continue immediately without waiting for the caption. However, if I miss a word, I can glance at the transcription, get the word or two I need and get back into the conversation."

From the video, it's easy to see that this isn't quite a perfect system, as the need to speak into a separate device and wait for that to be transcribed creates unnatural gaps in the conversation. Professor Thad Starner says the companion app is necessary for the system to work, however. "Glass has its own microphone, but it's designed for the wearer... The mobile phone puts a microphone directly next to the speaker's mouth, reducing background noise and helping to eliminate errors." The speaker can also edit any errors in transcription, although presumably that would only further slow down the discussion.

With a starting price of $1,500 and plenty of establishments banning its use [http://www.escapistmagazine.com/news/view/135206-Google-Glass-Banned-From-Alamo-Drafthouse], Google Glass is far from mainstream. That's a steep price to pay just to wear a computer on your face--but for the hearing impaired, $1,500 might be worth it to be able to converse (almost) normally again.

Source: Digital Trends [http://www.news.gatech.edu/2014/10/02/researchers-create-software-google-glass-provides-captions-hard-hearing-users]

Permalink

 

[gardian06]

well if this is deemed to be a viable medical aid that assists in a daily life function then it is possible that medical insurance companies could subsidize the purchase of a google-glass device much like they currently do with hearing aids (though most hearing aids range from $300 to $1,000), or at the very least another company could take this line of logic, and run with it making a "watered down" version directly for this purpose (better mic so you don't have to talk into a phone, and maybe the ability to delineate what audio is coming from what source by color coding), and then instead of marketing this to just hearing impaired you could market as a "deaf hearing aid"

 

[legend forge]

The all important question:

Can I listen to music and put subtitles of what people are saying to me?

 

[Svarr]

This is pretty nice! Glad to see some better uses for the Glass out there, maybe with this and some other future improvements the bans on Glass will be lifted on some places?

 

[Parasondox]

See, now this is a positive thing coming from Google Glass. Able to help those with hearing difficulties cope more better with communication. Hope the microphone within the Glass is able to hear from further distance and clear.

 

[Deshin]

Not that I have anything against the hearing impaired but wouldn't it be a more worthwhile venture to have designed it from the start to be a form of language translator? Sure it would be slow and you'd also need either an internet connection to access Google Translate or a downloadable, portable version of it being lugged around on your phone, but think of the utility of it! Sure it'd be a lot of teething problems at first and probably wouldn't work with EVERY language and dialect, but come on, one step closer to universal translators we wear on our heads!

 

[weirdee]

I'm pretty sure that most establishments will scramble to allow these kinds of aids much like they did for allowing service animals.

 

[Lightknight]

As a medical device this is pretty cool. I'm deaf in one ear so I've always been concerned about what would happen if I lost the good one but haven't bothered to learn sign language since the good one has perfect hearing.

Even so, people talking to my immediate right (deaf zone) are often having to repeat themselves and this would resolve that.

That is, if there wasn't a complete stigma surrounding wearing the glasses. It would be interesting to call someone out for bitching about me wearing one if I bought one as a medical aid. Like picking on a blind person for walking into a store with a dog or walking with a white cane.

 

[Imperioratorex Caprae]

As a medical device this is pretty cool. I'm deaf in one ear so I've always been concerned about what would happen if I lost the good one but haven't bothered to learn sign language since the good one has perfect hearing.

Even so, people talking to my immediate right (deaf zone) are often having to repeat themselves and this would resolve that.

That is, if there wasn't a complete stigma surrounding wearing the glasses. It would be interesting to call someone out for bitching about me wearing one if I bought one as a medical aid. Like picking on a blind person for walking into a store with a dog or walking with a white cane.

I'm almost deaf in my right ear, its tone-dependent at the moment but getting worse as years progress, so I can understand the fear and annoyance of having to get people to repeat themselves.

OT: My dreams already have had subtitles and captions (not joking, its really creepy). I don't think I could handle real life being captioned... I'd be looking for a live studio audience next.

 

[Cid Silverwing]

Seems we're that one ever so tantalizing step closer to having a BabelFish device on our heads. :3

 

[Lightknight]

As a medical device this is pretty cool. I'm deaf in one ear so I've always been concerned about what would happen if I lost the good one but haven't bothered to learn sign language since the good one has perfect hearing.

Even so, people talking to my immediate right (deaf zone) are often having to repeat themselves and this would resolve that.

That is, if there wasn't a complete stigma surrounding wearing the glasses. It would be interesting to call someone out for bitching about me wearing one if I bought one as a medical aid. Like picking on a blind person for walking into a store with a dog or walking with a white cane.

I'm almost deaf in my right ear, its tone-dependent at the moment but getting worse as years progress, so I can understand the fear and annoyance of having to get people to repeat themselves.

OT: My dreams already have had subtitles and captions (not joking, its really creepy). I don't think I could handle real life being captioned... I'd be looking for a live studio audience next.

Ah, well, an unfortunate word of warning on profound single sided deafness. It actually has a list of symptoms that will kick in if the deafness increases to the point of having no functional hearing ability. So if you find yourself sweating in an air-conditioned room with a lot of different noise sources going at once, be ware that you may need to excuse yourself to a quieter area. Car trips can also be particularly bad. I didn't know single sided deafness actually had side effects that explained everything I was going through. I mean, I've been totally deaf in that ear for my entire life and didn't find out until I was in my mid-20's that there was a reason why I responded so negatively to certain situations:

Hopefully you will always have at least some functional hearing in that ear so none of the following will ever apply to you.

SSD is known to cause: [http://en.wikipedia.org/wiki/Unilateral_hearing_loss]

Irritability
Body language and mannerisms which appear socially awkward or unusual
Frequent headaches, stress
Social isolation
Chronic interpersonal communication difficulties due to inability of brain to isolate or beam form sounds and voices of other individuals

Appearance of anxiousness even in low noise situations
Jumpiness
Trouble figuring out where sounds are coming from.
Variable light dizziness
Trouble paying attention to what people are saying: "evasive" behaviour.
Misdiagnoses as ADHD
Seeming lack of awareness of other people's personal space and moods since brain is hyper-focused on deciphering auditory information in lieu of non-verbal social cues.

Lack of sound depth: any background noise (in the room, in the car) is flat and wrongly interpreted by the brain. The effect is similar to what happens when trying to hear someone speaking in a noisy crowd on a mono TV. The effect is also similar to talking on the phone to someone who is in a noisy environment (see also: King-Kopetzky syndrome)

Inability to filter out background noise or selectively listen to only the important portion of the noise in the environment.

For sensorineural hearing loss, the lack of input coming from the damaged sensory apparatus can cause "ghost beeps" or ringing/tinnitus as the brain attempts to interpret the now missing sensory data. The frequency and the volume of the noise can increase according to one's physical condition (stress, fatigue, etc.). This can aggravate social problems and increase the difficulty of speech comprehension.

Talking loudly or "broadcasting": the affected person cannot perceive the volume of his or her voice relative to other people in the same room or close company, resulting in being characterized by others (who may be located beyond normal auditory range) as domineering or boorish

 

[Strazdas]

this would be nice in theory. in practice the problems are thus:

Noise. you have to either try translating noise, which is stupid, or filter it, whitch means deaf people wont understand me (seriously, phones filter my voice sometimes as noise, have to pick a phone to buy where it doesnt, first world problems ech).

Multiple audio sources. like forementioned music and speaking. also would it try translate song singing, because that usually ends terrible.

Google text to speech algorythm. if youtube autogenerated subtitles are any indication - the mistranslation is terrible to the point where at best its a laugh and at worst you may get in trouble from using it.

The all important question:

Can I listen to music and put subtitles of what people are saying to me?

if you got a directional mic and said person does not hide behind the speaker i dont see why not?

OT: My dreams already have had subtitles and captions (not joking, its really creepy). I don't think I could handle real life being captioned... I'd be looking for a live studio audience next.

you know how info shows up above people in Watch Dogs when you look at them. i wish i could do this kind of thing in real life, look at the person, get subtitles for him, other inportant info (not like watch dogs, more like his mood, ect or if hes annoyed with me, because i havent developed the ability to sense that like some people.). its kinda creepy, but IMO if everyone could see whether the other person likes him or not it would be a much more honest world.

 

[MetalMagpie]

OT: My dreams already have had subtitles and captions (not joking, its really creepy). I don't think I could handle real life being captioned... I'd be looking for a live studio audience next.

you know how info shows up above people in Watch Dogs when you look at them. i wish i could do this kind of thing in real life, look at the person, get subtitles for him, other inportant info (not like watch dogs, more like his mood, ect or if hes annoyed with me, because i havent developed the ability to sense that like some people.). its kinda creepy, but IMO if everyone could see whether the other person likes him or not it would be a much more honest world.

Just names would be good!

I'm really terrible at remembering people's names. At work I can resort to checking the company website (which has labelled photos of everyone), but in normal life I end up in awkward situations where I've known someone for six months but didn't manage to commit their name to memory when we first met and now it's too awkward to ask. :/

 

[Sarah LeBoeuf]

well if this is deemed to be a viable medical aid that assists in a daily life function then it is possible that medical insurance companies could subsidize the purchase of a google-glass device much like they currently do with hearing aids (though most hearing aids range from $300 to $1,000), or at the very least another company could take this line of logic, and run with it making a "watered down" version directly for this purpose (better mic so you don't have to talk into a phone, and maybe the ability to delineate what audio is coming from what source by color coding), and then instead of marketing this to just hearing impaired you could market as a "deaf hearing aid"

That's a really good point! It'll be interesting to see whether this takes off enough for it to be considered a viable medical expense.

 

[hermes]

It seems like a cool application, but we are still a long way to make it usable.

I can't seem to picture it working better than the automatic close captions in youtube.

 

[ExileNZ]

This would also be very useful for non-native speakers. It's much easier to miss a key word or two when you're working in another language, and the "unnatural" pause in conversation is still much more natural than looking in a dictionary, asking them to repeat for the umpteenth time or, worst of all, staring at them blankly like an idiot.

This is of course without taking into account the possibility of on-the-fly translations.

 

[gardian06]

Not that I have anything against the hearing impaired but wouldn't it be a more worthwhile venture to have designed it from the start to be a form of language translator? Sure it would be slow and you'd also need either an internet connection to access Google Translate or a downloadable, portable version of it being lugged around on your phone, but think of the utility of it! Sure it'd be a lot of teething problems at first and probably wouldn't work with EVERY language and dialect, but come on, one step closer to universal translators we wear on our heads!

upon thinking on this further I have the feeling that if the translation was a key feature then it might be seen as something of an "invasion of privacy"

if we consider that many people who are raised bi-lingual they will use one language to speak to others, and then go to the other language for more "private" conversations, and if this device was able to not only capture what they were saying from say across the room, and then translate it then to some it could be seen, and eavesdropping in on someones living-room conversations.

though (I can not think of a better way to put this) it would finally end some peoples paranoia over "what those people over there are saying in a language I don't understand"

the other implication of the translation system would be that within a short time it could remove the need for interpreters (though if they could formalize a significant way to translate sign-language (taking into account the different dialects, and some of the minutia) then it might either denigrate, or aid the system.

 

[Strazdas]

OT: My dreams already have had subtitles and captions (not joking, its really creepy). I don't think I could handle real life being captioned... I'd be looking for a live studio audience next.

you know how info shows up above people in Watch Dogs when you look at them. i wish i could do this kind of thing in real life, look at the person, get subtitles for him, other inportant info (not like watch dogs, more like his mood, ect or if hes annoyed with me, because i havent developed the ability to sense that like some people.). its kinda creepy, but IMO if everyone could see whether the other person likes him or not it would be a much more honest world.

Just names would be good!

I'm really terrible at remembering people's names. At work I can resort to checking the company website (which has labelled photos of everyone), but in normal life I end up in awkward situations where I've known someone for six months but didn't manage to commit their name to memory when we first met and now it's too awkward to ask. :/

indeed, names would be a good start. i also have hard time remmebering names at first, but eventually work it out. and yes labeled photos at work is a godsend

I usually just refer to such person as "You" "him/her" till i hear someone else call out the name then try to use it as much as i can to help remmeber. Most of the time they dont notice.