Seeing AI Logo
News

Seeing AI iOS App: Recognizing People, Objects and Scenes

"Turning the visual world into an audible experience.

Microsoft has just released the FREE Seeing AI iOS app.  This amazing AI – Artificial Intelligence – app narrates the world for users with visual impairements and blindness by describing people, objects, scenes and text.  This app is fully accessible with VoiceOver.

AI

AI is when a computer system is able to perform tasks that normally require human intelligence, such as visual perception, speech recognition, decision-making and translation between languages.  AI is being used to enhance our daily lives; examples of AI include personal assistants (such as Siri and Google Assistant), real-time traffic movement for GPS routes, ridesharing apps, self-driving cars, plagiarism checkers, spam filters, online shopping, and so much more!

The video below is an overview of the Seeing AI app.

 

Seeing AI Categories

The Seeing AI app is divided into categories.  The following categories are located at the bottom of the app’s home screen:

Short Text

Short Text speaks text as soon as it appears in front of the camera. I have used this to quickly find and read things such as a return address on a letter, quick note, etc. FYI:  Seeing AI did read my husband’s neat handwritten note but was unable to decipher my personal sloppy writing!  

The video below demonstrates using the Short Text feature.

Documents

The Document feature is basically an OCR app that provides audio guidance to capture a printed document and reads the text – in its original format – outloud. I tested this feature on several pieces of mail, including an invoice and a flier.  Like most OCR apps, normal text is easily captured.  Seeing AI correctly read an invoice that was set up like a table with multiple lines and boxes.  

The following video demonstrates how to use the Documents feature.

Products

Screenshot of Seeing AI app showing Products reads the barcodes to identify products.  Audio beeping enables the user to know if a bar code is detected and the faster beeping sound provides additional information to enable the user to fully locate the bar code.  For example:  When searching for a bar code on a box, there are 6 sides to box.  Hold the iPhone away from the box in order for the entire side of the box to appear in the camera lens.  If there is a bar code on this side of the box, the app will beep and the bar code will be recognized – I did not have to adjust the camera to focus only on the bar code!  I tried the Seeing AI app with a vareity of food products from my kitchen pantry.  When holding a jar of salsa, I simply held the iPhone in front of the jar and slowly turned the jar.  The app beeped when the bar code was located and beeped faster when the full bar code was in the camera lens.  This bar code scanner worked well – the image did not need to be focused and seemed to work from various distances.  With the jar, I held the iPhone approximately 6″ inches from the jar.  With the cereal box, I held the iPhone about a foot away.

The video below demonstrates how to use the Product feature.

Person

Person uses facial recognition AI to recognize family and friends; it will describe the person and their emotions.  Users do have to initially identify the person before the app will recognize the person independently.  To add a person, select the Menu button in the top left corner then select Face Recognition.  This screen will display the people that you have already identified.  To identify a new person, select the Add button. The reverse camera will turn on and the app will ask you to take three pictures of the person you would like to identify.  Have the person hold the iPhone (as the reverse camera is activated) or use the Switch to Back Camera button to take a picture of a person in front of you.  When I tried the Person feature, the app typically announced the gender, age, hair color or characteristic of the person and emotion of the person.  If the person’s name was previously added to the app, then the app would also announce the name of the person.  To assist visually impaired users to take a picture, the app provides directional guidance you so that you can center the people in the picture and it tells you how many people are in the picture.

The video below demonstrates the Person feature.

Scene

Scene will describe the scene around you.  Microsoft states that this is an experimental feature as they continue to fine tune the AI’s scene identification skills.  I tried a variety of scenes that were identified correctly, including a family room scene, a pool scene, a parking lot, etc.  I did try one scene that included a mini australian shepherd (small dog) that the app identified as a “black cat laying on a wooden floor”.  Overall, the scene feature worked better than I expected!

 

Currency

Currency will identify US paper bills.  Note:  This feature is currently not available!

The video below demonstrates how Seeing AI will be able to identify US currency.

Recognizing Images in Other Apps

Want to know more about an imagine that came in an email, Twitter, and other apps?  Open the app > open the image (double tap on the image)  > Share > Recongize with Seeing AI.  The text within the image is announced along with the image’s scene.  

The video below demonstrates how you can use Seeing AI to recognise images and text within images in other apps.

Thank you Microsoft for apply AI technology to solve real-life challenges for users who are visually impaired!  Educators and students: Please share ways and activities that YOU use the Seeing AI app!

Collage of Seeing AI app

By Diane Brauner

SHARE THIS ARTICLE
Photo of Kate explaining the tactile telescope model to a high school student.
Guide

Accessible astronomy: Tactile telescope

A smiling child holding a toy eagle
Activity

Bird song book and bird-related activities

EPIQ logo
Event

EPIQ (Experience Programming in Quorum) 2024