New apps for visually impaired users offer virtual labels for controls and a way to explore images

Visually impaired iPhone users now have two new free tools at their disposal, developed by a team based at the University of Michigan. One can read labels on a control panel while the other identifies features in an image so users can explore it through touch and audio feedback.

VizLens is essentially a screen reader that can function in the real world. It reads the label at the direction of the user, who points his fingers at the button of interest on the control panel. With it, users can employ their smartphone cameras to sense and operate various interfaces in their everyday environments, including home appliances and public kiosks.

“A visually impaired user can take a picture of an interface, and we use optical character recognition to automatically detect text labels. A user can first become familiar with the layout on their smartphone touchscreen. Then, they You can move your finger over the physical device.” The control panel, and the button under the app user’s finger will speak,” said Anhong Guo, UM assistant professor of computer science and engineering, who led the development of both apps.

Another app, ImageExplorer, helps people with visual impairments better understand the content of images. To this end, Guo and his team have integrated a suite of object detection and segmentation models – including Meta’s Detectron2 visual recognition library and Google OCR (optical character recognition) and image analysis models – to help visually impaired users detect To be able to see what is in the image and how the different objects are related to each other.

Guo aims to give agency to people with visual impairments when alt text is missing or incomplete, as AI-generated captions are often not sufficient.

“There are many automatic captioning programs out there that blind people use to understand images, but they often contain errors, and are impossible for users to debug because they can’t see the images,” Guo said. “Our goal, then, was to stitch together a suite of AI tools to give users the ability to explore images in greater detail with greater agency.”

Upon uploading an image, ImageExplorer provides an in-depth analysis of the image’s content. It gives a general overview of the image, including detected objects, relevant tags, and a caption. The app also features a touch-based interface that allows users to explore the spatial layout and content of the image by pointing to different areas.

ImageExplorer is unique in the level of detail it provides. It gives users a comprehensive description of the objects in an image, to what extent a person is wearing what type of clothing and what activities they are engaged in, as well as the position of these objects in the image.

“ImageExplorer helps users understand the content of an image, even when they can’t see it,” Guo said.

Hundreds of visually impaired, user-testing participants have experimented with VisLens and ImageExplorer, providing feedback to Guo’s team, which continues to develop these tools. First discussed in 2022, ImageExplorer is a much newer concept than VizLens, which Made its academic debut in 2016, Some of its details need further refinement – for example, most of the top is simplified to “shirts” and the various tools within ImageExplorer sometimes give conflicting information.

“The accuracy depends on the models we use, and ImageExplorer will improve as they get better,” Guo said. “Despite these flaws, Results we presented in 2022 show that ImageExplorer enables users to make more informed decisions about the accuracy of AI-generated captions.”

Guo is also looking forward to the feedback that will come with the public deployment.

“We’ll be able to see how people use these tools and adapt them to their lives,” he said.