RSA technology advances with AI


Innovators and scientists have been trying to find out new ways and means to help visually impaired people over the years starting with the magnifying glass and Braille technology. The most recent of them is the Remote Sighted Assistance (RSA) technology which provides a great support for visually impaired people in performing their daily tasks. Now the technology is going to get even better with Artificial Intelligence.

What is RSA?

Remote Sighted Assistance is basically a conversational assistance technology designed to aid visually impaired individually by connecting them with human agents-volunteered or paid-through computer vision technology (live video calls).  The agents are supposed to guide the individuals with low or no sight in navigating the tasks that require sight such as finding an empty table in a restaurant, reading a food package label or identifying what colour an object is. The service is rendered in scene description, navigation, task performance and social engagements. Simply the agents see the world of these individuals through a camera and verbally recreate it for them. But with the existing computer vision technology, the agents might find it difficult to fulfil certain requests.


The most commonly recognised issue is the poor network coverage in certain areas, especially indoors. Due to the slow or weak internet connection, video starts to lag or stuck making it very difficult to communicate and perform the task.

Nonetheless the researchers who study RSA technology have recognised more serious issues related to the technology. In a recent study presented at the 27th International Conference on Intelligent User Interfaces (IUI) in March, they highlighted five emerging problems with RSA that need serious attention;

Recognising that objects commonly identified as obstacles by smartphone cameras may not be considered obstacles by visually impaired individuals, but instead are useful tools. For example, a wall bordering a sidewalk may be displayed as an obstacle in common navigational app, but a visually impaired person walking with a cane may rely on it to navigate their steps.

– Helping users navigate their environment when a live camera feed may be lost during low cellular bandwidth, which frequently occurs in indoor settings.

– Recognising content on digital LCD displays, such as flight information in an airport or temperature control panels in a hotel room.

– Recognising texts on irregular surfaces. Often, important information is printed in ways that make it difficult for human agents assisting visually impaired individuals to read; for example, medication instructions on a curved pill bottle or a list of ingredients on a bag of chips.

– Predicting how out-of-frame people or objects will move. Agents must be able to quickly communicate environmental information in a user’s public surroundings, for example other pedestrians or a moving car, to help the user avoid collision and keep the user safe. However, the researchers found that it is currently difficult for agents to track these other people and objects, and nearly impossible to predict their trajectories.

Overcoming the challenges

The researchers and the other stakeholders collaboratively have been able to come out with new development ideas and designs which would benefit both the users and agents of RSA. They highlighted how the RSA has been more mentally demanding for the agents and therefore the new designs would offload much of the work to the computer.

They are hoping to use computer vision to give the agent a very immersive experience and provide them with the mixed reality technology so that it is easier for them to navigate the context. A 3D reconstruction algorithm run in computers also will be used in this case. Also, the researchers will be developing a system to directly help the users get some basic information about their environment based on computer vision technology. In fact, it would profit a myriad of visually impaired individuals all over the world. 

By Induwara Athapattu