A Wearable Embedded System for Assisting Cognition of Visually Impaired People by Street Scene Description

A Wearable Embedded System for Assisting Cognition of Visually Impaired People by Street Scene Description

Visually impaired people face many challenges in their everyday lives. Thus, many studies have been conducted to provide guidance and support that are more helpful for them. However, while existing studies mostly focus on navigation and obstacles avoidance, there is less focus on semantic information of a current scene, such as color, emotion, object description, etc. In this paper, we propose a wearable embedded system that aims to convey semantic information to visually impaired people based on an embedded module-based approach and deep learning scheme. Furthermore, the proposed system is designed to develop an automated assessment of urban streetscapes through landscape indexes. First, the current scene in front of visually impaired people is captured by the webcam of the system. After the captured image is segmented using a semantic segmentation model, we perform street scene description scheme based on combining image semantic segmentation and color-based emotion classification. Besides, the landscape indexes are calculated to produce the data describing the current scene. Finally, the descriptions of the current scene are delivered to visually impaired people by a headphone or speaker. Moreover, a system prototype is implemented to verify the feasibility of the proposed system. In total, street scenes on 27 locations are selected in this study, and each scene includes 8 streetscape images taken from 8 different directions (front, right front, right, right back, back, left back, left, and left front). After image semantic segmentation and landscape index calculation, these results provide a method to evaluate urban streetscapes.