Image Captioning for Ambient Awareness on a Sidewalk

Ambient awareness on a sidewalk is critical for safe navigation, especially for the people who are blind or visually impaired. An affordable and interactive system is necessary for this purpose. In this paper, we present the the outcome of the experiment of off-the-shelf image captioning systems. Design and implementation of a system embedded in a RPi3 is part of the experiment. The main component of the system include: a. generation of meaningful caption from images, b. implementation of personalized feedback mechanism for efficient communication with a minimal cognitive load, and c. an interactive user interface and energy efficient integration that account for multiple configurations to be more inclusive. In particular, the performance of off-the-shelf image captioning systems was compared e.g., Microsoft Cognitive Service, Clarifai, Google Vision API, and IBM BlueMix to determine best platform for meaningful caption generation. We implemented three different schemes, namely text-to-speech synthesis, haptics, and ring tone to provide personalized feedback. The implemented system interface is energy efficient and interactive to provide ambient awareness. The objective evaluation of the fully integrated system was performed on the sidewalk. In particular, we focus on the accuracy of the captioning system and the usage analytics to provide helpful tips on the spot and to understand long term system behavior.