Classification and Direction Detection of Ambient Sounds on Microsoft HoloLens to Support Hearing-impaired People

Hearing-impaired people are exposed to greater dangers in everyday life, due to the fact that they are not able to perceive danger and warning signals. This paper addresses this problem by developing an application, that could help by classifying and detecting the direction of ambient sounds using Microsoft HoloLens 2 devices. The developed application implements a client-server architecture. The server-side REST-API supports not only the classification of sounds from audio files via deep-learning methods, but also allows the results of the sound source localization to be saved and read. The sound source localization is performed by a Maix Bit microcontroller with a 6-channel microphone array. For the user integration and interaction with the application, a 3D scene has been designed using Unity and the Mixed Reality Toolkit (MRTK). The implemented application showcases how classification and direction detection of ambient sounds could be done on the Microsoft HoloLens to support hearing-impaired people.