Neural network-based small cursor detection for embedded assistive technology

Assistive technology (AT) is invaluable to people with special educational needs and disabilities, enabling them to interact with computers more efficiently. It is able to improve the interactivity between humans and computers for learning purposes. However, there is a lack of accessibility and interactivity with touchscreen devices and visual content that AT users encounter. These issues are critical in providing much-needed assistance and enhancing the daily lives of disabled individuals. Therefore, we propose an AT framework based on neural networks using embedded systems for improving the accessibility and interactivity of AT users. The proposed framework utilizes the Jetson Nano and is mainly used with a speech-to-intent neural network to process speech and move cursors. When improved with cursor object detection, the framework could obtain the location of the cursors in external displays and move the cursors of other devices. Since cursor datasets are very limited and not many detection models are up for the task, we investigated the use of Slicing Aided Hyper Inference (SAHI) pipeline along with two fine-tuned models, Fully Convoluted One-Stage (FCOS) and Task-aligned One-stage Object Detection (TOOD), to identify the minimum data required for these models to work optimally. With less than 120 annotated images and a data multiplier of 5 and 30, both models were able to achieve ~ 52 and ~ 60mAP, respectively. These results were comparable to performance on other small object detection datasets. In addition, we also present a working proof-of-concept for our proposed embedded assistive technology framework.