Design of emotion-aware mobile apps for autistic children

Sensor technologies and facial expression recognition are now widely used by mobile devices to sense our environment and our own physical and mental state. With these technologies today, we have the ability to sense emotions and create emotion-aware apps. One target group that would benefit from emotion-aware Apps are autistic children as they have difficulty understanding and expressing emotions and they are keen mobile device users. However, current mobile apps aimed at autistic children are not emotion-aware. This led our team to design a suite of Apps, called CaptureMyEmotion, that uses wireless sensors to capture physiological data together with facial expression recognition to provide a very personalised way to help autistic children and their carers understanding and managing their emotions. This paper describes how we designed CaptureMyEmotion and it discusses our experience while using sensors and facial expression recognition to detect emotion. It presents in more details the first App we developed for Android phone and tablets, called MyMedia. MyMedia enables children to take photos, videos or sounds, and simultaneously attach emotion data to them. The photos can then be reviewed together with a carer providing them a new way to understand emotions and discussing their daily activities.