A mulsemedia framework for delivering sensory effects to heterogeneous systems

Technological advances in computing have allowed multimedia systems to create more immersive experiences for users. Beyond the traditional senses of sight and hearing, researchers have observed that the use of smell, taste, and touch in such systems is becoming increasingly well-received, leading to a new category of multimedia systems called mulsemedia—multiple sensorial media—systems. In parallel, these systems introduce heterogeneous technologies to deliver different sensory effects such as lighting, wind, vibration, and smell, under varied conditions and restrictions. This new paradigm shift poses many challenges, mainly related to mulsemedia integration, delay, responsiveness, sensory effects intensities, wearable and other heterogeneous devices for delivering sensory effects, and remote delivery of mulsemedia components. In addition, new approaches to interacting with multimedia applications have emerged such as multi-touch interfaces, voice processing, and brain–computer interfaces, giving rise to new kinds of complex interactive systems. In this article, we underpin fundamental challenges for delivering multisensory effects to heterogeneous systems. We propose an interoperable mulsemedia framework for coping with these challenges, meeting the emerging requirements. It is achieved through the evolution of an open distributed mulsemedia system. We changed its core following architectural and design patterns to accommodate different profiles of communication, connectivity, and sensory effects metadata standard according to the need of mulsemedia applications and devices available in the user’s environment. The results include case studies where the framework has been duly applied.