Sign in to view more
Sign in to view more
Dr. Ko’s research interests include multimodal sensor fusion for human-computer interaction that incorporates both video and audio (speech) inputs such that the integration results in an increased performance in detection, recognition, and tracking, thus enabling real-time interactions. The research in multimodal sensor fusion requires investigations on optimization for signal processing in both image and acoustics, as well as advanced machine learning algorithms, which include deep learning. To illustrate an example of the fusion problem, consider a two-hour movie, or a short movie clip as its subset, which is intended to capture and present a meaningful (or significant) story in the video to be recognized and understood by human audience. What if we substitute the human audience with that of an intelligent machine or robot capable of capturing and processing the semantic information in terms of audio and video cues contained in the video? By using both auditory and visual means, the human brain processes the audio (sound, speech) and video (background image scene, moving video objects, and written characters) modalities to extract the spatial and temporal semantic information, that are contextually complementary and robust. Smart machines equipped with audiovisual multi-sensors (e.g. CCTV equipped with cameras and microphones) should be capable of achieving the same task. An appropriate fusion strategy combining the audio and visual information would be key in developing such artificial general intelligent (AGI) systems. My research work has been on various sensor fusion techniques to combine the audiovisual information cues for video content analytics. There can be a wide range of fusion strategies at various information levels (e.g., feature, decision, and semantic) to extract meaningful information by providing the attention mechanism in terms of weighting significance of the cue to represent the intended world. In light of the recent advancement of deep-learning, my research effort has been to develop new fusion strategies by addressing the relevant research issues toward solving many applications requiring artificial general intelligence.