Real-time multimodal sensory detection using widefield hippocampal calcium imaging

The hippocampus is a complex structure that has a major role in learning and memory. It also integrates information from multisensory modalities, supporting a comprehensive cognitive map for both spatial and non-spatial information. Previous studies have been limited to real-time spatial decoding, typically using electrodes. However, decoding hippocampal non-spatial information in real time has not been previously described. Here, we have constructed a real-time optical decoder driven by the calcium activity of large neuronal ensembles to decode spatial, visual, and auditory information effectively. Using advanced machine learning techniques, our rapid end-to-end decoding achieves high accuracy and provides a multisensory modality detection method. This method enables the real-time investigation of hippocampal neural coding and allows for direct neural communication with animals and patients affected by functional impairments. The ability to decode multimodal sensory inputs in real time thus forms the basis for an all-optical brain-computer interface.