Multimode hyperspectral data fusion for fish species identification using supervised and reinforcement learning

Our goal is to use multiple spectroscopy methods in a single system and develop novel multimode spectroscopic data fusion techniques for fish species identification in real-time. We collected spectral signatures of fish fillets from six fish species using four hyperspectral imaging systems: (1) Reflectance spectral imaging in the visible and NIR (VIS-NIR), (2) Reflectance spectral imaging in the short wave infrared (SWIR), (3) Fluorescence visible spectral imaging with UVA and violet excitation, (4) Raman imaging with a 785 nm laser excitation. All fish fillet samples were confirmed by DNA testing. We built multiple classification/ dimension reduction combination methods to calculate the average sensitivity and associated variance for each class and each spectroscopy mode. In our prototype, the derived statistics are used to form policies for Monte Carlo prediction reinforcement learning. We compared the results of our weighted fusion decisions against individual spectroscopy mode decisions to show an overall sensitivity improvement. We believe this is the first reported use of reinforcement learning applied to multimode spectroscopy data classification in food fraud applications.