Image sensing with multilayer nonlinear optical neural networks

Optical imaging is commonly used for both scientific and technological applications across industry and academia. In image sensing, a measurement, such as of an object’s position or contour, is performed by computational analysis of a digitized image. An emerging image-sensing paradigm relies on optical systems that—instead of performing imaging—act as encoders that optically compress images into low-dimensional spaces by extracting salient features; however, the performance of these encoders is typically limited by their linearity. Here we report a nonlinear, multilayer optical neural network (ONN) encoder for image sensing based on a commercial image intensifier as an optical-to-optical nonlinear activation function. This nonlinear ONN outperforms similarly sized linear optical encoders across several representative tasks, including machine-vision benchmarks, flow-cytometry image classification and identification of objects in a three-dimensionally printed real scene. For machine-vision tasks, especially those featuring incoherent broadband illumination, our concept allows for a considerable reduction in the requirement of camera resolution and electronic post-processing complexity. In general, image pre-processing with ONNs should enable image-sensing applications that operate accurately with fewer pixels, fewer photons, higher throughput and lower latency.