Dynamic Non-Regular Sampling Sensor Using Frequency Selective Reconstruction

Both a high spatial and a high temporal resolution of images and videos are desirable in many applications, such as entertainment systems, monitoring manufacturing processes, or video surveillance. Due to the limited throughput of pixels per second, however, there is always a tradeoff between acquiring sequences with a high spatial resolution at a low temporal resolution or vice versa. In this paper, a modified sensor concept is proposed which is able to acquire both a high spatial and a high temporal resolution. This is achieved by dynamically reading out only a subset of pixels in a non-regular order to obtain a high temporal resolution. A full high spatial resolution is then obtained by performing a subsequent 3D reconstruction of the partially acquired frames. The main benefit of the proposed dynamic readout is that for each frame, different sampling points are available, which is advantageous since this information can significantly enhance the reconstruction quality of the proposed reconstruction algorithm. Using the proposed dynamic readout strategy, gains in the peak-signal-to-noise ratio (PSNR) of up to 8.55 dB are achieved compared with a static readout strategy. Compared with the other state-of-the-art techniques, such as frame rate up-conversion or super-resolution, which are also able to reconstruct sequences with both a high spatial and a high temporal resolution, average gains in PSNR of up to 6.58 dB are possible.