New Color Filter Patterns and Demosaic for Submicron Pixel Arrays

Typical image sensor the color filter size is the same as pixel size. Current technology trends demand higher imager resolutions and hence smaller pixel size. There are several routes to reducing pixel size, for example back side illumination structure, and elevated organic compound photodetector to enhance QE, sensitivity, and full well capacity. However, the bottleneck of reducing pixel size is inability to reliably reduce color filter size in sub-micron regime. Current state-of-the-art Color Filter Array (CFA) technology only support production worthy pixel size down to 1.1um. Color filter material consists of pigments to define the spectrum of the color filter, and its ability to resolve the minimum pixel size has to maintain acceptable color error while achieving color fidelity & uniformity [1]. In dealing with sub-micron pixel color image sensor, new innovated CFA pattern is essential to compensate for CFA pigment limitations. INTRODUCTION As today camera phone sensors are striving to deliver image quality close to the image quality delivered by DSCs, the demosaic algorithm plays a very pivotal role in camera phone image processing pipe [2]. A new de-mosaic algorithm is developed to interpolate the sub-sampled new color patterns. Previously, irregular color patterns, relative to Bayer, are already being introduced, for examples, new CFA patterns where each primary color pixel surrounded by secondary color pixels in both vertical and horizontal directions (RGBCY and RGBCWY) to combat severe crosstalk [3]; and white pixel patterns (RGBW, RCCB, etc.) sensors to boost sensitivity & SNR for small pixel mobile imagers [4]. The ratio between color filter size and pixel can be varied based on the technology capabilities and desired pixel size. Use color filter size that is larger by the pixel results in different color patterns, which under some constraints can be made periodic. De-mosaicking the new CFA patterns presents significant challenge due to the reduced full well and signal to noise ratio in submicron pixel. Besides, the overlapping of the color filters within a pixel cell increases the complexity in the color process. Therefore, it requires innovative de-mosaic algorithm to achieve satisfactory color performance. We have therefore developed a robust de-mosaic algorithm to allow for graceful degradation in presence of noise, and at the same time be tolerant to high color correlations. DEVICE FORMATION We have developed sub-micron pixel arrays of low noise readout, 8MP, 0.9um BSI image sensor as the test vehicle, and implemented color filters with different than 1:1 ratio to pixel size, integer or fractional, as illustrated in Fig. 1. This device take advantage of the dedicated Multi-Project-Wafer, MPW 65nm CIS BSI shuttle processed at TSMC. Color filter and micro-lens were subsequently processed at VisEra Technologies with the baseline CF material designed for 1.1um pixel, and the CFA patterns of 1.5X and 2X-pixel size were applied, while maintaining 1:1 of micro-lens pattern. The pixel architecture adopts a 2x4-shared in order to gain pixel cell utilization as photo-detector. Design of the low readout noise imager adopts readout architecture comprises a column amplifier, a comparator, and a 10 to 12-bit programmable column parallel Single-Slope ADC along with a global 12-bit current DAC, which is programmable to generate various slope of ramp signals with implemented timing schemes to suppress readout noise. DEVICE CHARACTERISTICS The performance of imager device has been limited by the MWP BSI baseline, and it features full well capacity of 2000-e @saturation, and 1650-e of linear full well, with sense node conversion gain of 92.57 uV/e, and sensitivity of 1520 e-lux-sec @530 nm. The lack of FWC of this device is mainly due to that MPW shuttle photo-detector process baseline was so designed to serve larger pixel sizes, therefore not optimized for the sub-um pixel. The read noise is measured at 8X analog gain, and input referred of 1.8-e_rms has been achieved. The photo-response characteristics, as shown in Fig. 3, exhibits the detector light response of various color channels, with green channel tops the output signal. The mismatch of the pixels with the same color channel, G/YW/CY, is caused by the overlay control of CFA coating process, and it also attenuates the white channel response. Besides, the detector blooming remains since no lateral overflow path from photodiode to floating node under the transfer gate was implemented. The normalized QE, spectral response curves of the combined primary and complimentary color channels are illustrated in Fig. 4. Since the CFA material was not optimized for the sub-um pixel, and combined with the overlay control challenge when printing 1.5X-pixel size filters, it worsen the pixel cross-talk and color separation. It is our objective in this study to examine our de-mosaic algorithm robustness on mitigating the intrinsic cross-talk of sub-um pixel, and to maintain the trade-off balance between SNR and color error. DEMOSAIC METHODOLOGY & RESULT The de-mosaic algorithm proposed herein is essentially a universal de-mosaic algorithm. It performs maximum a-posteriori (MAP) estimation [5] of the unknown RGB color tuples at all pixel locations given a compressed spectral image captured using an arbitrary color filter array (CFA) pattern. The forward spectral compression or mosaicking operation is defined by a sparse NN × 3NN matrix transformation, where N is the size of the observed CFA image and 3N is the size of the full 3-channel RGB color image to be recovered. Each row of the forward transformation characterizes: (1) linear combination of the red, green, and blue spectral components sampled at a given pixel location and (2) cross-talk among neighboring pixels surrounding the given pixel. The problem of recovering the full 3N-dimensional RGB image given the Ndimensional CFA data is ill-posed. Therefore, we recover the unknown signal using MAP estimation, wherein a novel image prior model, comprising a linear combination of quadratic terms, allows us to recover the full color image without significant blurring and color-aliasing artifacts. In general, iterative MAP restoration is hardware inefficient and computationally intensive for real-time applications. Our proposed approach involves pre-computing and storing the inverse operator required for MAP restoration of the full RGB image from the observation data. We purposely use quadratic data-fidelity and prior model terms to ensure that the inverse operator is independent of the observed CFA data. The inverse matrix is computed offline for a specified imaging sensor and CFA pattern. For the demosaic application, the inverse matrix turns out to be sparse and block-wise Toeplitz [6], thereby reducing MAP estimation to computationally efficient convolution operations. The number of unique convolution filters, constituting unique rows of the inverse block-wise Toeplitz matrix, is 3PQ, where PxQ represents the period of the CFA pattern. We used a set of 24 standard Kodak color images to evaluate the quality of our de-mosaic algorithm. The 3channel color images are first compressed to single-channel images using the CFA pattern shown in Figure 2, which is essentially Bayer pattern with a 1.5:1 color filter to pixel size ratio. The peak signal-to-noise ratios (PSNR) of the full 3-channel RGB images recovered from the CFA data using our proposed universal demosaic algorithm are shown in Fig. 5. Comparing our De-mosaic algorithm with Hirakawa’s Bayer[7], our approach is clearly standout in PSNR. A simulated example of de-mosaicking using our algorithm is shown in Fig. 6, which indicates the fine matched from the original lighthouse fence without noticeable color aliasing. Finally, the universal de-mosaic algorithm is applied to the real device as shown in Fig. 7. Raw image was processed after BLS, WB, Universal De-mosaic, & Gamma routines with no CCM applied. The residual 3x3 periodic artifact seen in smooth regions of the image is most likely due to pixel crosstalk, and this artifact is not visible when de-mosaic simulated 1.5x pixel size CFA images. Although the color fidelity cannot match those of 1.1um and larger pixels, however, we believe it presents a promising approach to recover full color when CFA overlay control, and pixel isolation schemes are improved, which is our on-going improving activities. CONCLUSION We’ve presented the sub-um pixel array device characteristics, and color performance resulting from our de-mosaic processing applied to the new CFA patterns. To maximize the color fidelity when processed with our universal demosaic algorithm, it requires optimized detector technologies for improved CFA overlay control, and pixel isolation. The new CFA patterns offer a solution to address the bottleneck of reducing pixel size, while keeping color filter size fixed. The solution is flexible as different ratios between color filter size and pixel size can be chosen to accommodate design requirements. In addition, in certain cases this approach offers the possibility for resolution improvement. REFERENCES [1] H. Taguchi, et al., “Technology of color filter materials for image sensor”, Proc. 2011 IISW, pp. 34-37, June, 2011. [2] S. Goma, K. Atanassov, “Evaluation Methodology for Bayer Demosaic Algorithms in Camera Phones”, Proc. SPIE Vol. 7537, Digital Photography VI, 2010. [3] L. Anzagira and E.R. Fossum, “Color filter array patterns for small-pixel image sensors with substantial cross talk”, J. Opt. Soc. Am. A, Vol. 32(1), pp. 28-34, 2015. [4] H. Honda, et al., “High Sensitivity Color CMOS Image Sensor with WRGB Color Filter Array and Color Separation Process Using Edge Detection,” Proc. 2007 IISW, pp. 263–266, June, 2007. [5] Bouman, Charles, and Ken Sauer. "A generalized Gaussian image model for edge-preserving MAP estimation." Image Processing, IEEE Transactions on2.3 (1993): 296-310. [6] Gray, Robert M. "Toe