A Hybrid Camera System for Low-Light Imaging
Feng Li, Yu Ji and Jingyi Yu
ABSTRACT SETUP PIPELINE ALGORITHM RESULTS


Abstract
 Capturing high quality color images under low light conditions is a challenging problem in computer vision. Images captured by commodity cameras are usually underexposed and very noisy. In this paper, we design a hybrid camera system that combines the advantages of high-speed, high resolution, and full-spectrum image sensors to solve this problem. It consists of a pair of high resolution monochrome (HR-M) cameras, one high speed monochrome (HS-M) camera, and one single high resolution color (HR-C) camera. To recover the high quality color images, we develop a novel alternating optimization algorithm to remove noise from the HS-M sensor, and then estimate the motion information from the denoised image sequences for non-blind deconvolution of the HR-C camera. We first conduct patch matching between the HR-M image pairs and the HS-M images, and then use these patch priors and a ℓ1 total variation term to regularize the objective function for image denoising. Our experiments on both synthetic and real images demonstrate that our system is robust and reliable.

Hybrid Camera System Setup
Our goal is to combine the benefits of different types of cameras (with respect to aperture, shutter, resolution, and spectrum) to construct a low-light imaging system. Figure shows our proposed hybrid camera system: it consists of one Pointgrey Grasshopper high speed monochrome (HS-M) cameras (top-left), one Flea2 high resolution color (HR-C) camera (top-right), and two Pointgrey Flea2 high resolution monochrome (HRM) cameras (bottom). All cameras are equipped with the same Rainbow 16mm Cmount F1.4 lens. We mount the four cameras on a T-slotted aluminum grid, which is then mounted on two conventional tripods for indoor applications. To deal with the long working range of outdoor applications, we also build a giant “tripod” from a 6 foot step ladder to hold the camera array grid, as shown in Figure(right).
Pipeline
Our system pipeline. We first preprocess the HS-M images to improve the brightness of dark regions, and then apply BM3D denoising algorithm to partially remove sensor noise. Next we find low-noise patch priors from HR-M images for full-spectrum denoising of HS-M and HRC images. Finally, we design an alternating minimization algorithm to denoise the local contrast enhanced HS-M images and reconstruct high quality color images from the HR-C inputs.
Hybrid Denoising
we model image noise as white, zero-mean Gaussian noise, with a standard deviation σ, and assume the noisy image I can be decomposed into a latent image and noise η: 
     (1)
where x is the pixel index.
Multi-view Block Matching
we present a multi-view block matching (MVBM) technique for patch-based image denoising. The goal of MVBM is to find similar patches from HR-M image pairs for each patch in HS-M (or HR-C) images. 
Patch-Based Denoising
Based on the image noise formulation Eq.(1), we design a new full-spectrum spatial regularization that uses patches from the HR-M images to add back the high frequency details for HS-M images. We formulate the full-spectrum denoising as:
     (2)
where  and are two positive weights used to control the strength of TV and spatial prior regularization respectively; || · ||1 represents the ℓ1-norm. is obtained from our MVBM method. We choose ℓ2-norm for the data fidelity term since noise is assumed to be additive and Gaussian. We also add a Total Variation(TV) regularization term to the data fidelity term to make the reconstruction process more reliable.
Iterative Optimization
Compared with Tikhonov-like regularization, the minimization problem formulated by Eq.(2) is computationally expensive to solve due to nonlinearity and non-differentiability of the regularization terms. we present an effective solution: we introduce two auxiliary vectors u, v ∈ to the objective function in Eq.(2) and then reformulate this denoising problem equivalently as an optimization problem with linear constraints
     (3)
To solve for the Eq.(3), we apply the alternating minimization method to the augmented Lagrangian function of Eq.(3) using the following iterative scheme:
      (4a)

(4b)

(4c)

(4d)

(4e)
where is the augmented Lagrangian function of Eq.(3) defined by
  (5)
where and are two vectors of Lagrange multipliers and , ∈ , and are penalty parameters and γ is the step length for updating and . By iteratively updating the Lagrange multipliers and , the solution to Eq.(4a)-(4e) will eventually converge to the one to Eq.(3).
 
 
Results
Synthetic Scene Denoising Results
Input Image BM3D Result Our Result
 
Synthetic Scene Debluring Results
 
Indoor Scene Denoising Results
Input Image High-Resolution Left High-Resolution Right
Pre-Processed Image BM3D Result Our Result
 Indoor Scene Debluring results
 
Outdoor Scene Denoising Results
Input Image BM3D Result Our Result
 
Video 1: Indoor Scene Denoising
Left: Pre-Processed Input Right: Our Denoised Result
 
Video 2: Outdoor Scene Denoising
Left: Pre-Processed Input Right: Our Denoised Result

 
Acknowledgments
This project is supported by the Air Force Office of Scientific Research (AFOSR) Young Investigator Program (YIP). We would like to thank Dr. Sjogren, our Program Director, for his invaluable guidance on this project.