Multi-perspective Imaging, Reconstruction, and Rendering
CVPR 2010 Tutorial

Sunday, June 13, 2010 PM

Organizers:      Jingyi Yu,      Srikumar Ramalingam

Short Bio

  Jingyi Yu is an Assistant Professor at the Computer and Information Science Department at the University of Delaware. He received his B.S. from Caltech in 2000 and M.S. and Ph.D. degree in EECS from MIT in 2005. His research interests span a range of topics in computer vision, computer graphics and computational photography, including multi-perspective imaging, non-conventional optics and camera design, tracking and surveillance, and graphics hardware. He has received an NSF Career Award in 2009 and an Air Force Young Investigator Award in 2010.

  Srikumar Ramalingam is a Research Scientist at Mitsubishi Electric Research Lab (MERL). He received his M.S from University of California, Santa Cruz and his Ph.D. (funded by Marie Curie Fellowship from European Union) from INRIA Rhone Alpes in 2006. His research interests include multi-view geometry for non-conventional camera models and discrete optimization algorithms (particularly graph cuts). His doctoral thesis received INPG best thesis prize and AFRIF thesis prize (honorable mention) from the French Association for Pattern Recognition.

Course Descriptions

A perspective image represents the spatial relationships of objects in a scene as they appear from a single viewpoint. In contrast, a multi-perspective image combines what is seen from several viewpoints into a single image. Despite their incongruity of views, effective multi-perspective images are able to preserve spatial coherence and can depict, within a single context, details of a scene that are simultaneously inaccessible from a single view, yet easily interpretable by a viewer. In computer vision, multi-perspective images have been used for analyzing structure revealed via motion and generating panoramic images with a wide field-of-view using mirrors.

In this tutorial, we provide a practical guide on topics in multi-perspective modeling and reconstruction methods and multi-perspective imaging systems. We start with a brief review of multi-perspective image techniques frequently employed by artists such as the visual paradoxes of Escher, the Cubism of Picasso and Braque, and multi-perspective panoramas in cel animations. We then characterize existing multi-perspective camera models, with an emphasis on their underlying geometry and image properties. Furthermore, we demonstrate how to use these camera models for creating specific multi-perspective rendering effects.

In the context of multi-view geometry in computer vision, there has been considerable research in generic imaging model which treats every camera as a mapping between pixels and their corresponding (half-)projection rays.  For such a non-parametric model, practical calibration and multi-view geometry algorithms have been developed.  Several existing multi-perspective imaging systems such as Catadioptric cameras, Light field cameras and other systems designed using computational photography methods can be addressed using generic algorithms. This tutorial will show practical (self-)calibration and explain various multi-view geometry concepts like Epipolar Geometry, Essential matrix, Fundamental matrix, Pose estimation, Motion estimation and 3D reconstruction algorithms for several of these multi-perspective imaging systems.

The participants learn about topics in multi-perspective modeling and imaging for extracting 3D geometry for computer vision and in rendering for generating compelling pictures for computer graphics. We hope to provide enough fundamentals to satisfy the technical specialist without intimidating curious graduate students interested in multi-perspective images.

Technical Content

A. Introduction to Multi-perspective Cameras: from Renaissance artists to real world problems

B. Multi-perspective Modeling Methods: Ray Geometry Analysis

C. Multi-perspective Images

     1. Image-based rendering methods

     2. Ray-tracing and GPU techniques

     3. Multi-perspective distortions analysis

     4. Shape-from-distortions for specular surface reconstruction

D. Multi-perspective Imaging Systems

     1. Light field cameras

     2. Catadioptric cameras

     3. New computational photography techniques that can simulate multi-perspective cameras.

E. Calibration of Multi-perspective Imaging Systems

     1. A brief introduction of generic imaging model

     2. Practical calibration of multi-perspective imaging systems

          a. Using calibration grids

          b. Without calibration grids

     3. Minimal solutions for generic calibration algorithms

 F. Multi-view geometry algorithms for multi-perspective imaging systems

     1. Epipolar Geometry

     2. Essential and Fundamental Matrices

     3. Pose and Motion Estimation algorithms

     4. 3D Reconstruction algorithms

G. Conclusions and Future work:

     1. Theoretical directions

     2.  Multi-perspective Displays

     3.  A list of unanswered questions

     4. Rendering hardware

     5. Sharing of code and images


Relationship to Previous Short Courses/Tutorials

Jingyi Yu has given a closely related tutorial on Multi-perspective modeling, rendering and imaging at SIGGRAPH Asia 2008, with an emphasis on the graphics and computational photography aspects of the multi-perspective modeling problem.  Srikumar Ramalingam has given a tutorial on Generic Imaging: Calibration, 3D Reconstruction and Applications at ECCV 2006, focusing on the calibration and reconstruction aspects of non-conventional imaging systems.

This new CVPR tutorial will tightly integrate the vision and the graphics components of the previous two tutorials and provide a complete overview of the state-of-the-art in the area. It aims to stimulate new ideas in both computer vision and graphics, ranging from new omni-directional camera designs to novel image-based modeling and rendering techniques.

Relevant Publications

1.   State of the Art Report: Multi-perspective Rendering, Modeling, and Imaging, Jingyi Yu, Leonard McMillan and Peter Sturm, in Proceedings of Eurographics, 2008. A journal version will be published in Computer Graphics Forum, 2009.


2.   Multi-perspective Stereo Matching and Volumetric Reconstruction, Yuanyuan Ding, Jingyi Yu and Peter Sturm, to appear in Proceedings of the Twelfth International Conference on Computer Vision (ICCV), 2009.


3.   General Linear Cameras, Jingyi Yu and Leonard McMillan, in the 8th European Conference on Computer Vision (ECCV), 2004, Volume 2: 14-27.


4.   Towards Complete Generic Camera Calibration, Srikumar Ramalingam, Peter Sturm and Suresh Lodha, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2005.


5.   A generic structure-from-motion framework, Srikumar Ramalingam, Suresh Lodha and Peter Sturm, Computer Vision and Image Understanding (CVIU), Volume 103, Number 3, page 218-228, 2006.