Camera 2.0: New computing platforms for computational photography

This cell phone... ...took this picture. The Frankencamera F3, our latest experimental camera platform.
Can your point-and-shoot camera do better?
Here are more N95 pictures of Stanford. And here is another N95 album. 		The camera runs Linux, and its metering, focusing, demosaicing, denoising, white balancing, and other processing is programmable. Click here for our SIGGRAPH 2010 paper, which describes our previous camers - the F2.

News flashes:
  • 3/15/12 - The FCam API is now available on the Nokia N9 Linux-based smartphone. See below for details.

    Marc Levoy has retired from Stanford University to lead a team at Google. This project is no longer active, and the Stanford Frankencamera is not available for purchase. However, the FCam API is available on cell phones and tablets that run Android version "L" (Lollipop), as the Camera2 API (Application Programming Interface) and HAL v3 (Hardware Abstraction Layer) for the cameras on these devices. See also this presentation by former Stanford PhD student Eddy Talvala at Google I/O 2014, or this summary by Android Police. Apple offers a similar API in iOS.

    Overview

    Computational photography refers broadly to sensing strategies and algorithmic techniques that enhance or extend the capabilities of digital photography. The output of these techniques is an ordinary photograph, but one that could not have been taken by a traditional camera. Representative techniques include high dynamic range imaging, flash-noflash imaging, coded aperture and coded exposure imaging, photography under structured illumination, multi-perspective and panoramic stitching, digital photomontage, all-focus imaging, and light field imaging. Although interest in computational photography has steadily increased among graphics and vision researchers, progress has been hampered by the lack of a portable, programmable camera platform with enough image quality and computing power to be used for everyday photography. To address this problem, we have been pursuing two subprojects:

    1. Computational Photography on Cell Phones. Over the past five years, the cameras in cell phones have improved dramatically in resolution, optical quality, and photographic functionality. (See 1st and 2nd images above.) Moreover, camera phones offer features that dedicated cameras do not: wireless connectivity, a high-resolution display, 3D graphics, and high-quality audio. Finally and perhaps most importantly, these platforms run real operating systems, which vendors have begun opening to third-party developers. We have therefore begun developing computational photography applications for commerical cell phones. Our Eurographics 2008 paper on viewfinder alignment is an example.

    2. The Stanford Frankencamera. Despite these encouraging trends, there are computational photography experiments that cannot be implemented on today's cell phones. either because the cameras' sensor or optics aren't good enough, the computing resources aren't powerful enough, or the APIs connecting the camera to the computing are too restrictive. We have therefore built a custom camera that runs Linux, is fully programmable (including its digital signal processor) and connected to the Internet, and accommodates SLR lenses and (soon) SLR-quality sensors. Our current prototype (3rd and 4th images above) is constructed from off-the-shelf parts, in some cases borrowed from dead cameras. It's also ugly - hence the name.
    After pursuing these two subprojects independently for two years, we decided that the best approach was to define a single open architecture for programmable cameras. Our architecture consists of a hardware specification, a software stack based on Linux, and an API with bindings for C++. The architecture permits control and synchronization of the sensor and image processing pipeline at the microsecond time scale, as well as the ability to incorporate and synchronize external hardware like lenses and flashes. To demonstrate the viability of this architecture, we have built two reference implementations: a Nokia N900 smartphone with a modified software stack, and the Frankencamera F2 pictured above. Click here for our SIGGRAPH 2010 paper about this architecture.

    Going public

    On July 21, 2010 we released the source code for our implementation of the Frankencamera architecture and FCam API on the Nokia N900 smartphone. A binary executable was also released, which can be downloaded to any retail N900 without bricking the phone, thereby making its camera programmable. To get started turning your N900 into a Frankencamera, click here. FCam is now also available on the Nokia N9 as part of the Meego/Harmattan 1.2 operating system update. The Harmattan documentation is here. The N9 is not sold by carriers in the U.S., but unlocked N9's can be purchased on Amazon, and we believe they work with AT&T and T-Mobile.

    In addition to our implementation on the N900, we plan to make the more powerful custom-built Frankencamera F's available to the research and teaching community. However, the Frankencamera F2 (pictured above) is not worth buying, since its sensor is only cell-phone sized. (Hence, it has a 6x crop factor relative to a 35mm sensor. The only Canon lens that makes sense given this crop factor is a 10-22mm set to 10mm. This configuration is pictured on the web page of our SIGGRAPH 2010 paper.) Instead, we are actively working on the Frankencamera F3, which will have a 24mm x 24mm sensor, hence a 1x crop factor (i.e. full-frame), at least in the vertical direction. We don't have a release date for the F3, but we are shooting for Spring 2012. Our plan is to distribute the camera "at cost" (might be $8K) to anyone who wishes to purchase one.

    Using FCam in university courses

    Are you a university instructor teaching a course in computational photography? If so, then we want to talk to you. We are currently assembling packages to help teach computational photography at the university level. Each package will consist of 20-30 lectures on computational photography written by Marc Levoy (Stanford) and Fredo Durand (MIT), a box containing 10 Nokia N900 smartphones, and one Frankencamera F3. The lecture material will be free, available to anyone and downloadable from the Internet. The N900s will also be free, but as loaners only, and subject to available supply. This loan program is supported by a generous grant from Nokia. The Frankencamera F3s will be sold "at cost", as mentioned earlier.

    As of Summer 2010, this package has been used in two courses - at Stanford University (co-taught by Marc Levoy and Fredo Durand) and at ETH Zurich (taught by Marc Pollefeys). In both courses we loaned Nokia N900s to every student, and assignment #1 was to replace its autofocus algorithm. (The phone's camera, while small, has a movable lens. For more details look at the "Hello Camera" link on the course web site.) We graded the assignment on the accuracy they could focus on a test scene we provided, as well as focusing speed in milliseconds. An assignment like this would have been impossible before FCam. Two of the students submitted algorithms that were better than Nokia's; these were presented in Finland at the conclusion of the course. The students also did interesting projects for the course, some of which might be submitted for publication over the coming year.

    At present this program is designed mainly for universities in North America and Europe. However, we will entertain all reasonable requests. In addition, for universities in the United States, the Frankencamera F3s will be free, supported by a grant from the National Science Foundation. Want to find out more about this program? If you're a U.S. university looking to join the NSF-supported program, send email to Marc Levoy. If you're any other university looking for loaner N900s, contact Timo Ahonen at Nokia.

    Funding sources

    The Camera 2.0 project, which began as a collaboration between the Stanford Computer Graphics Laboratory and the Nokia Research Center Palo Alto Laboratory, has also received funding from Adobe Systems, Kodak, Hewlett-Packard, The Walt Disney Company, Intel, Texas Instruments, Google, NVIDIA, and Sony. The NSF portion of the project is in collaboration with Fredo Durand and William Freeman of MIT.

    People

    Collaborators

    Alumni

    Recent papers in this area:

    Decoupling algorithms from schedules for easy optimization of image processing pipelines
    Jonathan Ragan-Kelley, Andrew Adams, Sylvain Paris, Marc Levoy, Saman Amarasinghe, Fredo Durand
    ACM Transactions on Graphics 31(4) (Proc. SIGGRAPH 2012)
    Experimental Platforms for Computational Photography
    Marc Levoy
    IEEE Computer Graphics and Applications, Vol. 30, No. 5, September/October, 2010, pp. 81-87.
    The Frankencamera: An Experimental Platform for Computational Photography
    Andrew Adams, Eino-Ville (Eddy) Talvala, Sung Hee Park, David E. Jacobs, Boris Ajdin, Natasha Gelfand, Jennifer Dolson, Daniel Vaquero, Jongmin Baek, Marius Tico, Henrik P.A. Lensch, Wojciech Matusik, Kari Pulli, Mark Horowitz, Marc Levoy
    Proc. SIGGRAPH 2010
    Reprinted in CACM, November 2012, with an introductory technical perspective by Richard Szeliski
    Gaussian KD-Trees for Fast High-Dimensional Filtering
    Andrew Adams Natasha Gelfand, Jennifer Dolson, Marc Levoy
    Proc. SIGGRAPH 2009
    Spatially Adaptive Photographic Flash
    Rolf Adelsberger, Remo Ziegler, Marc Levoy, Markus Gross
    Technical Report 612, ETH Zurich, Institute of Visual Computing, December 2008.
    Viewfinder Alignment
    Andrew Adams, Natasha Gelfand, Kari Pulli
    Proc. Eurographics 2008

    Publicity about the project

    We'd love to hear from you, but first check out our list of frequently asked questions.

    Left: Marc Levoy. Right: Andrew Adams with camera.
    Behind them is the Stanford Multi-Camera Array.
    Another picture of the camera, used in the July 2010 Stanford Report article. Images by Linda Cicero.

    Left: The Frankencamera F2 with two Canon flash units attached. Right: A picture taken by this Franken-arrangement. The smaller 430EX flash was programmed using our FCam API to strobe repeatedly, and the larger 580EX was programmed to fire once at the end of the exposure.

    Other links

    © 2007-2009 Marc Levoy
    Last update: October 24, 2014 01:58:22 PM