In the past 10 years, the image sensor market has grown tremendously, fueled in part by the combination of CMOS image sensor technology and the proliferation of camera phones. More recently, the rate of innovation in image sensors has stabilized and resulted in a relatively standard set of technologies for producing high-quality, low-cost image sensors for high-volume markets. Many in the imaging industry believe that it is now time for the next wave of digital imaging innovation — transitioning from digital cameras that are effectively film cameras with an image sensor located at the film plane or ordinary video cameras that simply record scenes to new devices with capabilities that go beyond the capture-and-view concept.Stacked pixel sensorsUnlike traditional image sensors, which use a single layer of light-sensing pixels and overlaying layers of color filter materials, stacked pixel sensors designed by Foveon Inc. of San Jose, Calif., use a three-dimensional array of pixels, increasing the information density in the captured image. These sensors were investigated previously, although before the ×3 technology, most methods were not commercially or technologically viable. But research continues, and many industry leaders have sought new patents based on these devices. Stacked pixels allow the production of much smaller (and therefore less expensive) sensors for the demanding mobile device market; they eliminate artifact-reducing blur filters and, through on-chip pixel aggregation, provide fully sampled video rather than lower-quality subsampled video. Beyond this, stacked pixel sensors can be used for multispectral imaging because they do not use absorbing color filters; therefore, they “see” the full spectral range of silicon photodetectors.These sensors are manufactured in an all-semiconductor process that renders them robust — free of organic materials that can easily melt or be adversely affected by bright light or heat. These qualities make the sensors ideal for applications requiring them to operate in a difficult environment. Additionally, sensors that do not require an organic filter layer offer optical advantages because they eliminate the thickness of organic filters, permitting design of small pixels with much higher angular acceptance than traditional sensors.High-definition camerasCamera phones have changed the way people think about and use cameras. It is not uncommon to see footage on news stations featuring video clips captured by a camera phone. The images are usually noisy and choppy in appearance. However, what if camera phones contained high-quality high-definition-capable recording devices? Figure 1. Schematic and spectral response plots highlight the difference in operation between conventional color image sensors and stacked pixel image sensors (CFA = color filter array). Handset manufacturers place stringent constraints on the size and cost of a camera module for a mobile phone, but high-resolution still capture and high-definition video are possible within these parameters. By using a size-efficient stacked pixel technology, an 8-megapixel camera module capable of true “3-CCD” broadcast-quality 1920 × 1080 high-definition video could be made in an optical format of only 1/3 in., well within the tolerance of a hand-set maker’s requirements. When portable, affordable high-definition video cameras become ubiquitous, news production and distribution could change dramatically and quickly.In the medical field, pill cameras are already in use; however, if they were made using stacked pixel sensors, they might reach more places in the human body while still producing vivid images. In addition, readying a large and expensive endoscope for an invasive examination is time-consuming and expensive, which makes endoscopy a procedure that insurance companies want to avoid as a standard protocol. Smaller and less expensive endoscopes and pill cameras using stacked pixels could enable more affordable routine screening for many medical conditions. Figure 2. Shown is a cross section of a four-channel stacked pixel design suitable for automotive applications such as night vision. Automobile cutaway ©Cambridge Consultants Ltd. Beyond these applications, there is an abundance of possibilities for other uses. The automotive industry has entered the imaging arena by adding cameras that aid the driver in some automobile models. However, the real “gold” is in the use of cameras for built-in safety systems that avoid collisions and lane departure. As one would expect, advanced features come at a price and initially will be available only in top-of-the-line, high-priced vehicles. However, making the necessary components for such advanced features more compact and affordable will accelerate the adoption in the mass market. Because the stacked pixel sensor relies only on silicon for color detection and because silicon can detect out to the near-IR, the devices can serve multiple purposes. A single camera can be an imaging device for human operation and serve as a night-vision system for object detection or collision avoidance. One could envision a stacked pixel sensor with four or more pixels stacked per location, providing for a very different spectral sensitivity characteristic from that of a conventional digital camera sensor and simultaneously providing IR and color data. More simply, a stacked pixel sensor can offer a high sensitivity night mode simply by operating in a panchromatic mode that combines signals electrically from multiple pixels in the stack. Figure 3. An extended-dynamic-range infrared image was captured with a three-channel stacked pixel sensor. The illumination difference between the MacBeth chart in bright light and that in deep shadow is approximately 16×, and total image dynamic range is roughly 10,000:1. Courtesy of Peter Manca. Surveillance applications are moving in new directions as well. It is no longer of interest simply to record hours of video and then review it for information when a security problem occurs. Modern methods involve taking action automatically in response to certain kinds of camera footage. These responses can include calling the authorities to lock buildings, shutting down computers, re-directing traffic or even activating automated defense systems. Split-second decisions of such magnitude require the best information, and stacked pixels can be used to obtain it. Having fully sampled information in all spectral bands of interest allows the device to operate in a high-quality video mode through simple analog combination of adjacent pixels, which is not possible with sensors that spatially subsample spectral bands and therefore compromise quality when switching between maximum-resolution and reduced-resolution operating modes. The use case is fairly straightforward — the surveillance camera wants to acquire information on a large scene very quickly, so a lower-resolution, higher-speed readout is desirable. Once some activity of interest is detected, the camera can focus on the area of interest. With a stacked pixel sensor, there are minimal quality compromises when switching between these operating modes. Another key feature for the surveillance camera business is dynamic range. With stacked pixels, it is possible to dedicate the usual three pixels in the stack to normal color imaging, while reserving additional channels with various sensitivity characteristics for dynamic range extension both in the infrared and visible — in essence, high-dynamic-range channels can be had with only minor impact on the behavior of the normal color sensor operation. Image of the futureIt is difficult to predict all of the places where we will see advanced digital imaging systems in the coming years. As our ability to build imaging devices improves, the uses for communication can be expected to grow exponentially, and it is quite reasonable to believe that, within the next 10 years, many of the imaging-oriented appliances and devices that have populated science fiction movies in recent history will actually be things we use and interact with every day.Meet the authorRichard M. Turner is vice president of marketing, and Rudolph J. Guttosch is vice president of applications, both at Foveon Inc. in San Jose, Calif.; e-mail: richard.turner@foveon.com, rudy.guttosch@foveon.com.