Multispectral machine vision systems are one way to speed up quality control on production lines – and to make sure our food is safe for consumption. Foodborne diseases affect approximately 48 million people in the US each year, according to the Centers for Disease Control. The Food Safety Modernization Act, signed into law in early 2011, was written to strengthen the food safety system by boosting the US Food and Drug Administration’s ability to be more proactive in managing the nation’s food production industry. Improving online food inspection techniques will go a long way toward striking down foodborne illness, and improved machine vision systems will make this possible. Take poultry processing: The industry today is experiencing real growth and higher demand, but current regulations limit the throughput of human inspectors to a maximum of 35 birds per minute. Obviously, there is a great need to reduce costs, remain competitive and increase throughput. As technological improvements are considered, the US Department of Agriculture (USDA) has published Standard Operating Procedures for Notification and Protocol Submission of New Technologies as a guide for development and implementation; machine imaging systems can provide improved productivity within this framework. At the Food Safety Laboratory in the Agricultural Research Service’s arm of the USDA, researchers work to improve technologies and methodologies for online machine inspection of food processing facilities. Their underlying goal is to provide data on how to improve yields using food inspection systems, and how to best identify aesthetic and contamination defects. Research topics include (1) improving techniques for current-generation technology, (2) identifying next-generation technology and methods and (3) implementation of these new techniques. Each generation of imaging technology has provided more data and insights about how to identify defects, but it has also become obvious that limitations exist in the technology as it has evolved from monochromatic to color and from multispectral imaging to hyperspectral. In the 1980s, as CCD image sensors became commonplace, monochromatic imaging was used to sort and/or grade agricultural products. The image could be created by collecting a signal from the entire visible light spectrum or by using a filter to limit exposure of the image sensor to a selected region of interest (ROI) within the spectrum. In addition, because image sensors typically have a larger range of sensitivity than the human eye, ultraviolet and infrared wavelengths also can be used. Depending on the product, an ROI could be designed to yield the best information for inspection. Examples of monochromatic imaging include applications such as identifying bruises and other blemishes on apples, and assessing the amount of fat marbling in beef rib eyes. The major challenge that has emerged – and has continued until now – is producing quality images that clearly show the features that can be used to identify bruises or areas of contamination. Monochromatic imaging systems were limited with respect to the types of flaws and defects they could reveal. Inspection in full color Food inspection evolved in the 1990s with the use of color imaging. Bruises and blemishes on apples were better distinguished by a difference in color than by a difference in gray scale. Crops such as coffee, apples, nuts and seeds could be graded based on color criteria. Inspection methods also advanced in the classification of wholesome and unwholesome poultry. Color imaging and its close relative multispectral imaging are the result of combining the signals from multiple monochromatic sensors. Filters are used to divide the visible spectrum into parts, most commonly red, green and blue. However, these may not be the best wavelengths to use for evaluating a particular food product. Multispectral imaging is similar to color imaging in that it uses multiple selected regions of the electromagnetic spectrum, but it is not limited to using red, green and blue. Any combination of ROIs, including the UV and IR, can be used. To create an image, light must pass through various filters to multiple image sensors or to subsections of a single image sensor, and these signals are then recombined to produce an image. When employing multiple image sensors, dichroic filters are used to select the signal received by each sensor. In the case of a single image sensor, an array of filters is used, typically in a Bayer pattern (see Figure 1). This is very common in most digital cameras and color displays, and can be seen by looking closely at a color monitor or television. Depending upon sensing requirements and size restrictions, either approach can be used to create color images. Figure 1. Bayer pattern on a sensor. Images courtesy of Deposition Sciences Inc. Little spectral information Multispectral imaging provides great spatial information from the selected wavelengths, but it offers little in the way of spectroscopic information. Hyperspectral imaging for food production was developed to provide spectral and spatial data. The spectral data reveals which wavelengths should be used for inspection; as few as two and as many as six ROIs may be required to detect chemicals present in food, such as chlorophyll, hemoglobin and anthocyanin (see Figure 2), or to find the type of defect to be measured. But ROIs identified for one product are likely to differ from those for another. For example, apples will use different ROIs than spinach, and both will use different ROIs than chicken. Fruits, vegetables and meat products can all be inspected using hyperspectral methods. Figure 2. Absorption spectra for anthocyanin, chlorophyll a and hemoglobin. But there is a limitation. Hyperspectral imaging is expensive in terms of the amount of computer processing required for full-speed online inspection. The detectors collect more information than is necessary for inspection of a single item, making them best suited for identifying the regions of interest of each food product. Although multispectral imaging is the best candidate for full-speed online food inspection, hyperspectral imaging is necessary to identify the spectral ROIs. Recent research at the Food Safety Laboratory demonstrated that machine inspection using multispectral imaging is successful in accurately identifying wholesome and unwholesome poultry at a line speed of 140 birds per minute. This is a major improvement compared with one human inspecting a maximum of 35 birds per minute. Having identified multispectral imaging as the method of choice for full-speed online inspection, one can divide the light signal a number of ways. A single image sensor can be used with multiple filters, either in a filter wheel or as a patterned array. Filter wheels generally are less expensive, but they are slower, and no imaging can occur while the wheel is changing position (i.e., a different filter is moving into place). To avoid this time penalty, the light signal can be manipulated after it enters the camera. A system of dichroic beamsplitters can be used to send the signal from each ROI to each separate image sensor. Alternatively, a patterned filter array through which the signal passes into a single image sensor also can be used. Note that this discussion has centered on imaging techniques, with no hardware specification. Although the techniques have evolved as digital technology has matured, digital technology is not necessarily a requirement of multispectral imaging. These techniques can be used with any imaging hardware, provided the sensor is responsive to the wavelengths of interest: CCD or CMOS in the UV, visible and IR, or lead sulfide and indium antimonide sensors in the IR. Figure 3. Wafer with four patterned optical coatings. Many industries that use multispectral imaging, including food inspection, are interested in reducing the weight of their optical systems. This can be achieved by eliminating the beamsplitters and multiple image sensors. Instead, using a patterned filter array, or even applying the patterned array directly onto the image sensor, can decrease the system’s weight. Due to the small (approximately 10 µm) pixel sizes required for these filters, photolithography may be the best method for patterning them. Other methods, such as conventional masking or dicing and bonding individual filter elements, have larger limitations of feature size, on the order of 20 to 50 µm. The semiconductor industry has used photolithography with great success to reduce the achievable feature size on chips. Methods have been developed for etching thin layers of the materials used in chip fabrication. However, these processes are not very effective in removing the thick multilayer optical coatings used to make these patterned filter arrays. Figure 4. Representative spectral scans of each of the four filters on the wafer shown in Figure 3. A technique called resist liftoff was developed by Santa Rosa, Calif.-based Deposition Sciences Inc. to address this need. As opposed to the submicron-thick layers of metals or oxides used in the semiconductor industry, dielectric optical filters typically are 1 to 20 µm thick, depending upon the wavelength of the ROI. Using the resist liftoff process, filters can be created with features as small as 5 µm; as many as four filters have been patterned onto a single substrate. Figure 3 shows an example of a representative patterned filter wafer, and Figure 4 plots the transmission properties of the four filters patterned on the wafer. The wafer has 232 four-color chips and 52 large, single-color pads that are produced using the resist liftoff process. Patterning can be used to divide image sensors into subregions that will image within a single ROI; whole-sensor imaging can be achieved by using a filter array similar to the Bayer pattern used for color imaging (Figure 5). Figure 5. The sputter coating chamber at Deposition Sciences. A technique called resist liftoff enables whole-sensor imaging for food safety inspection by using a filter array similar to the Bayer pattern used for color imaging. Machine imaging is an important quality control instrument, and the continued development of improved techniques and hardware – including new filter and sensor technologies – will allow food quality to be tested beyond simple aesthetic assessment at full production speeds. Meet the author Ryan S. Thompson is a process engineer at Deposition Sciences Inc.; e-mail: solutions@depsci.com.