Ryan S. Thompson, Deposition Sciences Inc.
Multispectral machine vision systems are one way to speed up quality control on production lines – and to make sure our food is safe for consumption.
Foodborne diseases affect approximately 48 million people in the US each year, according to
the Centers for Disease Control. The Food Safety Modernization Act, signed into
law in early 2011, was written to strengthen the food safety system by boosting
the US Food and Drug Administration’s ability to be more proactive in managing
the nation’s food production industry.
Improving online food inspection techniques will go a long way
toward striking down foodborne illness, and improved machine vision systems will
make this possible.
Take poultry processing: The industry today is experiencing real
growth and higher demand, but current regulations limit the throughput of human
inspectors to a maximum of 35 birds per minute. Obviously, there is a great need
to reduce costs, remain competitive and increase throughput. As technological improvements
are considered, the US Department of Agriculture (USDA) has published Standard Operating
Procedures for Notification and Protocol Submission of New Technologies as a guide
for development and implementation; machine imaging systems can provide improved
productivity within this framework.
At the Food Safety Laboratory in the Agricultural Research Service’s
arm of the USDA, researchers work to improve technologies and methodologies for
online machine inspection of food processing facilities. Their underlying goal is
to provide data on how to improve yields using food inspection systems, and how
to best identify aesthetic and contamination defects. Research topics include (1)
improving techniques for current-generation technology, (2) identifying next-generation
technology and methods and (3) implementation of these new techniques. Each generation
of imaging technology has provided more data and insights about how to identify
defects, but it has also become obvious that limitations exist in the technology
as it has evolved from monochromatic to color and from multispectral imaging to
hyperspectral.
In the 1980s, as CCD image sensors became commonplace, monochromatic
imaging was used to sort and/or grade agricultural products. The image could be
created by collecting a signal from the entire visible light spectrum or by using
a filter to limit exposure of the image sensor to a selected region of interest
(ROI) within the spectrum. In addition, because image sensors typically have a larger
range of sensitivity than the human eye, ultraviolet and infrared wavelengths also
can be used. Depending on the product, an ROI could be designed to yield the best
information for inspection. Examples of monochromatic imaging include applications
such as identifying bruises and other blemishes on apples, and assessing the amount
of fat marbling in beef rib eyes.
The major challenge that has emerged – and has continued
until now – is producing quality images that clearly show the features that
can be used to identify bruises or areas of contamination. Monochromatic imaging
systems were limited with respect to the types of flaws and defects they could reveal.
Inspection in full color
Food inspection evolved in the 1990s with the use of color imaging.
Bruises and blemishes on apples were better distinguished by a difference in color
than by a difference in gray scale. Crops such as coffee, apples, nuts and seeds
could be graded based on color criteria. Inspection methods also advanced in the
classification of wholesome and unwholesome poultry.
Color imaging and its close relative multispectral imaging are
the result of combining the signals from multiple monochromatic sensors. Filters
are used to divide the visible spectrum into parts, most commonly red, green and
blue. However, these may not be the best wavelengths to use for evaluating a particular
food product. Multispectral imaging is similar to color imaging in that it uses
multiple selected regions of the electromagnetic spectrum, but it is not limited
to using red, green and blue. Any combination of ROIs, including the UV and IR,
can be used.
To create an image, light must pass through various filters to
multiple image sensors or to subsections of a single image sensor, and these signals
are then recombined to produce an image. When employing multiple image sensors,
dichroic filters are used to select the signal received by each sensor. In the case
of a single image sensor, an array of filters is used, typically in a Bayer pattern
(see Figure 1). This is very common in most digital cameras and color displays,
and can be seen by looking closely at a color monitor or television. Depending upon
sensing requirements and size restrictions, either approach can be used to create
color images.
Figure 1. Bayer pattern on a sensor. Images courtesy of Deposition Sciences Inc.
Little spectral information
Multispectral imaging provides great spatial information from
the selected wavelengths, but it offers little in the way of spectroscopic information.
Hyperspectral imaging for food production was developed to provide spectral and
spatial data. The spectral data reveals which wavelengths should be used for inspection;
as few as two and as many as six ROIs may be required to detect chemicals present
in food, such as chlorophyll, hemoglobin and anthocyanin (see Figure 2), or to find
the type of defect to be measured. But ROIs identified for one product are likely
to differ from those for another. For example, apples will use different ROIs than
spinach, and both will use different ROIs than chicken. Fruits, vegetables and meat
products can all be inspected using hyperspectral methods.
Figure 2. Absorption spectra for anthocyanin, chlorophyll a and hemoglobin.
But there is a limitation. Hyperspectral imaging is expensive
in terms of the amount of computer processing required for full-speed online inspection.
The detectors collect more information than is necessary for inspection of a single
item, making them best suited for identifying the regions of interest of each food
product. Although multispectral imaging is the best candidate for full-speed online
food inspection, hyperspectral imaging is necessary to identify the spectral ROIs.
Recent research at the Food Safety Laboratory demonstrated that
machine inspection using multispectral imaging is successful in accurately identifying
wholesome and unwholesome poultry at a line speed of 140 birds per minute. This is
a major improvement compared with one human inspecting a maximum of 35 birds per
minute.
Having identified multispectral imaging as the method of choice
for full-speed online inspection, one can divide the light signal a number of ways.
A single image sensor can be used with multiple filters, either in a filter wheel
or as a patterned array. Filter wheels generally are less expensive, but they are
slower, and no imaging can occur while the wheel is changing position (i.e., a different
filter is moving into place). To avoid this time penalty, the light signal can be
manipulated after it enters the camera. A system of dichroic beamsplitters can be
used to send the signal from each ROI to each separate image sensor. Alternatively,
a patterned filter array through which the signal passes into a single image sensor
also can be used.
Note that this discussion has centered on imaging techniques,
with no hardware specification. Although the techniques have evolved as digital
technology has matured, digital technology is not necessarily a requirement of multispectral
imaging. These techniques can be used with any imaging hardware, provided the sensor
is responsive to the wavelengths of interest: CCD or CMOS in the UV, visible and
IR, or lead sulfide and indium antimonide sensors in the IR.
Figure 3. Wafer with four patterned optical coatings.
Many industries that use multispectral imaging, including food
inspection, are interested in reducing the weight of their optical systems. This
can be achieved by eliminating the beamsplitters and multiple image sensors. Instead,
using a patterned filter array, or even applying the patterned array directly onto
the image sensor, can decrease the system’s weight. Due to the small (approximately
10 µm) pixel sizes required for these filters, photolithography may be the best
method for patterning them. Other methods, such as conventional masking or dicing
and bonding individual filter elements, have larger limitations of feature size,
on the order of 20 to 50 µm.
The semiconductor industry has used photolithography with great
success to reduce the achievable feature size on chips. Methods have been developed
for etching thin layers of the materials used in chip fabrication. However, these
processes are not very effective in removing the thick multilayer optical coatings
used to make these patterned filter arrays.
Figure 4. Representative spectral scans of each of the four filters on the wafer shown in Figure 3.
A technique called resist liftoff was developed by Santa Rosa,
Calif.-based Deposition Sciences Inc. to address this need. As opposed to the submicron-thick
layers of metals or oxides used in the semiconductor industry, dielectric optical
filters typically are 1 to 20 µm thick, depending upon the wavelength of the ROI.
Using the resist liftoff process, filters can be created with features as small
as 5 µm; as many as four filters have been patterned onto a single substrate. Figure
3 shows an example of a representative patterned filter wafer, and Figure 4 plots
the transmission properties of the four filters patterned on the wafer. The wafer
has 232 four-color chips and 52 large, single-color pads that are produced using
the resist liftoff process. Patterning can be used to divide image sensors into
subregions that will image within a single ROI; whole-sensor imaging can be achieved
by using a filter array similar to the Bayer pattern used for color imaging (Figure
5).
Figure 5. The sputter coating chamber at Deposition Sciences. A technique
called resist liftoff enables whole-sensor imaging for food safety inspection by
using a filter array similar to the Bayer pattern used for color imaging.
Machine imaging is an important quality control instrument, and
the continued development of improved techniques and hardware – including
new filter and sensor technologies – will allow food quality to be tested
beyond simple aesthetic assessment at full production speeds.
Meet the author
Ryan S. Thompson is a process engineer at Deposition Sciences Inc.; e-mail: [email protected].