By 2020, the number of smartphone users in the world is expected to reach almost 2.9 billion — nearly doubling in six years from about 1.6 billion in 2014. Technology companies and researchers have been directly and indirectly imbuing smartphones with additional capabilities, including spectroscopy for biological and medical applications, among other uses (Figure 1). Figure 1. The Changhong H2 is the first consumer device with an integrated imaging spectrometer. Courtesy of Consumer Physics. The result is reminiscent of something from “Star Trek,” the seminal 1960s television series that delivered futuristic inventions and Capt. James T. Kirk’s roguish smile into American homes on a weekly basis. Among the show’s memorable gadgets were the communicator, which inspired the design of the clamshell cellphone, and the tricorder, a portable, Swiss Army knifelike gadget that allowed the crew of the USS Enterprise to diagnose medical conditions in seconds with the flash of a hand-held probe across a patient’s body. Despite decades of attempts, no one device has completely replicated the original tricorder’s capabilities. Portable analysis Progress for consumer-oriented devices is being made. According to Brian Cunningham, professor of electrical and computer engineering at the University of Illinois at Urbana-Champaign, “The ‘Star Trek’ tricorder was absolutely a touchstone as we developed our device — the TRI Analyzer.” His group is one of several across the world exploiting the convergence of smartphone computing and spectroscopy. The TRI Analyzer is a 3D-printed cradle that physically attaches to a smartphone and uses the phone’s internal camera as a visible-light spectrometer (Figure 2). “Cellphones today are very well-engineered,” Cunningham said. “And their hardware and software are very well-integrated.” Figure 2. The TRI Analyzer, inspired by the “Star Trek” tricorder, detects color changes in biological assays. Courtesy of Kenneth Long. This point-of-care device detects the photon emission spectrum of liquids that are used in three common biological diagnostic tests: an enzyme-linked immunosorbent assay for fetal fibronectin protein, which is indicative of preterm birth; the phenylketonuria screening for newborns; and an assay for the soluble transferrin receptor, which is a biomarker of iron deficiency. The results were comparable to those from clinical-grade spectrometers1,2. “We view it as a portable lab,” Cunningham said. “In practice, we could adapt the analyzer to thousands of already developed tests. It would have a lot of benefit where [patients] may not have access to them.” His research group took an intrinsic approach to creating the TRI Analyzer, leveraging the smartphone’s existing components rather than modifying the phone or using an external spectrometer. An optical fiber steers light from the phone’s LED flash into a liquid compartment in a microfluidic chamber. The light shines through a liquid sample or off of an antibody-linked photonic crystal, bounces off of a mirror on the opposite side of the compartment, and then travels through a second optical fiber onto a diffraction gradient that spreads the light spectrum across the internal CMOS active-pixel sensor. An app on the phone analyzes the incoming video to detect the subtle color changes that occur in diagnostic tests. The cradle itself is low-cost and should have widespread appeal. “This approach is just better for the biological assays we have in mind, which generate colored products in the visible light range,” Cunningham said. “We anticipate making these cradles for popular phone models.” Earlier this year, the Austin, Texas-based medical diagnostic company Reliant Immune Diagnostics licensed the TRI Analyzer. According to Reliant CEO Henry Legere, the company is developing a SmartLab Suite of products that uses the analyzer “to run complete metabolic panels, lipid panels, cardiac and respiratory panels, and much more.” Cunningham’s research showcasing the technology placed first at the National Institutes of Health’s 2017 conference on health care innovations and point-of-care technologies held in Bethesda, Md. Democratizing discovery Fuhong Cai of Hainan University in Haikou, China, concurs with Cunningham’s assessment on the state of portable spectroscopic technology. He expects that eventually “the imaging spectrometer will replace the ordinary camera in many applications.” However, the price of current spectrometers is prohibitively high. To address cost and “democratize” access to the technology, Cai, together with Sailing He’s group from Zhejiang University in Hangzhou, China, developed a hand-held, smartphone-connected spectrometer for less than $300 using off-the-shelf components (Figure 3). Figure 3. Fuhong Cai’s hand-held spectrometer connects to a Wi-Fi camera and displays spectral images on a smartphone. Courtesy of Fuhong Cai. Instead of relying on the internal smartphone camera, Cai and colleagues opted for an external approach to marrying these technologies. Their penlike device consists of optical lenses and a diffraction grating, all housed within a metal tube the length of a smartphone. It weighs 140 g, which is about the same as a baseball. The device has a resolution of approximately 17 nm in a range of 400 to 675 nm. The research group used the hand-held spectrometer to image biological samples, including chlorophyll in bananas and myoglobin in pork. They also created — in about 16 s — a 3D spectral image data cube of hemoglobin in the human hand3. Cai’s group is deploying the device in biomedical imaging. They designed a compact multispectral imaging endoscope that consists of an objective, a galvanometer mirror, and the imaging spectrometer; it can connect to a smartphone using Wi-Fi. According to Cai, “This spectral image scanning system has the potential of connecting to a real medical endoscope system to perform imaging, even for point-of-care applications.” Another of their devices can perform push-broom scanning of the human face, which could be used for oxygenation and perfusion monitoring4, 5 (Figure 4). Figure 4. Fuhong Cai’s (Hainan University) spectrometer can be used as a medical endoscope (a), and potentially for oxygenation and perfusion monitoring of the human face (b). Courtesy of Fuhong Cai. Cai said this current work is proof of concept. As the cost of IR cameras decreases, he and his colleagues hope to extend their range. Moving to the IR will “provide more meaningful optical information, which is in accordance with the concept of ‘big data’ in this [scientific] era.” Managing data Data was one worry that the USS Enterprise’s doctor, “Bones” McCoy, didn’t seem to have when he pulled out the tricorder and waved it over a wounded officer. But Cai and others, such as the California- and South Korea-based company Stratio Inc., realize that the information both acquired and generated by the devices is key to long-term viability. Similar to Cai’s spectrometer, Stratio’s LinkSquare is an external penlike device that connects to a smartphone app via Wi-Fi. According to Su Ryeo Oh, chief strategy officer at Stratio, the device operates in the visible and near-infrared (NIR), up to 1000 nm, with an optical resolution of about 5 nm. It’s a sleek instrument that arrives in an iPhone-like case and feels like an oversized fountain pen in one’s hand. The LinkSquare app contains applets that allow users to verify the authenticity of medications6, identify certain types of skin cancer, discern rotten from fresh tuna, and even estimate the percentage of cocoa in various chocolates. The app currently has about 15 applets — a number that is increasing. The device was born from the Stratio founders’ research at Stanford University into IR imaging, and the company is now collaborating on cancer research with scientists there (Figures 5). Figure 5. The LinkSquare device (a) is an easy-to-use, hand-held spectrometer that connects to a smartphone via Wi-Fi. The device can analyze the freshness of meat (b), and the authenticity of medicine (c). Courtesy of Stratio Inc. According to Oh, data integrity is ensured by designing both the hardware and software in-house. “We run machine learning algorithms ourselves, and we always make sure that the applets are above 95 percent accuracy,” she said. “We do not use crowdsourced data.” Both Oh and Cai agree that artificial intelligence (AI) is crucial to leveraging information in the big data era of science. Cai said that He’s group at Zhejiang University is planning to use AI to “make spectral image data smarter,” while Stratio is using its machine learning algorithms to engineer applets for clients if they upload their data to the company’s servers. Stratio has also produced a software development kit that allows developers to build their own apps. In late 2018, they will release an NIR LinkSquare (800 to 1200 nm) and have plans for a shortwave-infrared (SWIR)capable device (800 to 1600 nm) with a germanium detector instead of the more traditional indium gallium arsenide (InGaAs) detector. The additional capabilities should generate even more interesting data, Oh said. “The LinkSquare NIR will provide more sophisticated applications for food freshness.” With Stratio’s proprietary germanium sensor, the LinkSquare SWIR will be able to “monitor blood glucose levels and detect skin veins — the best contributions we can make to the medical world. However, being a medical device and requiring a higher resolution than 128 × 128 [pixels], we expect to offer [more] applications further down the road. We are open to partnership opportunities for these applications.” Integrating the spectrometer Although the processing power of today’s smartphones rivals that of past supercomputers, the integrated cameras have received incremental upgrades. “The performance of the mobile phone’s camera is gradually improving, but it is still not suitable for imaging spectrometers due to its short focal length and small sensor area,” Cai said. It’s one reason why he and companies such as Stratio have developed external smartphone spectrometers. University of Illinois professor Cunningham takes a more agnostic approach, noting that he doesn’t “have a religious belief about either [internal or external spectrometers], because in some projects, we’re taking the extrinsic approach.” Cunningham, whose TRI Analyzer was named after the “Star Trek” tricorder, sees a robust future for smartphone spectroscopy. “The iPhone [X] has four cameras — two back-facing, one selfie camera, and the face ID camera,” he said. “In the future, why couldn’t there be a spectrometer that’s devoted toward medical diagnostics, the color of your couch, [and] an entire range of undreamed applications?” One company working to those ends is Sichuan Changhong Electric Co. (Changhong), a Chinese consumer electronics giant. In January 2017 at the Consumer Electronics Show in Las Vegas, the company unveiled the Changhong H2, which integrates an NIR spectrometer into an Android smartphone. Changhong worked in collaboration with Analog Devices Inc., which designs and manufactures electronic components, and Consumer Physics, developer of the underlying sensor technology. Dror Sharon, CEO of San Francisco-based Consumer Physics, said the phone’s sensor is an iteration of their SCiO hand-held spectrometer, which is similar to Stratio’s LinkSquare device. Figure 6. The back of the Changhong H2 features sensor technology from Consumer Physics. Courtesy of Consumer Physics. “We designed the architecture [of the SCiO] to be very small so that it could be scaled down,” he said, adding that the Changhong H2’s sensor is about 10 × 8 mm and less than 4 mm in height (Figure 6). The challenge was optimizing the whole platform — from optical design to software — to fit in a phone and run spectroscopic applications. “Systemwide, we had to ask questions differently.” In tests with the Changhong H2, Sharon said they “were able to show applications around nutritional content, which is health-related; fraud detection for pharmaceuticals; body-fat analysis; heart-rate analysis; and tissue analysis — and there are some initial indications around skin analysis [Figure 7]. I think that overall in terms of body measurements, we feel that there are enough initial applications to be quite interesting for consumers, and we have [initial proof] of validity in terms of their performance.” The device is currently not available outside of China. Figure 7. The Changhong H2 can be used for several health-related analyses (a) and can detect the freshness and sweetness of fruits such as strawberries (b). Sharon believes in the revolutionary power of smartphone spectroscopy. He said that once the adoption of the spectrometer into smartphones reaches a critical mass and end users understand the full capabilities of the technology, “we will create a world where fraud is almost nonexistent because of this type of device, whether it’s pharmaceuticals [or] construction materials.” “The best technology,” he said, “is the one you don’t think of as technology — it’s just part of your life.” Acknowledgments The author would like to thank Su Oh, Stratio Inc.; Fuhong Cai, Hainan University, Haikou, China; Dror Sharon, Consumer Physics; Brian T. Cunningham, University of Illinois at Urbana-Champaign; and Henry Legere, Reliant Immune Diagnostics. References 1. K.D. Long et al. (2017). Multimode smartphone biosensing: the transmission, reflection, and intensity spectral (TRI)analyzer. Lab Chip, Vol. 17, Issue 19, pp. 3246-3257. 2. R.D. Peterson (2017). Comparison of methods study between a photonic crystal biosensor and certified ELISA to measure biomarkers of iron deficiency in chronic kidney disease patients. Sensors, Vol. 17, Issue 10, p. 2203, doi:10.3390/s17102203. 3. F. Cai et al. (2017). Pencil-like imaging spectrometer for bio-samples sensing. Biomed Opt Express, Vol. 8, Issue 12, pp. 5427-5436, doi:10.1364/BOE.8.005427. 4. F. Cai et al. (2018). The design and implementation of a low-cost multispectral endoscopy through galvo scanning of a fiber bundle. Opt Commun, Vol. 428, pp. 1-6. 5. J. Chen et al. (2018). Experimental demonstration of remote and compact imaging spectrometer based on mobile devices. Sensors, Vol. 18, Issue 7. 6. K.J. Baik et al. (2017). Pharmaceutical tablet classification using a portable spectrometer with combinations of visible and near-infrared spectra. Ninth International Conference on Ubiquitous and Future Networks (ICUFN), pp. 1011-1014, doi:10.1109/ICUFN.2017.7993951.