A technology that estimates heart rate using a video camera and software could lead to new applications in the medical equipment and consumer markets. But first, the technology will be used to help new parents rest easier by remotely monitoring the heart and breathing rates of their sleeping babies. The technology processes color data recorded by the camera — the green channel provides the data needed for heart rate estimation — and computes an average heart rate over the regions of the image where skin is visible on the face, neck and arms. USU alumnus Nate Ruben (left) and Dr. Jake Gunther are the inventors of a USU-patented technology that estimates heart rate using a video camera and specialized software. Courtesy of Matt Jensen/USU. “Hemoglobin in the blood has an absorption peak for green light. When the heart pushes blood into arteries near the skin, more green light is absorbed and less is reflected. This means we see fewer green values in the images from the camera,” said Jake Gunther, a co-inventor of the technology and Utah State University (USU) professor. Nate Ruben, co-inventor and CEO of Photorithm Inc., compared the operation of the remote monitoring system to a pulse oximeter. “When using a camera to measure heart rate, we are exploiting one particular phenomena. It is the same observation that a standard pulse oximeter makes, just in a different way,” he told Photonics Media. “A pulse oximeter will emit light through your finger (or other tissue) and measure the light received on the other side. As the heart pumps, the many vessels in the tissue will expand with each pulse and will thus absorb varying amounts of light which, when measured over time, correspond with the heart beating," Ruben said. "This is a transmission-based method for obtaining heart rate.” Ruben told Photonics Media that a similar effect can be observed by measuring the light reflected from the skin. He said that the light that is manifested in the pigment of the skin actually changes slightly with each pulse, but the change is so subtle that our eyes cannot see it. Smartbeat monitors a baby’s breathing rate as she sleeps. Courtesy of Nate Ruben/Photorithm Inc. “Our algorithms focus on using an ordinary camera to capture all of the light reflected by the person’s skin (such as the pixels on the face) and combining them intelligently to uncover the signal of interest. We have spent a lot of time learning how to separate the heart beat signal from the ‘noise,’ which can also cause a person’s face color to change — for example, if they move their head or if another object reflects light on the person’s face. “Instead of looking at the light transmitted through skin tissue, we’re looking at the light being reflected from it,” he said. Ruben said that he and Gunther use a different method when it comes to measuring breathing with a camera. He noted that in contrast to heart rate, breathing is usually visible to the naked eye, even in its most subtle form. “If someone was shown a video of a sleeping swaddled infant, they might not discern any signs of breathing. This is because our eyes are poorly designed to see slow and subtle motion. However, if that same video was played at seven times the speed, the breathing would become obvious,” he said. The software can be used to evaluate frame differences over time and to combine those differences in a constructive way. This data can then be used to produce a real-time breathing signal. “This is similar to other video motion magnification techniques, only ours has been adapted specifically for breathing and is capable of running real-time on embedded hardware,” Ruben said. The system for monitoring breathing in a sleeping infant, called Smartbeat, is currently being used in a video baby monitor to watch for signs of Sudden Infant Death Syndrome (SIDS). Ruben told Photonics Media that the algorithm’s accuracy is startling. Smartbeat monitor. Courtesy of Nate Ruben/Photorithm Inc The idea for the heart rate estimation technology came when Ruben and his wife had their first child in 2012. Like most new mothers, Ruben’s wife constantly checked on the child as he slept. All those trips to the crib inspired Ruben to make a better baby monitor. At the time, he was experimenting with a similar concept that used webcams to estimate heartbeat. But the technology faced a hurdle — sleeping infants don’t hold still, and the baby’s motion caused the camera to capture competing signals. The inventors overcame this obstacle and developed a way to extract only the signals needed. The researchers have started a new company, Photorithm Inc., based on their technology. In the future, the technology could be expanded to include applications such as tools for monitoring blood pressure and blood oxygen levels. For now, it’s helping to keep young children safe throughout the night. The technology has been patented by USU. The Smartbeat baby monitor is able to watch baby's breathing and alert parents if there is a problem. Courtesy of Nate Ruben/Photorithm Inc.