While human bodies and single cells are three-dimensional, imaging of them often was not. Data might be captured in two-dimensional slices during a computed tomography, magnetic resonance, ultrasound or even microscope study. But the interpretation of how those 2D images related to real-world medical and life science structures sometimes required specialized expertise. Now, advances in display-related technology and software are changing that, with benefits to medicine and the biosciences ranging from education to operation planning. Educational lessons created with cinematic rendering of a routine CT scan of the head show the anatomy of the intracranial arteries. Courtesy of Franz Fellner/Kepler University Hospital. Some of those advances borrow from fields far removed from the life sciences. That was the case for Franz Fellner, a professor of radiology and head of the Central Radiology Institute at Kepler University Hospital in Linz, Austria. He has been putting to use 3D rendering based on techniques similar to those found in movies. “I use cinematic rendering to create anatomical figures for my conventional educational lessons using PowerPoint files,” he said. Fellner does this working from CT and MRI data sets, turning these into 3D anatomy lessons, both for medical students and the public. Cinematic rendering of an MRI data set of the brain enables viewing of 3D data by students and the public. Courtesy of Franz Fellner/Kepler University Hospital. This is done in a high-resolution 8K projection room that offers dual 16 × 9-m viewing surfaces on the walls and floor. It allows stereoscopic 3D images, transforming image slices into an assembled lifelike whole. The image processing technology behind this has been developed over the past few years by what is now Siemens Healthineers, a separately managed health care business of Siemens AG. The company provides CT, MRI, PET and x-ray devices, with every method capable of 3D imaging. “In recent years, the amount of data generated by these scanners is significantly increasing,” said Andre Steinbuss, head of global product marketing for Syngo (Siemens Healthineers’ radiology imaging post-processing business line). The growing flood of data generated by the various 3D imaging techniques presented problems, particularly when radiologists had to communicate with outside experts such as surgeons and others. Traditional rendering techniques that transformed 2D grayscale data into 3D presentations did not yield high-quality images and these were frequently not accepted by outside experts, Steinbuss said. To overcome this drawback, the company created software based on movie techniques to create photorealistic images. The input data can come from any source that adheres to the DICOM (Digital Imaging and Communications in Medicine) standard for medical imaging information. Thus, it could include data captured with a CT or MRI scanner. Imaging software allows researchers and developers to access cinematic rendering, enabling the realistic depiction of volume datasets such as this one of blood vessels and large aneurysm in the brain. Courtesy of Radiologie im Israelitischen Krankenhaus. At present, the 3D rendering software system can be used for teaching and presentations, but not for diagnostic purposes. Regulators require that providers of a new diagnostic tool or method supply proof that the new approach is better than the old. Siemens Healthineers is looking for a clinical benefit, for example, in the visualization of soft tissue and muscles, but does not yet have evidence of it. Few locales will have the massive display found in Austria’s Kepler University Hospital. Many settings, in fact, may have diagnostic monitors that can only display 2D images. In that case, the 3D data would be presented as a two-dimensional, high-quality, high-resolution image, Steinbuss said. As this demonstrates, 3D imaging ultimately must be viewed on the available display. In response to the need for 3D presentation methods, there are efforts underway to bring new display techniques into production. One comes from Holografika Kft. CEO Tibor Balogh said that the goal is to create a display in which the screen cannot be distinguished from the same image viewed through a window. While not yet able to achieve that ideal, the company’s HoloVizio 3D light field displays average 100 million pixels. They can offer full-angle 180-degree viewing, ranging from desktop monitors up to a glasses-free 3D movie system. Today, these 3D displays are secondary to 2D ones, which offer higher monochrome resolution. Even with that, though, 3D displays can provide better subjective resolution, particularly when it comes to areas such as surgical planning. There, a nonradiology expert will be trying to figure out the best approach to take to reach a problem spot and then remove diseased tissue. Properly understanding the 3D image may benefit greatly from the right rendering. “You can better see where features are in relation to each other,” Balogh said, of putting the data on a 3D display. “Greater uptake of contrast material may make a structure in the back appear brighter and with more contrast than one that is actually in the front.” LightSpace Technologies SIA is another company working on 3D display technologies. Ilmars Osmanis, CEO, said that traditional 3D image displays, which often make use of glasses, suffer from problems in the collaborative environment. He added that with short viewing distances — under two meters — there also can be eye fatigue and discomfort. Osmanis said that tests of volumetric, holographic or light field displays, such as those offered by his company, have been conducted for real-time angiography and for molecular-level 3D scans and simulations. Other tests are planned for 3D endoscopy using 3D depth mapping cameras and image display, as well as image-guided robotic surgery. In general, there is a trend toward greater use of 3D imaging and data in medicine and the life sciences, Osmanis said. “The presence is certainly increasing. However, there is initial confusion by the introduction of stereoscopic 3D systems that are not comfortable for the eyes.” A different type of 3D imaging and display can be found in smart glasses, the type of devices made by Google, Microsoft and others. Because these are consumer devices, they could be produced in large volumes. That would make them less expensive than the alternatives and possibly lead to more useful applications, much as has happened with smartphones. Although to date, these devices have not entered mass production, so lower prices and greater capabilities would arrive in the future. These wearable computers have see-through displays. If they present data to both eyes, they can easily handle 3D images. They also have the advantage of being able to do so in a variety of contexts, from a laboratory or office setting where surgeries are planned, to potentially an operating room where surgeries are carried out. The use of Microsoft’s HoloLens in medicine is being investigated by the Intervention Centre of Oslo University Hospital in Norway. The Centre and the IT firm Sopra Steria Group of Paris won an award from Microsoft for their use of the HoloLens in a virtual 3D model for surgical planning. “By good depth perception and accurate 3D models, a precise plan can be defined that may be less conservative than with conventional planning,” said Ole Jakob Elle, head of medical cybernetics and image processing at the Intervention Centre. “The see-through glasses give the possibility of augmented reality where a team of surgeons see each other and at the same time can interact with the same virtual model in the planning/decision process.” Professor Bjørn Edwin, a surgeon performing liver tissue removal laparoscopically at the Intervention Centre, worked with Elle and his technology team, as well as developers from Sopra Steria, to create a 3D anatomical map visualized on the HoloLens. This was used for planning how to best remove a tumor from the liver. “This technology is promising, and can be a valuable tool for planning and later possibly guidance during surgery,” Edwin said. A picture of the liver model and how it appears when looking through a Microsoft HoloLens. Ole Jakob Elle, head of medical cybernetics and image processing (left), and Bjørn Edwin, a surgeon, both at the Intervention Centre of Oslo University Hospital, demonstrate augmented reality technology that is being investigated for collaborative operation planning. Courtesy of Hanne Kristine Fjellheim/Sopra Steria. The HoloLens displays information derived from an appropriate source, which can vary based upon what is being imaged. “[Three-dimensional] CT and MRI has come to a sufficient precision level for use as a data source for 3D heart models in most cases,” said Henrik Brun, a consultant in pediatric cardiology and researcher in 3D cardiac visualization at Oslo University Hospital. “Challenges are very small children’s hearts and heart valves that are fine structures not well-visualized by CT and MRI, but better by 3D ultrasound. The present challenge is establishing a good workflow for the visualization of 3D ultrasound images in HoloLens,” he said. Currently, see-through wearables suffer from a limited angular field of view. They also cannot be used in all situations, such as when ambient light overwhelms what is being displayed. Additionally, there are ergonomic concerns, with vendors having trouble putting enough capabilities and batteries in a form factor that must be small and light enough to be worn comfortably for hours. The hope is that with continued improvement, it will be possible to overcome these issues. One potentially significant advantage of see-through displays is mixed reality, in which virtual or synthetic objects interact with real ones. In medicine, this might mean that a CT- or ultrasound-generated image could overlay the living tissue of a patient, revealing structures in real time that would otherwise be invisible. That is beyond the ability of today’s systems to deliver, but that may not be the case in the future. Finally, the reason behind this activity in 3D imaging and visualization is that they are useful in many areas. Kepler University’s Fellner noted that 3D techniques are being used by doctors to create realistic images of fractured bones and other bone lesions. Facial surgeons use 3D techniques to plan procedures, as do cardiac surgeons. By doing so, they can help patients achieve better outcomes. In the future, Fellner may use a large visualization display for education purposes, but that will not be the case for other applications, he predicted. “Moreover, in the future, real-time applications of 3D visualization may be used within surgical procedures, for example real-time 3D rendering of CT and MRI data sets using HoloLens and so on.”