Everybody is different. Even routine surgical operations vary from patient to patient. That’s why the trend for increasingly personalised and efficient healthcare is driving advances in e-health towards augmented reality solutions right into the operating room, with interventions tailored to the individual’s pathological condition.
Surgery has always used patient data to figure out the best course of action for surgery. With the arrival of augmented reality, doctors can now not only plan the operation, but also draw on medical records, guides and other medical holograms without ever taking their eyes and hands off the patient, using only voice and gestures.
A hologram, from the Greek “holos” (total) and “gramma” (writing), is a 3D object projected in front of the user’s eyes. This is now possible thanks to visors (head-mounted displays, HMDs), developed by companies such as Microsoft and Magic Leap, but also with smartphones.
Medical holograms could be three-dimensional reconstructions of the affected organs or graphical suggestions generated through deep learning or by experts in real time. The hardware is in fact supported by programs developed by companies and universities around the world that explore the different potentials of the technology.
Although due to high costs they are still the prerogative of a few hospitals, these devices promise to bring enormous benefits to the frontline worker, a group of which health workers – it has been made clear by the pandemic – represent a very important share in social terms.
AUGMENTED REALITY AT THE SERVICE OF E-HEALTH
An architect shows you what it will be like to be in your new home before you even build it. You try on a piece of clothing without even trying it on. A virtual character stands in front of you in a simulated gaming environment. Technologies that enlarge our experience of the real, such as augmented reality (AR), virtual reality (VR) and mixed reality (MR) are increasingly a part of everyday life, both in entertainment and professionals. Such systems fall under the umbrella term of “spatial computing”, understood as a rapprochement of computational techniques from binary abstraction to physical reality.
It is precisely in healthcare, driven by the rapidly growing trend of e-health, that these innovations are finding more and more applications. But how are and will advances in spatial computing affect the healthcare industry?
E-HEALTH: THE USE OF HOLOGRAMS IN THE OPERATING ROOM
Practicing on a hologram before performing real-life surgery is already usual procedure in the most cutting-edge hospitals thanks to augmented reality lenses, which allow doctors to practice on a simulation of the patient’s organs superimposed on a mannequin before surgery. In this way, healthcare professionals can study the best course of action, anticipating as much as possible any complications in accordance with the specific conditions and anatomy of the patient. Young doctors can also acquire greater autonomy and skills that would require years of experience in the field.
But holography, a technology that experts note is one of the fastest-growing in the e-health sector, has already made its way to the operating room: now surgeons, wearing an AR or MR device, can interact with the 3D object, walk around it, rotate it, dissect it and zoom in on it.
Cloud computing technology supports the images, with real-time help and suggestions from algorithms and remote colleagues. Medical holography is particularly suitable for the treatment of cardiovascular diseases, liver diseases and in oncology.
WHAT ARE MEDICAL HOLOGRAMS
The holograms used today in various professional fields, especially in the medical sector, are not holograms in the strict sense of the word. Current technology is not yet able to generate true 3D images in space. However, it is possible to simulate their presence by wearing HMDs or smart glasses. However, there are undoubted advantages to visors, because frontline workers can view important information in real time, making task completion more efficient.
Medical holography, for example, allows 3D models of the patient’s anatomy to be displayed, with the ability to rotate or walk around them to observe them from different points of view. This allows the doctor to make more accurate decisions.
Holographic reconstruction of anatomical parts is done through the combination of traditional medical imaging such as magnetic resonance imaging (MRI) or computed tomography (CT). A mapping of the affected anatomy is performed, which is then processed through a 3D imaging process. The end result is an accurate three-dimensional scan of the pathological condition of the organ under examination. In this sense, medical holography represents a renewal in the use of more conventional technologies.
Holomedicine in the operating room is still in its infancy, but it is already bringing important benefits to healthcare professionals and, therefore, to patients.
AUGMENTED REALITY AND SURGERY: WHAT ADVANTAGES
Every day, surgeons use data from X-rays, MRIs, and other information from patients’ records to determine how surgery should be performed and the possible consequences of the aftermath. The use of augmented reality displays in e-health integrates virtual information into the surrounding space of the operating room to improve healthcare quality.
Even before entering the operating room, surgeons can visualize the mapped organ in 3D to first determine what the best course of action is. By dissecting, enlarging and measuring the model, the doctor can get a very sharp image of the patient’s organ. To date, three-dimensional organ reconstruction can only be used for planned surgeries, not emergency surgeries.
With the digital structure superimposed on the patient’s actual organ, the surgeon avoids making mistakes by taking incorrect paths. This is particularly useful, for example, in operations to remove a tumor that is not immediately visible to the operator’s eye. 3D imaging has a whole new level of realism compared to two-dimensional plates and images. The efficiency of training is also improved, as the young doctor no longer has to rely solely on book figures or the generic plastic model of an organ, but can study on realistic simulations.
Nowadays, several software houses and start-ups are busy developing programs for Microsoft’s HoloLens 2 and other hardware to optimize the UI or enable data collection and processing. In the future, machine learning algorithms will be able to combine huge amounts of direct experience from thousands of doctors to provide valuable suggestions in real time. This innovation will be very useful, in particular, during emergencies.
REMOTE SECOND OPINION
The images taken by the viewer can be streamed thousands of miles away (wherever there is a powerful enough internet connection) to the screens of other doctors, with whom the healthcare professionals in the room can consult.
MORE PATIENT AWARENESS
Before the surgery, the patient could also wear the HMD so that they have a clearer idea of the surgery they will undergo.
Access to a hybrid reality interface, where all buttons, medical records and templates are virtual, ensures a more sterile environment.
Among the disadvantages, a 2019 study found that headsets slow down operations in some cases because they are uncomfortable for operators. But this is a problem that will gradually reduce with the arrival of later generations of viewers.
AUGMENTED REALITY AND COVID-19
The use of augmented reality has already found application in pandemic crisis management. Since May 2020, staff at Imperial College Healthcare NHS Trust have been wearing HoloLens, allowing groups of doctors to observe treatments to COVID-19 patients from a safe distance. This has reduced the average exposure time to high risk areas by 83%.
At the same time, the use of personal protective equipment was limited – up to 700 units per week for each department – with considerable savings for the hospital at a time when resources were very scarce.
MR devices allow medical staff to consult 3D digital models using gestures and voice. Without having to engage their hands, doctors can interact with connected colleagues from around the world and the patient at the same time, exchanging timely information about a virus we don’t yet know enough about.
WHAT DEVELOPMENTS FOR AR AND MEDICAL TRAINING
Learning the medical profession is a complex process. Students have to learn complex structures and anatomy traditionally based on textbooks and images, but the distance between 2D figures and real bodies of patients can generate gaps that will have to be filled in a very short time directly in the hospital.
In medical education, simulations in virtual and augmented reality sessions may therefore provide a solution to remove this bottleneck congestion or at least improve the learning curve. Although currently the preserve of a few educational institutions, in the future HMDs will be able to provide potentially infinite teaching materials by facilitating access to educational content in disadvantaged parts of the world. Doctors from developing countries will be able to attend surgeries at prestigious hospitals and be guided and advised by connected colleagues from US and European universities.
Several studies have already shown how the use of HMD, which augments the experience (AR and MR) or creates a simulated environment from scratch (VR), is suitable for the acquisition of skills, especially cognitive skills related to memory and understanding of spatial information and knowledge.
The arrival of holographic technologies, in short, is already bringing several changes to the medical profession. From training to procedural planning, from diagnosis to the operating room, the fields of application are boundless. The current cloud computing power and modern hardware do not yet allow us to speak of a revolution in the healthcare industry, but the path leads towards it.