Home » Blog » Tech Trends » Report: Using Extended Reality in Healthcare

Report: Using Extended Reality in Healthcare

 

The healthcare industry is ripe for embracing immersive xR learning technologies in training professionals and informing patients.

After a 25-year career as a learning scientist and the last 3 years “immersed” in the private sector, my  opinion is that xR technologies have the potential to improve the quality and quantity of training, to reduce training costs and to enhance patient satisfaction through better care from healthcare professionals and a deeper understanding for patients.

To support this claim, I outline a framework for understanding how the brain learns. It derives from some of my own research supported by over $10 million in federal grants to support my laboratory, and a huge body of research conducted across the world over the past 100+ years focused on learning.

xR technologies have the potential to improve the quality and quantity of training, to reduce training costs and to enhance patient satisfaction Click To Tweet

This report focuses on a learning science evaluation of xR technology’s potential in the healthcare sector. xR technologies come in two major forms: virtual reality (VR) in which the learner is immersed in a completely new virtual environment and augmented reality (AR) in which the learner is in a combined real and virtual environment where digital information is overlaid onto the learner’s field of view.

Healthcare professionals must be trained on anatomy and physiology and a broad range of physiological systems both healthy and diseased, and learn to perform a wide range of procedures in their speciality. Additionally, they must learn situational awareness which involves the ability to perform effectively under stressful and high-pressure conditions.

Analogously, patients must learn what pills go in their pill box, or a patient or caregiver might need to learn how to care and maintain a central line. Patients must learn about an upcoming medical procedure, how to prepare and what to expect following the procedure. They might also learn about several medical options so that they can choose the one that is right for them, as is the case when a dialysis patient receives training on in-clinic and in-home dialysis so that they can make an informed decision on which suits them best. The more effective the training, the less stress on the patient, the higher likelihood of success.

Learning to perform medical procedures is about acquiring a technical or behavioral skill - this is best done through physical repetitions Click To Tweet
Learning Science of Healthcare Training

Learning science—the marriage of psychology and brain science— teaches us that different types of learning tasks are mediated by distinct brain system and psychological processes and thus are best trained with different training tools. For example, anatomy and physiology is about obtaining knowledge and facts to be remembered and available for recall at a later date. This is learned through mental repetitions and by the cognitive skills learning system in the brain. On the other hand, learning to perform medical procedures is about acquiring a technical or behavioral skill. This is best done through physical repetitions and uses the behavioral skills learning system in the brain. Finally, situational awareness—the ability to perform under a broad range of conditions and to reasonably anticipate the future–involves mental and physical repetitions across a broad range of scenarios and is mediated by the emotional learning system in the brain.

Dissociable Learning Systems in the Brain

The human brain is comprised of at least three distinct learning systems. Gaining a solid knowledge base (e.g., anatomy and physiology or what pills go in your pill box) relies on the cognitive skills learning system in the brain. The cognitive skills learning system in the brain has evolved to obtain knowledge and facts and to tie those to specific situations. I refer to this as the “what” system. Cognitive skill learning is mediated by the prefrontal cortex, hippocampus and associated medial temporal lobe structures in the brain and relies heavily on working memory and attention.

 

Tech Trends Virtual Reality Todd Maddox Immersive Tech and brain science

 

Obtaining a strong behavioral repertoire (e.g., drawing blood, performing surgery, or caring for your own central line) relies on the behavioral skills learning system in the brain. It is one thing to know what to do, but it is another to know how to do it. I refer to this as the “how” system. Behavioral skill learning is mediated by the basal ganglia and gradual, incremental dopamine-mediated changes in behavior. Interestingly, this system does not rely on working memory and attention. In fact, there is strong scientific evidence that “overthinking it” hinders behavioral skills learning. Processing in this system is optimized when behavior is interactive and is followed in real-time (literally within milliseconds) by corrective feedback. Behaviors that are rewarded will be more likely to occur in the future, and those that are punished less so.

The human brain is comprised of at least three distinct learning systems Click To Tweet

The ability to “read” a situation and to have strong situational awareness (e.g., an ability to perform under non-ideal conditions and to reasonably predict the near-term future) relies on emotional learning. The emotional learning system in the brain has evolved to facilitate the development of a nuanced understanding of situations and people that is critical to success. Whereas one can have all of the facts and figures available, and have a strong behavioral repertoire, in the end one has to extract the appropriate information and engage the appropriate behavior in each distinct situation. I refer to this as the “feel” system. The critical brain regions are the amygdala and other limbic structures. The detailed processing characteristics of this system are less well understood than the cognitive and behavioral skills learning systems, but emotional learning is at the heart of situational awareness. Emotional processing strongly affects both cognitive and behavioral skills learning. An individual with strong situational awareness can accurately read any situation, adapts quickly and knows what to do in each situation, and has the behavioral repertoire to engage each situation with the appropriate set of behaviors.

The ability to read a situation and to have strong situational awareness relies on emotional learning Click To Tweet
Cognitive Skills Learning System

Gaining a solid knowledge base such as understanding the anatomy and physiology of the human body, or knowing what medication to take, how much and when, relies on the cognitive skills learning system in the brain. This system learns the “what,” and is mediated by the prefrontal cortex, hippocampus and associated medial temporal lobe structures in the brain and relies heavily on working memory and attention.

The prefrontal cortex is where working memory and attention reside. The goal is to use focused study and mental rehearsal to transfer fleeting and poorly represented knowledge from the prefrontal cortex to long-term memory stores in the hippocampus and associated medial temporal lobe structures.

Tech Trends Virtual Reality Todd Maddox Immersive Tech and brain science

Suppose you are a medical student faced with the problem of learning the anatomy and physiology of the human body. The human body is a 3-dimensional structure that functions as a dynamic system. The ultimate goal of training is to facilitate the formation of a 3D dynamic mental representation of the human body in the learner’s brain that perfectly mimics the actual form. The best way to achieve this is to present the learner with a 3D dynamic visualization, yet most traditional teaching methods use textbooks or slideshows filled with 2D static images. Thus, the learner must convert a series of 2D static images into a 3D dynamic mental representation in the brain that accurately reflects the human form.

Attempting to construct a 3-dimensional dynamic representation of the human body from a series of 2-dimensional static images requires a huge amount of cognitive effort. First, you have to hold a mental representation of a series of 2-dimensional static images in short-term (working) memory. Second, you have to hold these 2-dimensional static mental representations in your working memory and combine them on the fly to construct an accurate 3-dimensional static representation. Finally, you have to infer and impart the dynamic nature of the human form onto this 3-dimensional static representation. Each of these steps requires an enormous amount of cognitive capacity (in the form of working memory), and an enormous amount of cognitive energy (in the form of executive attention). Any time working memory load and executive attentional demands are taxed, you are more likely to make an error and generate an inferior mental representation.

Now consider an xR solution in which you place a Microsoft HoloLens on your head and a 3-dimensional, dynamic representation of the human body appears in front of you. You can walk around the body and can rotate it so that you can see it from all angles. You can select a skeletal view and when you touch a bone its name appears. In this case, you have a highly accurate 3-dimensional dynamic xR visualization tool that is intuitive and facilitates the development of a highly accurate 3-dimensional dynamic representation in the learner’s brain. A tool like this engages the visual representation learning system in the brain that recruits occipital and temporal lobe structures. By removing the need to construct a 3-dimensional dynamic mental representation from a series of 2-dimensional static images, the working memory and executive attention load on the learner has been slashed. Those resources can be used to learn the names of the bones, muscles, arteries, etc., but with a rich visual mental representation upon which to attach them.

Constructing a 3-D dynamic representation of the human body from a series of 2-D static images requires a huge amount of cognitive effort Click To Tweet

The same logic holds for other cognitive skills learning problems. Suppose you are a patient or caregiver who needs to care for and maintain a central line. The most common approach to central line care and maintenance training is to have learners read documents describing in detail all of the required steps. From a psychological and brain science perspective translating this abstract text-based representation into a rich visual representation that you can use to guide your behavior is challenging. Yet putting on a HMD (Head Mounted Display) and being transported into an immersive setting in which you are watching someone care and maintain a central line in a virtual world in real-time instantly reduces the areas in the brain which are engaged. Learning will be enhanced, quicker and more robust.

Suppose you are a patient with a hefty medication regimen. You have a long list of medications that you take at different times and in different amounts. You have a pill box that facilitates the process but you struggle to use you list of medications when filling the pill box. Now imagine an augmented reality solution where all of your medication information is in the application. It knows what you take, when and how much. It also knows what each pill looks like and monitors the process with a small camera on your mobile device or tablet. Finally, the tool has information about the medication available that can be overlaid on the screen. The AR application prompts you with text, audio and a visual representation of each pill to facilitate loading the pill box. As with the xR solution to anatomy and physiology training and central line care and maintenance, this AR solution significantly reduces the cognitive load on the learner, which is especially critical in older adults.

AR solutions can significantly reduce the cognitive load on the learner, which is especially critical in older adults Click To Tweet
The Behavioral Skills Learning System

The behavioral skills learning system has evolved to learn behaviors. I refer to this as the “how” system because it allows you to learn how to do things. The behavioral skills learning system resides in the basal ganglia of the brain that is a subcortical structure deep in the center of the brain. Behavioral skill learning does not rely on working memory and attention, in fact, there is strong anecdotal evidence, and basic science research to suggest that “overthinking it” hinders behavioral skills learning. Behaviors are learned through gradual, incremental, dopamine-mediated reward/punishment learning in the basal ganglia of the brain. Processing in this system is optimized when behavior is interactive and is followed in real-time by corrective feedback.

Suppose you are a surgeon in training learning how to perform heart surgery. The patient’s vital signs, your hand on the scalpel, the sensation associated with breaking the skin all represent a collection of sensory inputs (e.g., visual, auditory, tactile; red arrow). Collections of neurons that represent the current situation are active and project down into the basal ganglia (in purple). There they connect with neurons that project from the basal ganglia to motor regions (red arrow) that initiate a specific behavior associated with the surgery. If the behavior was successful that behavior is rewarded. In this case, dopamine will be released into the basal ganglia, the neural connections that drove that behavior will be strengthened, and the likelihood that behavior will be elicited again under the same surgical situation will increase. On the other hand, if the behavior was incorrect that behavior is punished. In this case dopamine will not be released, the neural connections that drive that behavior will be weakened, and the likelihood that behavior will be elicited again under the same surgical situation will decrease.

Tech Trends Virtual Reality Todd Maddox Immersive Tech and brain science

 

Establishing behavioral skills requires learning a set of motor behaviors to be completed in a specific sequence. Although a common approach is to start by reading textbooks and slideshows or watching someone else complete the task, in the end the only way to learn a behavioral skill is to actually do it.

Because behavioral skills are learned gradually and incrementally via dopamine-mediated, error-correction learning in the basal ganglia of the brain, they require extensive practice, yet traditional real-world approaches are limited due to difficulty and expense.

xR medical training technologies are being developed which bypass these difficulties, however. Some of the more advanced platforms include realistic artificial cadavers that provide the appropriate haptic feedback, for example. When implemented correctly, technologies like this will speed the training of medical procedures and will give learners enough practice that they will be job-ready before entering a medical facility to see patients.

These xR technologies could also be used for patients and their caregivers. Instead of watching someone perform in-home dialysis once, the patient and caregiver could train in xR until they reach some level of proficiency. This would increase their confidence, reduce risk and reduce the likelihood of a hospital visit due to preventable errors.

When implemented correctly, technologies like this will speed the training of medical procedures and will give learners enough practice that they will be job-ready before entering a medical facility to see patients Click To Tweet
The Emotional Learning System

The emotional learning system has evolved to facilitate the development of situational awareness. I refer to this as the “feel” system because it is all about “reading” individuals and groups, and situations, in general. Emotional learning involves the amygdala and associated limbic system structures. This system is critical to situational awareness through its effects on processing in both the cognitive and behavioral skills learning systems in the brain. Emotional learning affects how one processes and links the cognitive and behavioral aspects of a situation. A nurse or doctor with strong situational awareness knows what to say and do when appropriate. They know how to be compassionate or firm – only after years of exposure to multiple situations do medical personnel gain the expertise and situational awareness necessary to respond effectively in all situations. With xR technologies though, the medical practitioner and the patient can receive training in a broad range of routine, but also non-routine situations.

The broad-based training available with an xR medical training platform can increase generalization and transfer of the trained information Click To Tweet

Although the three learning systems in the brain are distinct and involve different cognitive and emotional processes, they ultimately all reside within a single brain with massive interconnections. Thus, the optimal method for training situational awareness is to combine the cognitive skills associated with gaining facts and knowledge with the behavioral skills associated with appropriate action in all situations. xR technologies offer an ideal medium for achieving these goals and engaging all three systems simultaneously.

The broad-based training available with an xR medical training platform can increase generalization and transfer of the trained information, but most importantly can increase preparedness. Imagine, for example, the EMT who must deliver a baby in the back of the ambulance, without some critical piece of equipment and with a mother whose blood pressure is dangerously low.

xR technologies offer an ideal medium for engaging all three systems of the brain simultaneously Click To Tweet

The same goes for medical patients. More and more in-home medical options are available. To be effective though patients need extensive training. Imagine an xR experience that leads a patient through the process of in-home dialysis step by step, tests the patient’s knowledge throughout and is readily available 24/7 for follow-up training. In addition, suppose the xR training tool puts the patient-in-training under time pressure sometimes.

All these examples exist today or the solution is in development, so the future looks bright for xR applications in healthcare-related cognitive skills learning and the healthcare industry is ripe for the introduction of immersive technologies, which have the potential to improve all aspects of healthcare training, reduce costs, increase healthcare professionals’ readiness, and most importantly, instill confidence in patients thus enhancing patient satisfaction.

Todd Maddox is Science, Sports and Training Correspondent at Tech Trends, and the CEO of Cognitive Design and Statistical Consulting. Follow him on Twitter @wtoddmaddox