Augmented Reality Brings Jets Into The Classroom

Holograms, used for decades in the entertainment industry, may soon become part of the aviation training environment thanks to a team of students at Embry-Riddle Aeronautical University, Prescott.

The students are part of ERAU’s Simulation Science, Games and Animation program. Over the summer, the student team developed augmented-reality (AR) holograms, creating 3D models of the CRJ-700.

Using HoloLens 2 goggles, the user steps through a series of training modules composed of interactive slides.

The AR aspect of the project allows the student to move around inside the aircraft systems, making learning akin to something out of “Fantastic Voyage,” a classic science-fiction movie where a submarine and its crew is shrunk to microscopic size to allow them venture into the body of an injured colleague to provide life-saving medical treatment.

It allows students to explore the CRJ-700 in a way they never could before.

“We encouraged the team to consider the reasons for using augmented reality rather than virtual reality on the project,” associate professor and SSGA program chair Derek Fisher explained. “They determined that while virtual reality might provide a more immersive alternate experience, AR provides opportunities to focus on interacting with the computer-generated jet while collaborating in the familiar environment of the classroom, with real-life colleagues.”

Another type of hologram model in development could let the students look around the flight deck, manipulating the buttons, switches, and controls as they would in real life. The HoloLens 2 can even “see” their hands. This feature will be a benefit for tactile and kinetic learners, who need to see and touch something for optimum learning to take place.

The holographic slides contain informational labels. These labels can be manipulated by the user, making them larger or rotating them. The idea is that the users will become familiar with the controls of the aircraft and the system schematics, such as the radar danger zones and exhaust system.

The labels are removed when it is time for testing and the students can be asked to identify the parts and systems that they’ve been studying.

Evan LaBate, a senior SSGA major, notes he is gaining valuable experience as part of the developmental team, and envisions the simulations will not only enhance flight training on campus, but may also be used in other technical fields.

“Participating on this project has helped me decide that I want to do virtual or augmented reality simulations, either for military or police, as my career,” LaBate said. “I believe this technology can be used to help the military simulate live firefights, or even help with the Air Force, in giving pilots updated heads-up displays.”

“This project was a massive learning experience,” says Josh Snow, a SSGA senior. “Jobs in the field of interactive technology are plentiful but demanding, often working on large projects in small teams. The skills I have picked up from this research opportunity have better prepared me for all facets of working on and or managing a team in my career.”

The initial project was funded by an Undergraduate Research Institute grant. Now the team is focusing on improving the software to allow multiple users to interact with the same jet simulation simultaneously. After testing the new prototype, the group will seek funding to expand the program to a full-class experience.

“This is the first step down a very good road for us as a university, potentially toward an entire class in AR,” said Dr. Michelle Hight, assistant professor of aeronautical science and fixed wing program chair.

“Even if students had to attend class from home, this brings the jet to life. Every student who has a HoloLens 2 could join in 3D, standing or sitting in a virtual classroom, interacting with hologram learning tools rather than a flat screen. This is a timely pedagogical development for a world evolving and responding to a global pandemic.”

ERAU’s application of augmented reality is the next logical step in flight training, said Michael Puoci, a Seattle-area pilot and video game designer with more than 20 years of experience. Aviation games, especially those that involve vintage aircraft cockpits, are a passion for Puoci. During his student pilot days, he often used virtual-reality computer cockpits that he designed to practice real-world flights.

“Augmented reality will be what pushes aviation training devices to a new level,” Puoci said. “With VR, you have a mouse and screens. With AR, the person will put on the glasses and what used to be presented on screens will be a digital fabrication and full immersion and far better fidelity for simulation training for new pilots.”