Physiotherapy students could soon be going underneath the skin to see the movement of joints and muscles thanks to the power of augmented reality.

A collaboration between the University of Melbourne and Microsoft is trialling augmented reality to bring the theory of anatomy related to human movement to life for physiotherapy students in practical classes and lecture theatres.

Under the project, different layers of muscles and bones are projected onto a volunteer ‘patient’ to show an internal, and 360 degree, view of the mechanics of body movement in real-time.

Labelled the Augmented Studio, the technology uses projectors and Microsoft Kinect sensors to project images of an anatomy model onto a moving body - turning a human body into an intricate, complex live canvas of skeleton and muscles.

An Australian first for physiotherapy students, the augmented reality project caters for both tactile and visual learning, and aims to encourage cognitive and clinical reasoning.

The interactive technology enables the educator to draw annotations directly on to the virtual body - which are made in real time on the volunteer ‘patient’s’ body.



Video credit: Microsoft Research Centre SocialNUI, the University of Melbourne.

Augmented Studio is the brainchild of Dr Thuong Hoang, a Research Fellow at the Microsoft Research Centre for Social Natural User Interfaces at the University of Melbourne, and his colleagues.

Dr Hoang’s team is now working on linking the technology to tablet devices that will enable physiotherapy students to draw on their annotation of a virtual model, moving on their device.

“That allows them to interact with and also allows them to create their own annotation as a live notebook to capture all the information in the classroom, so that they can do revision,” Dr Hoang says.

“They have to do a lot of in-class revision of the technique being taught in a time-limited practical classroom, and then the tablet interface enables them to record the movement, record the annotations and allow them to do their own annotations, so they can draw directly on the muscle to help with their understanding throughout the class and for their revision.”

Piloting of Augmented Studio with students began last year and will continue this year before it’s launched, initially, as part of the teaching program with first year undergraduate students in 2018.

Dr David Kelly, a physiotherapy lecturer at the University of Melbourne, says the technology enables students to get a visual understanding of how the body moves internally.



Video credit: Paul Burston and Sarah Fisher, Pursuit, the University of Melbourne.

“When you’re talking about translating from a flat text book into something more dynamic, such as thinking about a movement that a patient might be undertaking, translating two dimensional anatomy into three dimensional anatomy is quite a challenge for students,” he says.

“We found, particularly in the first semester when students are just coming into the course, it really does challenge them in how to make that transition, and so I think this is an opportunity for them to actually see, interact with the model and feel.

“You can actually see the muscle moving on the person as they move, and if we use the skeleton projection, they can see the joint moving so we can put together the joint movement, how the patient is moving, which muscles are moving with that particular joint, and I think they get a much better sense of how anatomical structures work in real life.

“I’ve certainly had a number of students who have said there’s been areas which they never really understood, and for the first time they’ve seen it in application and said they really understand this now. For a number of students that’s been helpful.”

While Augmented Studio utilises commercially available technology with purchased virtual 3D anatomical models, the research team has worked to overcome a few implementation challenges, including scaling the images to fit a range of different sized volunteer ‘patients’.

“It would be fair to say that the technology is quite visionary but in application it probably hasn’t quite been there yet and Thuong’s been working hard with his team in developing the technology,” Dr Kelly says.

“We needed to select scenarios where the technology could fairly authentically show us and show the students what they need to know.

“It’s very difficult to look at the more distal areas, so it works best around the trunk, and the hip, knee and shoulder. It’s much more difficult to deal with the limbs further down, just because of the response rates of the technology.

“It’s getting better all of the time. Another aspect is that we’ve been able to work with a tablet with a tutor and the class, but we’re now working on situations where the students themselves can have their own tablets and here you come up against the network limitations.

“It’s very exciting because we can see what we want to do, and it’s how to get the technology to match that.”

While it remains in the early stages of development, Dr Kelly says the augmented reality model aligns the teaching of theory to students’ understanding and use of technology.

“The students that are coming through are much more technologically savvy, they are much more visual, and this technology is really feeding into their strengths,” he says.

“I think that in the future the potential is that this technology may not only be used in teaching institutions but it may be used in practices to demonstrate for patients.

“I think it’s a very exciting future and we’re just getting to the tip of the iceberg. I think there’s a lot more to come.”

Image credit: Microsoft Research Centre SocialNUI, the University of Melbourne.