See News

More accurate biopsy by Augmented Reality

The University of Twente is currently developing a smartphone technology based on the usage of Simultaneous Localization And Mapping (SLAM) and Augmented Reality (AR). This technology enables medical personnel to reconstruct 3D body sections quickly, only by holding the smartphone around the area of interest. This smartphone will display the augmented layers of the 3D skin surface in order to visualize invisible information for medical personnel.

“We would like to layer the information from the MRI and other vein scans (from sensor devices using laser speckle) and make them visible with augmented reality”, says Dr. Eng. Beril Sirmacek (Computer Science). This allows doctors to see the inner body segments, tumors, veins and the status of diabetic ulcers with the use of augmented reality. This visual support enables doctors to oversee the situation better and make better decisions. In a biopsy situation, this visual support can help with guiding the robot arms to reach the tumor for biopsy at the first attempt, instead of taking the off-chance and reach for the correct location by working with a patient for a whole day and making unsuccessful biopsy holes on the patient's body.

The augmented visualization of big data and the guidance with the smartphone application highly relies on the technology of Simultaneous Localization and Mapping (SLAM). “Even though I didn't have a medical background, our common focus on SLAM and AR provides a great platform.”

SLAM box

“When we started this project, our goal was to develop a SLAM box at the UT research group RAM (Robotics and Mechatronics). We can use our localization and 3D reconstruction software on different sensors and not only with the use of smartphones. Our vision will lead to many health applications, and other robotics applications will become more intelligent. With this technology, the algorithms quickly understand the 3D models of the surrounding objects regardless of the imaging sensors which are used.”

Prof. Dr. Ir. Stefano Stramigioli (Chairman of the Group, Robotics and Mechatronics): “SLAM is a core technology in robotics, but its universal usage will only be possible if we have methods to easily interface many sensors in a robotic system. At RAM, we intend to create a modular SLAM box which will automatically reconfigure itself if extra sensors are attached in a Plug&Play fashion. Then it would be possible to connect this “SLAM Box” to a complete robotic system, giving it powerful perception capabilities.”

We would be happy to know other researchers and interested students who would like to collaborate or take on a study and research in this field

Beril Sirmacek

Beril joined the University of Twente in October 2017. “Before starting to work with the Robotics and Mechatronics (RAM) group, she was working on remote sensing big data processing. She has developed a novel framework (known as farmAR) which provides augmented reality (AR) guidance in the field based on automatically processed remote sensing big data results. The implementation helped especially to people in the agricultural field for finding plant diseases and nutrition deficiencies easily with augmented reality. A simple smartphone application made the invisible data visible to users with a non-technical background. The framework received two awards. In 2016, farmAR was the finalist for the Environment, Health and Energy application of the year. And in 2017, it was awarded the winning B2B application of the year by the European Commision for serving to the satellite mission of Europe called Copernicus Masters.

drs. J.G.M. van den Elshout (Janneke)
Press relations (available Mon-Fri)