Spying on AR users with their Wi-Fi infrastructure
Problem Statement
Augmented Reality (AR) is increasingly used across gaming, productivity, social interaction, and accessibility applications, such as color blindness correction or spatial audio guidance. However, early or prolonged use of AR can lead to discomfort or stress, often expressed through changes in movement, posture, or interaction patterns [1]. Monitoring these behaviours is key to improving user experience, safety, and inclusivity. Meanwhile, some activities performed by the user may contain sensitive information (different activities, typing a pin code, or showing discomfort/illness). In recent research, it was shown that radio waves (specifically mmWave radar [1]) could be used to extract sensitive information while wearing a AR headset. However, while mmWave is not native to every home, the radio waves belonging to Wi-Fi are everywhere these days. Such radio waves may also include sensitive information related to activities and behaviour [3]. This combination raises concerns about the potential for passive surveillance of AR users through their own networks—often without their awareness.
[1] Alexis D. Souchet, Domitile Lourdeaux, Alain Pagani, and Lisa Rebenitsch. 2022. A narrative review of immersive virtual reality’s ergonomics and risks at the workplace: cybersickness, visual fatigue, muscular fatigue, acute stress, and mental overload. Virtual Reality 27, 19–50 (2023). https://doi.org/10.1007/s10055-022-00672-0. 32 pages. https://link.springer.com/article/10.1007/s10055-022-00672-0
[2] Luoyu Mei, Ruofeng Liu, Zhimeng Yin, Qingchuan Zhao, Wenchao Jiang, Shuai Wang, Kangjie Lu, and Tian He. 2024. MmSpyVR: Exploiting mmWave Radar for Penetrating Obstacles to Uncover Privacy Vulnerability of Virtual Reality. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 8, 4, Article 172 (December 2024), 29 pages. https://dl.acm.org/doi/10.1145/3699772
[3] Yongsen Ma, Gang Zhou, and Shuangquan Wang. 2019. WiFi Sensing with Channel State Information: A Survey. ACM Comput. Surv. 52, 3, Article 46 (May 2020), 36 pages. https://doi.org/10.1145/3310194
Task
You will use existing AR applications (developed by Gwen or self-developed) and collect CSI data while users interact with these apps. CSI data may be gathered from the AR headset itself (if supported) and/or surrounding Wi-Fi-enabled devices. The aim is to investigate whether (un)supervised machine learning can be used to extract meaningful patterns—such as activity type, interaction intensity, or behavioural anomalies—from the CSI data. This could include detecting transitions between tasks, identifying stress or discomfort, or even inferring sensitive actions like.
Your tasks and research likely include some form or combination of (depending on your interests):
• CSI data collection and synchronization with AR activity logs
• Signal processing and feature extraction from CSI streams
• Deep learning analysis (e.g., clustering, classification, anomaly detection)
• Visualization of behavioural patterns and system feedback
Contact
Jeroen Klein Brinke (j.kleinbrinke@utwente.nl)
Allow multiple students: YES