Pervasive Systems (PS)

Multimodal & Temporal Operational-Condition Recognition using Camera, GNSS and IMU

Multimodal & Temporal Operational-Condition Recognition using Camera, GNSS and IMU

AlFusIOn (RAAK-PRO) – WP4: AI & Data Management

Project Introduction

This assignment is part of AlFusIOn – AI-based robust indoor/outdoor localization for autonomous mobile robots – a RAAK-PRO project in which the Ambient Intelligence (AMI) research group leads Work Package 4 (WP4): AI & Data Management.

A camera alone can be fooled – a glass atrium looks “outdoor”, a covered yard looks “indoor”. Fortunately a robot also carries GNSS, an IMU and wheel odometry, whose signals strongly hint at the environment: a sudden drop in satellite count or a rising HDOP usually means the robot is entering a tunnel or building. This assignment combines camera images with GNSS-quality and motion features, over short time sequences, to recognise indoor / outdoor / transition more robustly and to anticipate a transition before it happens.

You will design a multimodal, temporal operational-condition model and the confidence-aware output that the AlFusIOn fusion stack (WP5) consumes – telling it which localization modules to trust, down-weight or switch during transitions. It builds naturally on the camera-only baseline.

Open-source datasets you will use

These datasets provide synchronised camera + GNSS + IMU (and often LiDAR), so operational-condition labels can be auto-derived from GNSS quality and route structure:

Dataset

Modalities / labels

Role in this assignment

Use

KITTI

Camera, LiDAR, GPS/IMU; odometry

Baseline multimodal sync and feature extraction

Core

Oxford RobotCar

Camera, LiDAR, GPS/INS; tunnels, weather

Long routes with transitions for sequence modelling

Core

NCLT

Camera, LiDAR, GPS/IMU; indoor + outdoor

Real indoor↔outdoor transitions for labels

Core

nuScenes / Boreas

Camera, LiDAR, radar, GNSS/IMU

Multimodal robustness; Boreas adds snow/rain

Support

i2Nav-Robot

Stereo, LiDAR, GNSS, IMU; IO routes

Explicit indoor/outdoor transition sequences

Benchmark

BDD100K

Dashcam video; weather/time labels

Temporal/weather variety for sequence augmentation

Support

 

Student Tasks

1.         Literature & dataset scan

•  Review multimodal learning and temporal models (CNN-LSTM, transformers, temporal smoothing / Bayesian filtering) and GNSS-quality-based indoor/outdoor detection.

•  Map the multimodal datasets above; document which signals (fix status, satellite count, HDOP, IMU, odometry) are available in each.

2.         Data preparation & auto-labeling

•  Extract synchronised camera frames + GNSS-quality features + IMU/odometry features and build short sequence samples (e.g. 1–3 second windows).

•  Auto-derive indoor / outdoor / transition labels from GNSS availability and route metadata; manually verify the transition boundaries.

3.         Multimodal & temporal model

•  Build a model that fuses image features with GNSS/IMU context features, plus a temporal layer (LSTM or sliding-window) to smooth predictions and estimate transition risk / distance.

•  Output operational condition + calibrated confidence + transition risk, and flag uncertainty for safe fallback.

4.         Interface & evaluation

•  Design a ROS2-style OperationalCondition output (operational_condition, condition_confidence, transition_risk, recommended_fusion_action, uncertainty flag) – specification and a rosbag-based prototype.

•  Evaluate transition precision/recall, detection delay before a transition, calibration (ECE) and robustness to GNSS degradation and lighting; compare against a camera-only baseline.

5.         Documentation & handover

•  Provide reproducible code and a short report; explain how the confidence-aware output helps WP5 decide which localization modules to weight, activate or deactivate during IO transitions.


Practical information

Student profile: for HBO-ICT (Saxion) and University of Twente students interested in sensor fusion, time-series / multimodal machine learning, Python and robotics. Experience with PyTorch and an interest in ROS2 are a plus. A proactive, independent attitude is key.

Scope: graduation or internship project (approx. one semester); individual or pair.


Contact

Alessandro Chiumento – Pervasive Systems, EEMCS, University of Twente (a.chiumento@utwente.nl)

Eyuel Ayele – Ambient Intelligence (AMI) research group, Saxion University of Applied Sciences (e.d.ayele@saxion.nl)

Javier Ferreira Gonzalez – Ambient Intelligence (AMI) research group, Saxion University of Applied Sciences (j.ferreiragonzalez@saxion.nl)

More info: www.saxion.nl/ami