Multimodal & Temporal Operational-Condition Recognition using Camera, GNSS and IMU
AlFusIOn (RAAK-PRO) – WP4: AI & Data Management
Project Introduction
This assignment is part of AlFusIOn – AI-based robust indoor/outdoor localization for autonomous mobile robots – a RAAK-PRO project in which the Ambient Intelligence (AMI) research group leads Work Package 4 (WP4): AI & Data Management.
A camera alone can be fooled – a glass atrium looks “outdoor”, a covered yard looks “indoor”. Fortunately a robot also carries GNSS, an IMU and wheel odometry, whose signals strongly hint at the environment: a sudden drop in satellite count or a rising HDOP usually means the robot is entering a tunnel or building. This assignment combines camera images with GNSS-quality and motion features, over short time sequences, to recognise indoor / outdoor / transition more robustly and to anticipate a transition before it happens.
You will design a multimodal, temporal operational-condition model and the confidence-aware output that the AlFusIOn fusion stack (WP5) consumes – telling it which localization modules to trust, down-weight or switch during transitions. It builds naturally on the camera-only baseline.
Open-source datasets you will use
These datasets provide synchronised camera + GNSS + IMU (and often LiDAR), so operational-condition labels can be auto-derived from GNSS quality and route structure:
Dataset | Modalities / labels | Role in this assignment | Use |
|---|---|---|---|
KITTI | Camera, LiDAR, GPS/IMU; odometry | Baseline multimodal sync and feature extraction | Core |
Oxford RobotCar | Camera, LiDAR, GPS/INS; tunnels, weather | Long routes with transitions for sequence modelling | Core |
NCLT | Camera, LiDAR, GPS/IMU; indoor + outdoor | Real indoor↔outdoor transitions for labels | Core |
nuScenes / Boreas | Camera, LiDAR, radar, GNSS/IMU | Multimodal robustness; Boreas adds snow/rain | Support |
i2Nav-Robot | Stereo, LiDAR, GNSS, IMU; IO routes | Explicit indoor/outdoor transition sequences | Benchmark |
BDD100K | Dashcam video; weather/time labels | Temporal/weather variety for sequence augmentation | Support |
Student Tasks
1. Literature & dataset scan
• Review multimodal learning and temporal models (CNN-LSTM, transformers, temporal smoothing / Bayesian filtering) and GNSS-quality-based indoor/outdoor detection.
• Map the multimodal datasets above; document which signals (fix status, satellite count, HDOP, IMU, odometry) are available in each.
2. Data preparation & auto-labeling
• Extract synchronised camera frames + GNSS-quality features + IMU/odometry features and build short sequence samples (e.g. 1–3 second windows).
• Auto-derive indoor / outdoor / transition labels from GNSS availability and route metadata; manually verify the transition boundaries.
3. Multimodal & temporal model
• Build a model that fuses image features with GNSS/IMU context features, plus a temporal layer (LSTM or sliding-window) to smooth predictions and estimate transition risk / distance.
• Output operational condition + calibrated confidence + transition risk, and flag uncertainty for safe fallback.
4. Interface & evaluation
• Design a ROS2-style OperationalCondition output (operational_condition, condition_confidence, transition_risk, recommended_fusion_action, uncertainty flag) – specification and a rosbag-based prototype.
• Evaluate transition precision/recall, detection delay before a transition, calibration (ECE) and robustness to GNSS degradation and lighting; compare against a camera-only baseline.
5. Documentation & handover
• Provide reproducible code and a short report; explain how the confidence-aware output helps WP5 decide which localization modules to weight, activate or deactivate during IO transitions.
Practical information
Student profile: for HBO-ICT (Saxion) and University of Twente students interested in sensor fusion, time-series / multimodal machine learning, Python and robotics. Experience with PyTorch and an interest in ROS2 are a plus. A proactive, independent attitude is key.
Scope: graduation or internship project (approx. one semester); individual or pair.
Contact
Alessandro Chiumento – Pervasive Systems, EEMCS, University of Twente (a.chiumento@utwente.nl)
Eyuel Ayele – Ambient Intelligence (AMI) research group, Saxion University of Applied Sciences (e.d.ayele@saxion.nl)
Javier Ferreira Gonzalez – Ambient Intelligence (AMI) research group, Saxion University of Applied Sciences (j.ferreiragonzalez@saxion.nl)
More info: www.saxion.nl/ami