Pervasive Systems (PS)

Ardarsh_Photorealistic

Photorealistic 3D Reconstruction from Camera and LiDAR

Problem Statement

UAV operators usually build situational awareness (SA) from live 2D video feeds. Video gives them one camera and one viewpoint at a time, and no persistent picture of the area. A 3D model that the operator can move through freely would make it easier to inspect the scene from any angle and to improve SA.

3D Gaussian Splatting (3DGS) makes this practical. It reconstructs a scene from images with known camera poses and renders new viewpoints photorealistically in real time. Cameras alone measure depth poorly, though, so the reconstructed geometry is often inaccurate, especially in regions seen from only a few angles. LiDAR measures depth directly and can correct this.

There is a second problem. A 3DGS model renders plausible-looking content even in regions the sensors barely observed, so an operator cannot tell measured structure from guessed structure. This gets worse when the operator looks at the scene from a different viewpoint than it was captured from, for example from above a scene recorded at street level.

This thesis studies whether adding LiDAR to 3DGS gives a more accurate reconstruction, and whether a simple coverage map can show the operator which parts of the rendered scene to trust. To keep the scope manageable, sensor poses and camera–LiDAR calibration are assumed to be ideal (taken from ground truth), and the scene is treated as static.

Research Questions

•      RQ1: How much does LiDAR improve the geometric accuracy and rendering quality of 3DGS compared with a camera-only reconstruction?

•      RQ2: How much does reconstruction quality degrade when the scene is viewed from operator-like viewpoints away from the capture trajectory?

•      RQ3: Can a simple coverage map, based on how often and from which angles each region was observed, indicate where the rendered scene is unreliable?

Approach

The student will reconstruct scenes from a public camera–LiDAR dataset with an open-source 3DGS implementation, in two versions:

•      Camera-only 3DGS, the baseline.

•      LiDAR-supported 3DGS: the Gaussians start from the LiDAR point cloud, coloured by projecting it into the cameras, and rendered depth is kept consistent with LiDAR depth during training.

Both versions are evaluated in the same way. The student will then compute a coverage map for each scene and check whether regions with low coverage are the regions where rendering errors are large.

Evaluation

Both versions use the same scenes, ground-truth poses and train/test split. Held-out views are split into views close to the capture trajectory and operator-like views away from it, such as elevated viewpoints. Image quality is measured with PSNR, SSIM and LPIPS. Geometric accuracy is measured as depth error against LiDAR. Training time and rendering frame rate are also recorded. The coverage map is evaluated by comparing it with the per-region rendering error on held-out views.

Recommended Background

•      Good Python skills; some experience with PyTorch.

•      Basic linear algebra and camera geometry (projection, camera intrinsics and extrinsics), or a willingness to learn them early in the project.

•      Interest in 3D computer vision and multi-sensor data.

•      Comfort working with GPUs, Linux and the command line.

•      No prior experience with 3D reconstruction is required.

Work

15% Theory, 65% Experiments & Implementation, 20% Writing

Contact

Adarsh Nanjaiya Latha (a.nanjaiyalatha@utwente.nl) Pervasive Systems Group, EEMCS