From image sequence to frontal image: reconstruction of the unknown face
In this thesis we explore how multiple images from a sequence which individually are considered not usable for a forensic procedure, can be combined to reconstruct a face model that is usable in a forensic face comparison procedure. We started by examining the current procedure of forensic face recognition. In the current forensic procedure usually only one image from an entire image sequence is used for face comparison, whereas the remaining images are discarded.
We started with the observation that all the images originate from the same 3D face and therefore contain useful information of the face. The best way to improve the current forensic face comparison procedure is to incorporate multiple images in the reconstruction process to reconstruct a high quality frontal view of the face. We reviewed the existing literature on 3D face reconstruction and found two different ways of reconstructing 3D faces. The first approach is a model driven approach, involving a Morphable Model, which is able to construct 3D faces from single or multiple images. Such an approach would be a considerable advance in forensic face recognition. Unfortunately the model driven approach uses information from external face models which biases the reconstruction towards the average face and is therefore unsuitable in a forensic context. The second approach is a data driven approach, which needs a considerable number of landmarks and images with different pose to reconstruct 3D faces. However, the quality of the images in many forensic cases is low and only a few landmarks can be detected,which makes the reconstruction a challenging task. We concluded that none of the existing face reconstruction methods currently support an unbiased reconstruction method suitable for us a gein a forensic context. Based on the available literature and the forensic setting, we chose a structure from motion approach based on landmarks, which seemed the most suitable method for forensic face recognition. The structure from motion method uses landmarks in multiple images to reconstruct the shape of the face and estimates the rotation and translation of the face simultaneously. We performed experiments on random point clouds to determine the minimum number of views and landmarks needed to obtain a proper 3D reconstruction. Experiments on random point clouds show that a minimum of 20 landmarks in around 30 frames is required to obtain a coarse 3D reconstruction. Secondly, we discovered that a low number of landmarks necessitates calibration of the internal camera parameters to be able to distinguish shape variation from variation in the field of view of the camera. In follow-up experiments with rigid face data we showed that it is possible to reconstruct face-like shapes with similar performance.In the proposed shape reconstruction algorithm, the initial reconstruction is based on a pair of frames with a suitable 2D reprojection error. In an iterative procedure multiple frames were added to improve the 3D reconstruction and the estimates of the rotation and translation of the face in each frame. The low number of landmarks caused the optimization to be sensitive to outliers. After the reconstruction of the shape, we triangulated the landmark model to obtain a surface for the texture of the face. Based on the Lambertian illumination model, we corrected and combined the texture from multiple views. The obtained 3D reconstruction is coarse and can be used to render frontal reconstructions of a face. Face recognition experiments showed that the reconstructed frontal faces already outperformed the original non-frontal images. Inthe final proposed method we revised the reconstruction of the texture and incorporated a denseshape reconstruction method into the proposed reconstruction method. The dense reconstruction method used the coarse shape reconstruction method as initialization, and creates a dense 3D face reconstruction. The dense reconstruction method is based on an iterative procedure where the normals of the reconstructed face are used to optimize both the shape and the texture of the face. The reconstructed dense 3D models can be used to render frontal faces or facesunder small pose. Additional recognition experiments showed that the reconstructed frontalfaces outperformed the original non-frontal images in most of the cases. The proposed reconstruction method is unbiased by design and is therefore suitable in a forensic face comparison process.
