PhD Defence Sven Dummer | Deep Learning with Infinite-Dimensional Priors

Deep Learning with Infinite-Dimensional Priors

Sven Dummer is a PhD student in the Department of Mathematics of Imaging & AI. (Co)Promotors are prof.dr. C. Brune and dr. N. Strisciuglio from the Faculty of Electrical Engineering, Mathematics and Computer Science.

Neural networks, the foundation of artificial intelligence, underlie many applications ranging from language models like ChatGPT to problems in science. While neural networks typically map vectors to vectors, in many of these (scientific) applications, these vectors correspond to discretizations of functions such as images or solutions to partial differential equations (PDEs). Building on such discretizations, recent works have developed models that treat inputs and outputs directly as functions in infinite-dimensional function spaces. Beyond this representational aspect, infinite-dimensional (function) spaces often carry rich prior information about the structure of the input or output. Since large neural networks may memorize the training data instead of learning general patterns, such prior knowledge can guide the training process toward models that work better on new inputs. Moreover, a neural network itself can be viewed as an element of an infinite-dimensional function space. Characterizing the space provides insight into, for instance, how optimization within these spaces yields neural architectures, which algorithms are suited for solving the optimization problem, and what types of generalization guarantees can be given.

This thesis examines the aforementioned infinite-dimensional aspects of neural networks. In particular, this thesis treats the following aspects:

• Infinite-dimensional input and output data representations. Chapters 4 and 5 show the benefits of modeling controls and shapes as infinite-dimensional objects for, respectively, finding efficiently controllable periodic trajectories and diffeomorphic shape latent modeling. For time-dependent PDEs, Chapter 6 combines reduced-order models and neural operators to study different infinite-dimensional representations. The chapter shows that simple input processing and output representation can already be effective. The chapter also includes a discretization error analysis and highlights the importance of understanding how discretized operators approximate their infinite-dimensional counterparts.
• Priors on infinite-dimensional spaces. Chapter 3 shows the benefits of optimal transport and manifold learning for data-driven dynamic imaging. Chapter 4 uses physics-informed periodicity losses derived from Hamiltonian structure and eigenmanifold theory to compute efficiently controllable periodic motions. Chapter 5 demonstrates the advantages of Large Deformation Diffeomorphic Metric Mapping priors for diffeomorphic shape latent modeling. Chapter 6 combines neural operators with the causality and manifold structure of reduced-order models, although more research is needed to better determine the effect of this structure.
• Hypothesis spaces for neural network models. Chapter 7 introduces a novel definition of vector-valued reproducing kernel Banach spaces, establishes their fundamental properties, and constructs concrete examples corresponding to neural networks and operators. Among the main results in this chapter are representer theorems.

Overall, this thesis explored infinite dimensionality in deep learning across various applications. The results shed light on how to handle and represent infinite-dimensional inputs and outputs. They also clarify when to incorporate prior information on the output and how to define the space corresponding to models that operate in infinite-dimensional settings. Together, these findings advance the understanding and development of methods that handle infinite-dimensional input and output spaces.