New Advances In Bayesian Inference For Gaussian Process And Deep Gaussian Process Models
Join Zoom Meeting
https://nus-sg.zoom.us/j/96578109431?pwd=NXhZNkZ1cE5XNmJSU3pBYlUxNGhwZz09
Meeting ID: 965 7810 9431
Password: 615180
Abstract:
Machine learning is the study of letting computers learn to perform a specific task in a data-driven manner. In particular, Bayesian machine learning has attracted enormous attention mainly due to their ability to provide uncertainty estimates following Bayesian inference. This thesis focuses on Gaussian processes (GPs), a rich class of Bayesian nonparametric models for performing Bayesian machine learning with formal measures of predictive uncertainty. However, the applicability of GP in large datasets and in hierarchical composition of GPs is severely limited by computational issues and intractabilities. Therefore, it is crucial to develop accurate and efficient inference algorithms to address these challenges. To this end, this thesis aims at proposing a series of novel approximate Bayesian inference methods for a wide variety of GP models, which unifies the previous literatures, significantly extends them and hopefully lays the foundation for future inference methods.
To start with, this thesis presents a unifying perspective of existing inducing variables-based GP models, sparse GP (SGP) models and variational inference for SGP models (VSGP). Then, to further mitigate the issue of overfitting during optimization, we present a novel variational inference framework for deriving a family of Bayesian SGP regression models, referred to as variational Bayesian SGP (VBSGP) regression models.
%We empirically evaluate the performance of VBSGP regression models on various datasets, including two real-world, massive datasets.
Next, taking into account the fact that the expressiveness of GP and SGP depends heavily on the design of the kernel function, we further extend the expressive power of GP by introducing Deep GP (DGP), which is a hierarchical composition of GP models. Unfortunately, exact inference in DGP is intractable, which has motivated the recent development of deterministic and stochastic approximation methods. However, the deterministic approximation methods yield a biased posterior belief while the stochastic one is computationally costly. In this regard, we present the implicit posterior variational inference (IPVI) framework for DGPs that can ideally recover an unbiased posterior belief and still preserve time efficiency. Inspired by generative adversarial networks, our IPVI framework casts the DGP inference problem as a two-player game in which a Nash equilibrium, interestingly, coincides with an unbiased posterior belief.
Though IPVI will recover the unbiased posterior belief ideally, there is no guarantee that the best response dynamics algorithm, which is used to search for the optimal posterior belief, converges to the optimal solution in practice, which may result in suboptimal performance. Moreover, similar to the stochastic approximation methods, IPVI can only represent the posterior distribution through samples which lacks a crucial property, probability density, of a distribution. To this end, a novel and interesting variational inference framework for DGP models based on the notion of Normalizing Flows (NFs) is introduced. Interestingly, empirical experimental results reveal that the NF framework is more robust than IPVI in terms of training and outperforms IVPI in terms of predictive performance. This makes the NF framework a superior alternative to existing DGP methods.
We hope this thesis at least provides additional confidence and clarity for researchers who are devoting themselves to Bayesian nonparametric models, Gaussian process models in particular. Moreover, We also wish this thesis to offer inspirations for future works, and some thoughts that could be useful for future solutions.