Resources

• Guassian Processes - mathematical monk@youtube on guassian processes (also follows on through 10 parts).

Guassian Process

Definition

For any set , a Guassian Process (GP) on  is a  set of random variables  such that  is multivariate gaussian.

Examples

From ML 19.1 (mathematicalmonk).

Trivial Example

Here we have d random variables, and any subset of these can form a perfectly valid multivariate guassian.

Random Lines

For any  randomly sampled form the guassian, the resulting  is a line. Hence, random lines.

Then forming given , we have

which we know is a multivariate guassian distribution because multivariate guassian distributions are closed under affine transforms (i.e. linear). It has zero mean and covariance:

This is equivalent to saying the guassian process has a covariance function:

Random Planes

Given data , .

Then bayesian linear regression models the  with random variables  which are assumed to be independent given some parameter vector . The distribution we assign the random variables here is of course guassian:

or equivalently:

The parameter vector we assume to have a prior of the form:

Note that  is a multivariate guassian. So for any , if we consider  (or ), this is then just a linear combination of univariate guassians which from the closure property of guassians  (Guassian Distributions:2) implies that  is also guassian, and more specifically, univariate guassian.

To see this, take any . Now since we know  is univariate,

which is an affine transform of a multivariate guassian, so it too must be a multivariate guassian. Hence  is indeed a guassian process. The mean of the guassian process:

and the covariance of the guassian process comes from:

where we have made use of the equation for the covariance (?) in breaking it down and substituting back for  at the end.  This is exactly the kernel from guassian process examples of linear planes.

Existence of Guassian Processes

For any set , any mean function  ,and any kernel (covariance) function , there exists a Guassian Process  on  such that , and covariance .

Examples

• Random Lines/Planes : 
• Brownian Motion :  
• Smooth Function : 

The first function is curious - the singular value decomposition for the guassian generator (?) permits only the very first element of  to contribute to the random vector. This causes all remaining elements to remain locked into the random lines. For the others, the covariance rules effect the sampled random vector by weighting covariant values closer or farther from each other differently (when covariances are higher for close points, and almost zero for far points, this intuitively contributes to continuity).  More details in mathematical monk's video  ML 19.3.

Stationary and Non-Stationary GPs

A kernel function  which is only dependent on , i.e. , is called a stationary guassian process. This induces similar behaviour of the GP everywhere over the entire domain. If instead, you wish for different behaviour (e.g. different smoothness properties in different subsets of the domain), then a nonstationary guassian process is required. These are generally far more computationally heavy.