4.5 Gaussian Distribution

 The Gaussian distribution is the most well studied probability distribution for continuous-valued random variables. It is also referred to as the normal distribution. Its importance originates from the fact that it has many computationally convenient properties.

The Gaussian distribution arises naturally when we consider sums of independent and identically distributed random variables. This is known as the central limit theorem.

There are several applications of this in Machine learning like linear regression, density estimation, reinforcement learning etc.

For a univariate random variable, the Gaussian distribution has a density that is given by

$p(x|\mu,\sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)$

The multivariate Gaussian distribution is fully characterized by a mean vectorvector $\mu$ and a covariance matrix $\Sigma$ and defined as

$p(x|\mu,\Sigma)=(2\pi)^{-\frac{D}{2}}|\Sigma|^{-1/2}exp(-\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu))$

where $x \in \mathbb{R}^D$. We write $p(x) = \mathcal{N}(x|\mu,\Sigma)$ or $X \sim \mathcal{N}(x|\mu,\Sigma)$

The special case of the Gaussian with zero mean and identity covariance, that is, $\mu = 0$ and $\Sigma = I$, is referred to as the standard normal distribution.

 The following fig shows a bivariate Gaussian (mesh), with the corresponding contour plot.

 Figure  shows a univariate Gaussian and a bivariate Gaussian with corresponding samples

Gaussians are widely used in statistical estimation and machine learning as they have closed-form expressions for marginal and conditional distributions.We use these closed-form expressions extensively for linear regression. A major advantage of modeling with Gaussian random variables is that variable transformations are often not needed. Since the Gaussian distribution is fully specified by its mean and covariance, we often can obtain the transformed distribution by applying the transformation to the mean and covariance of the random variable.