| Excerpt |
|---|
|
linear regression via maximum likelihood and some simple assumptions about the system. |
| Table of Contents |
|---|
| maxLevel | 2 |
|---|
| type | flat |
|---|
| printable | false |
|---|
|
Assumptions
- Independence : the data points are independent
- No knowledge about the weights : we assume no prior belief about the weights (the training data will dictate).
- Uniform uncertainty : each data point has the same uncertainty (i.e. ), again no prior belief about this.
Problem Definition
We wish to establish the parameters for a set of gaussian distributions that can straddle a linear function. These distributions should be centred (have means) equal to the estimated function, and a constant variance independent of
. The linear parameters (weights) and constant variance should maximise the likelihood for a given set of data points.
Mathematically, this can be stated as saying that for any
the corresponding optimal
is a guassian distributed variable around a linear function with weights
:
| Mathblock |
|---|
|
y = \aleph(x^T w, \sigma^2) = x^T w + \aleph(0, \sigma^2) |
This is basically saying we are taking a discrete set of points and expecting the corresponding
to vary around a mean given by
.
Estimation
The likelihood is the probability of an observation given the data. Given that the data points are independent,
| Mathblock |
|---|
\begin{align*}
p(y|X,w,\sigma) &= \prod_{i=1}^{n} p(y_i|x_i,w,\sigma) \\
&= \prod_{i=1}^{n} (2\pi\sigma^2)^{-1/2} \exp\left(-\frac{1}{2\sigma^2} (y_i - x_i^Tw)^2\right) \\
&= (2\pi\sigma^2)^{-n/2} \exp\left(-\frac{1}{2\sigma^2} \sum_{i=1}^{n}(y_i - x_i^Tw)^2\right) \\
&= (2\pi\sigma^2)^{-n/2} \exp\left(-\frac{1}{2\sigma^2} (y - Xw)^T(y-Xw)\right) \\
\end{align*} |
Given a set of observed
the goal here is to find the linear weights
and the
such that the maximum likelihood (the probability above) is maximised. This is essentially the same as finding the set of guassian distributions such that the means are located as close as possible to the observed
. To find the maximum, it is easier to find the equivalent maximum of the
log likelihood.
| Mathblock |
|---|
\begin{align*}
L(w) &= -\frac{n}{2}\ln(2\pi\sigma^2) - \frac{1}{2\sigma^2} (Y-Xw)^T(Y-Xw) \\
\frac{dL(w)}{dw} &= 0 - \frac{1}{2\sigma^2} (0 - 2X^Ty + X^TXw)\\
\end{align*} |
which by equating to zero, gives the maximum at,
| Mathblock |
|---|
\hat{w} = (X^TX)^{-1}X^Ty |
Similarly, we can find the best choice of
by taking the derivative and equating to zero, resulting in:
| Mathblock |
|---|
\hat{\sigma}^2 = \frac{1}{n}(Y-Xw)^T(Y-Xw) = \frac{1}{n}\sum_{i=1}^n(y_i-x_iw)^2 |
Prediction
The guassian distribution of
at some arbitrary point
follows from the generated estimation of the weights and variance above and can be expressed as:
| Mathblock |
|---|
|
y \sim \aleph(y|x_*^T\hat{w}, \hat{\sigma}^2) |
Implicit here is that the probability distribution for
above is actually a conditional probability assuming that all of the other characteristics are fixed, i.e.
| Mathinline |
|---|
| body | p(y|x_*,X,\hat{w},\hat{\sigma}) |
|---|
|
.
Our Naivete
The assumptions here represent a naivete about the problem that can be expounded upon in several ways.
- What if we have expert knowledge (prior belief) about what the weights should be? We should then be looking at MAP, not ML.
- Likewise can be said of having a prior belief about the uncertainties.
- The uncertainty itself may have a non-constant form, i.e. training points may be more certain on some sub-domain of and less certain elsewhere.
- In more complicated scenarios, data points are not necessarily independent. For example, continuity can influence covariance so that points close to each other are highly correlated.
- A traditional way of influencing the weights to keep them simple is to use optimisation techniques which add a penalty term to the optimisation.
- Refer to these lecture notes for notes on how to do this with ridge regression or lasso techniques.