Is linear regression prone to overfit? Because to me it is only prone to overfit, once we augment the features using a polynomial basis, or am I wrong?
Hello,
Linear regression is prone to overfitting.
Take the simple case where we have two data points i.e n=2 and with p=1 explanatory variable. Fitting a linear regression would yield a line that goes through the two points. Note that in linear regression, you suspect that y= xw+eps, where w are the true parameters and eps is random noise (Gaussian in the case of normal linear regression). Fitting the line that goes through the two points captures the noise and can fail to generalize if presented with a new y.
Please let me know if this does not answer you question.
Linear Regression & Overfit
Is linear regression prone to overfit? Because to me it is only prone to overfit, once we augment the features using a polynomial basis, or am I wrong?
1
Hello,
Linear regression is prone to overfitting.
Take the simple case where we have two data points i.e n=2 and with p=1 explanatory variable. Fitting a linear regression would yield a line that goes through the two points. Note that in linear regression, you suspect that y= xw+eps, where w are the true parameters and eps is random noise (Gaussian in the case of normal linear regression). Fitting the line that goes through the two points captures the noise and can fail to generalize if presented with a new y.
Please let me know if this does not answer you question.
Karim
5
Add comment