[Regularization] Why is it better to have small weights in a given model?
Hi,
This might be a very simple question to answer, but I cannot find the answer online...
The point of regularization (for linear models or NNs) is to keep our models as simple as possible. We do this by keeping the sum of the weights as small as possible, in order to minimize the regularization factor.
I realize that I never really understood why having smaller weights would give us a simpler model...
imagine that you have a little changes in x_i (i-th data point that is a part of R^d space), in other words, you have a little bit of variance. So in let's say linear regression you have a dot product of i-th data point x_i and weights vector w. So now lets say that you don't use regularization, and that your model tends to increase the weight vector w to very large values. Therefore, small changes in x_i (or small variance in the unknown distribution of data points x_1,..., x_N) will cause significant change in the dot product of x_i and w, because the w is large! Hence, that is what we call: model (the value of weights w) that has large variance, and is sensitive to small changes in the future (testing) data points. Hope this helps :))
[Regularization] Why is it better to have small weights in a given model?
Hi,
This might be a very simple question to answer, but I cannot find the answer online...
The point of regularization (for linear models or NNs) is to keep our models as simple as possible. We do this by keeping the sum of the weights as small as possible, in order to minimize the regularization factor.
I realize that I never really understood why having smaller weights would give us a simpler model...
Thanks a lot in advance for your clarifications.
1
Ohh okay, I will try to answer the question:
imagine that you have a little changes in x_i (i-th data point that is a part of R^d space), in other words, you have a little bit of variance. So in let's say linear regression you have a dot product of i-th data point x_i and weights vector w. So now lets say that you don't use regularization, and that your model tends to increase the weight vector w to very large values. Therefore, small changes in x_i (or small variance in the unknown distribution of data points x_1,..., x_N) will cause significant change in the dot product of x_i and w, because the w is large! Hence, that is what we call: model (the value of weights w) that has large variance, and is sensitive to small changes in the future (testing) data points. Hope this helps :))
5
Add comment