increasing RMSE in ridge regression with degree in polynomial expansion
We found out that, when doing cross-validation with ridge regression, increasing the degree always implies an increase in the RMSE of the test set, in contrast with what we saw in the lecture and the lab. Meaning that degree 1 has always the smallest RMSE, even if we choose a perfect lambda for each degree. We are cleaning NaN values before the cross-validation, and we are applying standardization in each k-fold for train and set data independently.
Could this be mistake on the implementation of our code? Or is this expected?
In principle, it's possible that you overfit by using a large polynomial degree and the test performance gets worse. But it's a bit suspicious that the best degree based on cross-validation is exactly 1.
Maybe as a sanity check you can try to check whether the training RMSE goes down (and training accuracy goes up) when you increase the polynomial degree. If this doesn't happen, maybe you have some optimization issues, and e.g., you may need to adjust your step size.
increasing RMSE in ridge regression with degree in polynomial expansion
We found out that, when doing cross-validation with ridge regression, increasing the degree always implies an increase in the RMSE of the test set, in contrast with what we saw in the lecture and the lab. Meaning that degree 1 has always the smallest RMSE, even if we choose a perfect lambda for each degree. We are cleaning NaN values before the cross-validation, and we are applying standardization in each k-fold for train and set data independently.
Could this be mistake on the implementation of our code? Or is this expected?
In principle, it's possible that you overfit by using a large polynomial degree and the test performance gets worse. But it's a bit suspicious that the best degree based on cross-validation is exactly 1.
Maybe as a sanity check you can try to check whether the training RMSE goes down (and training accuracy goes up) when you increase the polynomial degree. If this doesn't happen, maybe you have some optimization issues, and e.g., you may need to adjust your step size.
Add comment