I have a few questions:
context: carrying out cross validation by splitting into K folds, suppose you have 3 hyperparameters that can each take values (222) possible hyperparameter combination.
1) Should you first separate the whole data into 1) training and 2) hold out test set and perform CV only on 1). In our project 1 for example, would we be expected to have a hold out test set to estimate generalizability error. Or can we just use our entire training set and use the average MSE of the hyperparameters chosen as our generalization performance estimate?
2) If we do the option where we only train with our optimal hyperparameter model on the training portion of our data to test generalization on hold out set. Once we have this error estimate, are we then allowed to retrain on the entire data set? Or does it mean once we evaluate a model's performance on a test set, it is trained and thus we shouldn't retrain it including that test set?
3) In nested CV I understand we get K different sets of optimal hyperparameters, in that case which one should be chosen for the model training? and which set should the training be carried out on?
Hi, to make sure that the estimated error is unbiased, you could use a hold out test set to estimate error because in CV all the data is used for training. After you find the optimal hyper parameters, you could retrain in the entire data set in order to make full use of the data. Nested CV procedure also provides an almost unbiased estimate of the true error. Hyper parameters could be selected based on its performance on the outer loop CV.
CV, hold out test, and nested CV
Hello,
I have a few questions:
context: carrying out cross validation by splitting into K folds, suppose you have 3 hyperparameters that can each take values (222) possible hyperparameter combination.
1) Should you first separate the whole data into 1) training and 2) hold out test set and perform CV only on 1). In our project 1 for example, would we be expected to have a hold out test set to estimate generalizability error. Or can we just use our entire training set and use the average MSE of the hyperparameters chosen as our generalization performance estimate?
2) If we do the option where we only train with our optimal hyperparameter model on the training portion of our data to test generalization on hold out set. Once we have this error estimate, are we then allowed to retrain on the entire data set? Or does it mean once we evaluate a model's performance on a test set, it is trained and thus we shouldn't retrain it including that test set?
3) In nested CV I understand we get K different sets of optimal hyperparameters, in that case which one should be chosen for the model training? and which set should the training be carried out on?
I can't seem to find general consensus online :/
Thank you very much for your help
1
Hi, to make sure that the estimated error is unbiased, you could use a hold out test set to estimate error because in CV all the data is used for training. After you find the optimal hyper parameters, you could retrain in the entire data set in order to make full use of the data. Nested CV procedure also provides an almost unbiased estimate of the true error. Hyper parameters could be selected based on its performance on the outer loop CV.
1
Add comment