We said that (N<D) case is undetermined for linear regression problem.
But if we use SGD we may even not use all of the data points. We can actually use n' number of data points n'<D.
I am a bit confused with that. If we do such thing, is the model not trained well?
Undetermined for linear regression problems (rank deficient) means that there is no single solution. There are multiple solutions. My guess is that GD/SGD will find one of them.
Side note:
SGD uses one data point at the time to update the weights but in the next iteration it uses another data point. At the end, for one epoch, it uses ALL the data points (one at the time) to update the weights.
@paola, @ekrem_fatih, agreed. SGD and GD will only converge to the optimum gradually. they will not achieve the exact solution in most cases, no matter how many steps or epochs. the question how many steps are enough to get sufficient accuracy (prediction accuracy of ML) depends on the hardness of the task, the dataset and model. sometimes less than one epoch (=seeing all points once) can be enough, typically it's more a bit more than one epoch
SGD and (N<D)
We said that (N<D) case is undetermined for linear regression problem.
But if we use SGD we may even not use all of the data points. We can actually use n' number of data points n'<D.
I am a bit confused with that. If we do such thing, is the model not trained well?
Undetermined for linear regression problems (rank deficient) means that there is no single solution. There are multiple solutions. My guess is that GD/SGD will find one of them.
Side note:
SGD uses one data point at the time to update the weights but in the next iteration it uses another data point. At the end, for one epoch, it uses ALL the data points (one at the time) to update the weights.
1
@paola, @ekrem_fatih, agreed. SGD and GD will only converge to the optimum gradually. they will not achieve the exact solution in most cases, no matter how many steps or epochs. the question how many steps are enough to get sufficient accuracy (prediction accuracy of ML) depends on the hardness of the task, the dataset and model. sometimes less than one epoch (=seeing all points once) can be enough, typically it's more a bit more than one epoch
Add comment