We are dealing with time series and were wondering what metric and validation method was typically used. We tried at every step training on the data we have so far and then predicting the next immediate point (or n points into the future using known features at those points) and comparing with real values. At the end we will thus have n observations with n-1 predictions and n-1 accuracy measures. This gives an accuracy metric for the model, for example if we take the average of these accuracies. We in fact have multiple time series for different users. We want a model that can be trained across multiple different user. So in this case should the validation and hyperparameter tuning be done on a few users and then the test performance be found by applying on other unseen users. In this case we would also average accuracy measures at time n across all users for the overall model performance for a given set of hyperparameters.
Or would the validation/test split have to be done with respect to time? How do we decide along which dimension user or time the split should be, is there multiple right answers for how to do this? And if it was to be done wrt time I'm not sure how that would work since we want to predict the immediate next step I'm not quite sure what the validation set would be.
time series
Hello,
We are dealing with time series and were wondering what metric and validation method was typically used. We tried at every step training on the data we have so far and then predicting the next immediate point (or n points into the future using known features at those points) and comparing with real values. At the end we will thus have n observations with n-1 predictions and n-1 accuracy measures. This gives an accuracy metric for the model, for example if we take the average of these accuracies. We in fact have multiple time series for different users. We want a model that can be trained across multiple different user. So in this case should the validation and hyperparameter tuning be done on a few users and then the test performance be found by applying on other unseen users. In this case we would also average accuracy measures at time n across all users for the overall model performance for a given set of hyperparameters.
Or would the validation/test split have to be done with respect to time? How do we decide along which dimension user or time the split should be, is there multiple right answers for how to do this? And if it was to be done wrt time I'm not sure how that would work since we want to predict the immediate next step I'm not quite sure what the validation set would be.
Any insight would be appreciated
Add comment