Difference between objective function and evaluation metric
Hello,
I have a question about the difference between objective functions and evaluation metrics. Do they need to be the same?
I mean for example, in logistic regression, we optimize our model based on the binary cross-entropy (BCE) loss function, but we measure the performance of our model using other metrics such as "accuracy", or "F1-score", or "AUC score", .. etc.
However, how can we guarantee that low values for binary (or cross)-entropy loss functions map to high values for accuracy or F1-score, etc?
Also, in linear regression, do we include the regularization term during evaluating the model after the training is done? or it is just used for optimization during training, and then we evaluate in test time using the vanilla MSE without any additional components?
Thank you so much.
As a side note, I really enjoyed the last lecture about Fairness and Ethics in ML. I was going to ask if it's possible to include it as the "Guest Lecture", but it seems that I did not need to ask for that eventually :)
Very good question.
The objective is in optimization vocabulary the function that you want to optimize, in machine learning this objective is in general constituted of a goodness-of-fit term and a regularization term, that is a term that measures how much you fit observed data and a term that penalises complex models which tend to overfit (i. e. focus on interpolating rather than generalizing to unseen data).
An evaluation metric evaluates how the model performs on some set of examples (seen or unseen).
There is no reason why your goal and metric couldn't be the same, but in general when your metric is the same as your objective/goal, then this means that you know exactly what you want your model to do, in hard science(math, physics, maybe biology) this is true in general. However in other domains of application of ML this is not always the case (there is something for example that's called the specification problem in AI : how to tell a machine what you exactly want without the machine finding a way to inadvertently doing something that you didn't want?) for this reason your metric is in general different than your objective.
In addition, there are very simple reasons why you can't use for example the accuracy as an objective, how would you optimize for accuracy? it is just too difficult to optimize for some metrics directly, hence why they make bad objectives.
Also, some metrics are meaningful, and sometimes you want to use different metrics to evaluate your model.
I will end up by one last reason why you wouldn't want your metric to be your goal, Goodheart's law (when you optimize for a metric it ceases to be a good one, because it can be gamed).
As for the regularization question, in test you don't include the regularization part, because you only want to know how much your model fits unseen data, then again in general you use a metric that is different than your loss, and the metric has no regularization term.
Difference between objective function and evaluation metric
Hello,
I have a question about the difference between objective functions and evaluation metrics. Do they need to be the same?
I mean for example, in logistic regression, we optimize our model based on the binary cross-entropy (BCE) loss function, but we measure the performance of our model using other metrics such as "accuracy", or "F1-score", or "AUC score", .. etc.
However, how can we guarantee that low values for binary (or cross)-entropy loss functions map to high values for accuracy or F1-score, etc?
Also, in linear regression, do we include the regularization term during evaluating the model after the training is done? or it is just used for optimization during training, and then we evaluate in test time using the vanilla MSE without any additional components?
Thank you so much.
As a side note, I really enjoyed the last lecture about Fairness and Ethics in ML. I was going to ask if it's possible to include it as the "Guest Lecture", but it seems that I did not need to ask for that eventually :)
Very good question.
The objective is in optimization vocabulary the function that you want to optimize, in machine learning this objective is in general constituted of a goodness-of-fit term and a regularization term, that is a term that measures how much you fit observed data and a term that penalises complex models which tend to overfit (i. e. focus on interpolating rather than generalizing to unseen data).
An evaluation metric evaluates how the model performs on some set of examples (seen or unseen).
There is no reason why your goal and metric couldn't be the same, but in general when your metric is the same as your objective/goal, then this means that you know exactly what you want your model to do, in hard science(math, physics, maybe biology) this is true in general. However in other domains of application of ML this is not always the case (there is something for example that's called the specification problem in AI : how to tell a machine what you exactly want without the machine finding a way to inadvertently doing something that you didn't want?) for this reason your metric is in general different than your objective.
In addition, there are very simple reasons why you can't use for example the accuracy as an objective, how would you optimize for accuracy? it is just too difficult to optimize for some metrics directly, hence why they make bad objectives.
Also, some metrics are meaningful, and sometimes you want to use different metrics to evaluate your model.
I will end up by one last reason why you wouldn't want your metric to be your goal, Goodheart's law (when you optimize for a metric it ceases to be a good one, because it can be gamed).
As for the regularization question, in test you don't include the regularization part, because you only want to know how much your model fits unseen data, then again in general you use a metric that is different than your loss, and the metric has no regularization term.
I hope I answered your question.
1
Add comment