In the first lecture, we noticed how choosing the cost function to be convex leads to the optimisation of the loss function being "easy". However, we also noticed that, in order to be resistant to outliers, we want the cost of a huge mistake to be "just a little bigger" than that of a big mistake, which to me sounds like concavity.
Don't the two properties intrinsicly exclued one another?
Yes, exactly, if we are talking about different loss function for regression, then convexity of the loss with respect to the residual is at odds with the resistance to outliers. The best what we can do in terms of the resistance to outliers while preserving convexity is the L1 loss (but then we still give up smoothness). So there are different trade-offs of this kind which was essentially the message of that part of the lecture: "If we want better statistical properties, then we have to give-up good computational properties."
Outlier resistance VS ease of optimisation
Hello.
In the first lecture, we noticed how choosing the cost function to be convex leads to the optimisation of the loss function being "easy". However, we also noticed that, in order to be resistant to outliers, we want the cost of a huge mistake to be "just a little bigger" than that of a big mistake, which to me sounds like concavity.
Don't the two properties intrinsicly exclued one another?
Yes, exactly, if we are talking about different loss function for regression, then convexity of the loss with respect to the residual is at odds with the resistance to outliers. The best what we can do in terms of the resistance to outliers while preserving convexity is the L1 loss (but then we still give up smoothness). So there are different trade-offs of this kind which was essentially the message of that part of the lecture:
"If we want better statistical properties, then we have to give-up good computational properties."1
Add comment