I was wondering if when generating adversarial examples we have seen in class, if restricting the length of e to the distance, if this always guarantees to create an adversarial example? Or is it possible that even after updating x it is still correctly classified.
Additionally, we take the derivative with respect to x. I understand well what gradient decent is, but could someone explain in more basic terms what exactly it means when we take a derivative with respect to the data. It is more of an abstract concept it seems and I want to make sure my intuition of it is correct
First question: indeed, it's not guaranteed to find an adversarial example for an arbitrary perturbation radius \(\varepsilon\). For a very small\(\varepsilon\), the prediction of the model won't change. However, in many practical application (e.g., computer vision models), the required \(\varepsilon\) to change the prediction is quite small and definitely smaller than what one would like to have ideally from a good predictive model.
Second question: I think it's useful to think about optimization of an arbitrary function \(f(x)\) with respect to its argument \(x\). Then it's not so different whether \(x\) are model parameters or input features and the same algorithms (e.g., gradient descent) can be applied. Now if we want to be less abstract and more specific, the gradient of the loss \(\ell(x)\) with respect to the input \(x\) tells us which elements of the input \(x\) should be changed (and by how much) in order to maximize the function \(\ell(x)\) in the neighbourhood around \(x\). For the sake of an example, you want to change the digit 1 to the digit 7 (say, you have MNIST dataset), then the gradient can point out that some upper pixels should be changed from black to white to make the digit 1 look more similar to the digit 7 (and perhaps some other upper pixels changed from white to black as well, but other pixels shouldn't be changed).
Lecture 9 c- adversarial examples
Two questions:
I was wondering if when generating adversarial examples we have seen in class, if restricting the length of e to the distance, if this always guarantees to create an adversarial example? Or is it possible that even after updating x it is still correctly classified.
Additionally, we take the derivative with respect to x. I understand well what gradient decent is, but could someone explain in more basic terms what exactly it means when we take a derivative with respect to the data. It is more of an abstract concept it seems and I want to make sure my intuition of it is correct
Hi,
First question: indeed, it's not guaranteed to find an adversarial example for an arbitrary perturbation radius \(\varepsilon\). For a very small \(\varepsilon\), the prediction of the model won't change. However, in many practical application (e.g., computer vision models), the required \(\varepsilon\) to change the prediction is quite small and definitely smaller than what one would like to have ideally from a good predictive model.
Second question: I think it's useful to think about optimization of an arbitrary function \(f(x)\) with respect to its argument \(x\). Then it's not so different whether \(x\) are model parameters or input features and the same algorithms (e.g., gradient descent) can be applied. Now if we want to be less abstract and more specific, the gradient of the loss \(\ell(x)\) with respect to the input \(x\) tells us which elements of the input \(x\) should be changed (and by how much) in order to maximize the function \(\ell(x)\) in the neighbourhood around \(x\). For the sake of an example, you want to change the digit 1 to the digit 7 (say, you have MNIST dataset), then the gradient can point out that some upper pixels should be changed from black to white to make the digit 1 look more similar to the digit 7 (and perhaps some other upper pixels changed from white to black as well, but other pixels shouldn't be changed).
I hope that helps.
Best,
Maksym
3
Thanks great explaination
Add comment