Hi,
In the Adversarial ML lecture (robust and nonrobust feature) mentioned that"The first component contains a strong signal component", What's the definition of signal here? (I think it might similar to the signal in "signal processing ", but I never learn it before so any detailed explanation will be helpful). And how can we say a signal is strong or weak? (By value or something else?)
Thanks
In statistics, ML, signal processing, you will often consider the same kind of problems: you are observing a true signal (the meaningful input you want to recover) which will be corrupted by some noise. Let's take the simple Gaussian model. You have features \(x_i\) (let's say of unite norm to simplify), a true parameter \(\theta_*\) and some Gaussian noises \(z_i\sim \mathcal N(0,\sigma^2) \) and you observe the noisy observations:
$$ y_i = x_i^\top \theta_* + z_i ,$$
In order to quantify the difficulty of the problem, you have to quantify how strong is the signal compare to the background noise. Intuitively you see that if \(\|\theta_*\| \) is very small compared to the variance of the noise \(\sigma^2\), the problem will be more difficult and if \(\|\theta_*\| \) is very large compared to the variance of the noise \(\sigma^2\), the problem will be easier.
So you see that in order to say that a signal is strong or weak you have to compare it to the level of noise you have. The ratio of the power of the signal (the meaningful input you want to recover) to the power of the noise is called signal-to-noise ratio and, in some sense, quantify the difficulty of the problem.
The definition of signal
Hi,
In the Adversarial ML lecture (robust and nonrobust feature) mentioned that"The first component contains a strong signal component", What's the definition of signal here? (I think it might similar to the signal in "signal processing ", but I never learn it before so any detailed explanation will be helpful). And how can we say a signal is strong or weak? (By value or something else?)
Thanks
Hi,
In statistics, ML, signal processing, you will often consider the same kind of problems: you are observing a true signal (the meaningful input you want to recover) which will be corrupted by some noise. Let's take the simple Gaussian model. You have features \(x_i\) (let's say of unite norm to simplify), a true parameter \(\theta_*\) and some Gaussian noises \(z_i\sim \mathcal N(0,\sigma^2) \) and you observe the noisy observations:
$$ y_i = x_i^\top \theta_* + z_i ,$$
In order to quantify the difficulty of the problem, you have to quantify how strong is the signal compare to the background noise. Intuitively you see that if \(\|\theta_*\| \) is very small compared to the variance of the noise \(\sigma^2\), the problem will be more difficult and if \(\|\theta_*\| \) is very large compared to the variance of the noise \(\sigma^2\), the problem will be easier.
So you see that in order to say that a signal is strong or weak you have to compare it to the level of noise you have. The ratio of the power of the signal (the meaningful input you want to recover) to the power of the noise is called signal-to-noise ratio and, in some sense, quantify the difficulty of the problem.
4
Add comment