Sanity checks

[martinmatak]

Three tests introduced (all passing):

GIVEN a NN and fixed hyperparameters for the FGSM attack
WHEN the attack is executed twice against the NN
THEN results should be completely the same

GIVEN random seed and hyperparameters
WHEN two neural networks are trained
THEN they should be completely the same

GIVEN enough number of epochs for training
WHEN two neural networks with same architectures are trained (using the different seed)
THEN they should have similar accuracy

[zvonimir]

These are not all the tests we discussed, right?

[martinmatak]

What am I missing? What else should I add? This is a photo I took from the last meeting (first three points are related to testing) 

[zvonimir]

GIVEN fixed hyperparameters for the FGSM attack and enough number of epochs for training
WHEN two neural networks with same architectures are trained (using the different seed)
THEN results of attacks should be completely (or almost) the same

[martinmatak]

@zvonimir Thank you, I updated PR based on your comment.

I wasn't certain how to assert THEN results of attacks should be completely (or almost) the same precisely, so I did as follows:

• accuracies of NNs before the attack must be similar [3% diff allowed]
• accuracies of NNs after the attack must be similar [3% diff allowed]
• perturbations introduced must be similar (i.e. mean, std dev, min and max of differences between adv samples and legit samples) [1% diff allowed]

Should I add something else? What do you think?

[zvonimir]

Why would accuracies of NNs change before and after attack? I mean, your target NNs remain constant. So I don't get that part.

Yep, perturbations should be the same. Meaning that the generated pairs of adversarial images should be (almost) identical. So not just average diff and so on, but the actual adversarial images should be the same. Georg mentioned opening a few pairs of images in photoshop and doing a diff there to make sure they are the same.

[martinmatak]

Why would accuracies of NNs change before and after attack? I mean, your target NNs remain constant. So I don't get that part.

Sorry, I didn't express myself precisely enough. What I meant is the following:

• Accuracy of NN_1 and NN_2 measured on legit samples should be similar
• Accuracy of NN_1 measured on adv samples crafted for NN_1 should be similar to accuracy of NN_2 measured on adv samples crafted for NN_2

Meaning that the generated pairs of adversarial images should be (almost) identical. So not just average diff and so on, but the actual adversarial images should be the same. Georg mentioned opening a few pairs of images in photoshop and doing a diff there to make sure they are the same.

I added plotting of samples (image below).

Four columns in the image below represent the following:

1. original sample
2. adv sample for NN_1
3. adv sample for NN_2
4. aboslute difference between adv samples from 2. and 3. column

Regarding completely the same (values of pixels for) adversarial samples, they occur only when the attack is executed against the same NN twice (or two NNs with with same weights, i.e. trained with the same seed etc. - effectively same NN as verified in https://github.com/soarlab/AAQNN/pull/8/files#diff-d0b33b1baec7d17a5a87a9ce85c0f612).

This is verified with this assertion:

AAQNN/tests/attacks/fgsm-reproducibility.py

Line 74 in f4ce18f

assert np.array_equal(adv_1, adv_2)

and I added plotting of that (image below). Columns represent same values as in previous image.

Do you have maybe any other idea for sanity check? To me the attack seems good for our use case.

[martinmatak]

Now when I think about it, it might be the case that perturbation introduced by this attack is always of the same size because it just changes the image in the opposite direction of gradient for some eps.

Nevertheless, if we measure robustness per quantization level, it's still a suitable attack.

I believe an optimization approach would be more informative regarding the needed perturbation, i.e. results could vary depending on the quantization. For instance #7

[zvonimir]

I think we should maybe move this exchange to email so that Georg can participate as well. Could you please summarize all this in an email to Georg and me? Thanks!

[martinmatak]

@zvonimir can I merge this branch?

[zvonimir]

Yes, please go ahead and merge.