Model Accuracy Effect of FINN

[Ba1tu3han]

Hello All,

I would like to clarify a point that I'm stuck on.

Is there any setting, parameter or command of FINN that effects the model accuracy in FPGA?

I trained a custom simplified CNV network with a dataset by using Brevitas QAT. Then the test accuracy is around 95% however after compiling it with FINN and testing it again with the same test dataset in PYNQ-Z2 it gives around 40% accuracy. I could not solve it yet.

I am open to any suggestion.

Thanks in advance,

[iksnagreb]

You should have a look at the verification flow of the FINN build to see where your original model and the FINN transformed/generated model diverge. Normally your final accelerator should produce exactly (up to some small(!) rounding errors) the same outputs as your original ONNX export. The potential rounding errors should not be large enough to significantly reduce your accuracy, so there must be something going wrong here. With the verification flow, FINN checks after certain graph transformations whether the transformed model still produces the expected output for some known input. For an introduction to the built-in verification, you could have a look at the "Verification steps" section of the advanced builder settings notebook: https://github.com/Xilinx/finn/blob/dev/notebooks/advanced/4_advanced_builder_settings.ipynb

[Ba1tu3han]

Hello @iksnagreb

You are right. I didn't follow the verification step. I jumped into the building step directly.

Thank you for noticing about that!

[Ba1tu3han]

I have investigated it and I found the root cause. In streamlining steps something goes wrong. I change some steps to make it similiar with the builder's code, It worked. But I have no idea why it is working. I was using builder command and I started to use end2end CNV example. With the end2end example I am able to compile without any accuracy drop however, when I use builder example from cybersecurity example it compiles with accuracy drop. I can compare it in PYNQ by running inference code.

What do you recommend to use builder commands without accuracy drop?

What should I change for non-binary CNV example in end2end (it is for binary)?

My model is the exactly the same CNV which is given in Brevitas Repo.