Testing/jenkins integration #862

fionnodonohoe-xlnx · 2023-07-24T13:59:59Z

Main changes include:

test support for 4 boards: Pynq-Z1, Alveo U250, Kria SOM KV260 and ZCU104
added a bnn sanity test option (creates a single bitstream for the now 4 supported boards)
broke up existing bnn end2end tests into a per-board option
refactored existing Jenkins setup to a declarative Jenkins pipeline
XML results captured for tests - for jenkins GUI
HTML results captured for docker & hardware tests
HTML code coverage results captured for test suites: util, streamline, transform, brevitas_export, notebooks, fpgadataflow

When testing on HW, various issues were seen:

the following commit was needed as U250 tests were failing: e76f20d
there were some erroneous results due to incorrect data generated for the HW tests.
A workaround was put in just to get some tests passing - so a temporary hack script is seen in this PR.
I removed this script when I correctly adjusted the test data here: 61cba65 and 10d34b5
The Jenkinsfile became too large at one point so testing had to be done with multiple job runs. I put the end2end testing and html report code on separate 'feature' branches until I could shrink the Jenkinsfile appropriately and retest

Signed-off-by: Fionn O'Donohoe <[email protected]>

This reverts commit 6003306.

…ew directory Signed-off-by: Fionn O'Donohoe <[email protected]>

…ation

…r more boards to be parameterized Signed-off-by: Fionn O'Donohoe <[email protected]>

Signed-off-by: Fionn O'Donohoe <[email protected]>

…y board marker Signed-off-by: Fionn O'Donohoe <[email protected]>

Signed-off-by: Fionn O'Donohoe <[email protected]>

…ation

Signed-off-by: Fionn O'Donohoe <[email protected]>

…ation

Signed-off-by: Fionn O'Donohoe <[email protected]>

…djust test scripts to workaround these changes Signed-off-by: Fionn O'Donohoe <[email protected]>

Signed-off-by: Fionn O'Donohoe <[email protected]>

… from the pipeline Signed-off-by: Fionn O'Donohoe <[email protected]>

Signed-off-by: Fionn O'Donohoe <[email protected]>

The fpgadataflow tests were placed in their own stage with their own build parameter as the test takes longer than a day to run. This means that this suite cannot sensibly be used in daily CI test runs. Some notes on the stages and their setup: - the when{} block is used as an 'if' statement, checking if a certain input parameter to the pipeline has been set. By default - the fpgadataflow stage will not run unless explicitly set to true by the tester/CI system - FINN_HOST_BUILD_DIR is set to a unique directory per stage for ease of use/test cleanup - catchError is used in order to allow the pipeline to continue to possible future stages if a stage along the way fails. Otherwise the first failed stage found would end the test run Signed-off-by: Fionn O'Donohoe <[email protected]>

Signed-off-by: Fionn O'Donohoe <[email protected]>

…st scripts Signed-off-by: Fionn O'Donohoe <[email protected]>

Signed-off-by: Fionn O'Donohoe <[email protected]>

…d is offline Signed-off-by: Fionn O'Donohoe <[email protected]>

docker/jenkins/Jenkinsfile

[CI] Fix bug with build parameters and result flags sharing common names

…mmands Jenkins unexpectedly trims trailing spaces from env variables. This leads to badly formed inputs for docker. Appending an extra space solves this issues and causes no further problems.

smtalds · 2023-10-06T07:32:52Z

@fionnodonohoe-xlnx @auphelia Hello, can I get information about Kria KV260? Can we use this board for FINN? Do we submit this information to the FINN documentation if it is available?

… conflict

…on 2022.2 tools

auphelia · 2023-10-24T09:51:06Z

Thanks @fionnodonohoe-xlnx and @johnnoel-xlnx !

fionnodonohoe-xlnx added 30 commits April 3, 2023 15:35

[jenkins] add node label to Jenkinsfile

ad4678a

Signed-off-by: Fionn O'Donohoe <[email protected]>

[jenkins] introduce basic declaritive pipeline

a14bf7e

Signed-off-by: Fionn O'Donohoe <[email protected]>

[jenkins] move into the test dir before running quicktest

6003306

Signed-off-by: Fionn O'Donohoe <[email protected]>

[jenkins] keep 30 builds in build history

da54e37

Signed-off-by: Fionn O'Donohoe <[email protected]>

Revert "[jenkins] move into the test dir before running quicktest"

da57f9b

This reverts commit 6003306.

[jenkins] the './' was necessary to run the test, not moving into a n…

0d3d692

…ew directory Signed-off-by: Fionn O'Donohoe <[email protected]>

Merge branch 'dev' into testing/jenkins-integration

caa3548

Merge remote-tracking branch 'origin/dev' into testing/jenkins-integr…

074e6ce

…ation

Change end2end tests to use board type as opposed to kind to allow fo…

5180d1f

…r more boards to be parameterized Signed-off-by: Fionn O'Donohoe <[email protected]>

Add test_deploy method for BNN end2end tests for Jenkins setup

56e4315

Signed-off-by: Fionn O'Donohoe <[email protected]>

Add parameterized tests for all supported boards. Split test matrix b…

5c03333

…y board marker Signed-off-by: Fionn O'Donohoe <[email protected]>

Add scripts used by Jenkins to test bnn end2end hardware tests

8c98882

Signed-off-by: Fionn O'Donohoe <[email protected]>

Add U250 xclbin for end2end bnn testing

b3166e4

Signed-off-by: Fionn O'Donohoe <[email protected]>

Forgot to add test_support_board_map inclusion into test file

b355a6c

Signed-off-by: Fionn O'Donohoe <[email protected]>

Add missing itertools library import

f2872c7

Signed-off-by: Fionn O'Donohoe <[email protected]>

Merge remote-tracking branch 'origin/dev' into testing/jenkins-integr…

65bb10f

…ation

Remove reference to get_build_env

1d6d5ee

Signed-off-by: Fionn O'Donohoe <[email protected]>

Merge remote-tracking branch 'origin/dev' into testing/jenkins-integr…

04c96d6

…ation

Adjust how deployment dirs are created for sanity_bnn suite

1e898d8

Signed-off-by: Fionn O'Donohoe <[email protected]>

Latest dev changes has affected what tests pass or are destructive, a…

a641f01

…djust test scripts to workaround these changes Signed-off-by: Fionn O'Donohoe <[email protected]>

remove additional spacing

ba0d58f

Signed-off-by: Fionn O'Donohoe <[email protected]>

No need for buildDiscarder function in the pipeline itself

111c873

Signed-off-by: Fionn O'Donohoe <[email protected]>

Env variables are controlled by external CI system and can be removed…

746315c

… from the pipeline Signed-off-by: Fionn O'Donohoe <[email protected]>

Specific agent not required when setting up pipeline

91a5437

Signed-off-by: Fionn O'Donohoe <[email protected]>

Add boolean build parameters in order to select tests

8b7d781

Signed-off-by: Fionn O'Donohoe <[email protected]>

Add sanity bitstream build tests

80029f1

Signed-off-by: Fionn O'Donohoe <[email protected]>

Collect all files needed for HW testing, adding a stage to collect te…

3900428

…st scripts Signed-off-by: Fionn O'Donohoe <[email protected]>

Add hw testing stages - only run if build stage was successful

31ef8d6

Signed-off-by: Fionn O'Donohoe <[email protected]>

Only run HW tests if board is online first, fail the pipeline if boar…

674ef26

…d is offline Signed-off-by: Fionn O'Donohoe <[email protected]>

auphelia added 4 commits August 8, 2023 11:39

[GHA] exclude bnn_pynq from quicktest

3df0c17

[CI] Split Jenkinsfiles into CI and testing

066d027

[Tests] Rename board map for tests

6f84ed9

[Tests] Fix bug in reshaping input npy for remote execution

e080625

auphelia requested changes Aug 10, 2023

View reviewed changes

johnnoel and others added 6 commits August 23, 2023 09:54

[CI] Fix bug with build parameters and result flags sharing common names

fe09f06

Merge pull request #877 from johnnoel-xlnx/testing/jenkins-integration

99a589b

[CI] Fix bug with build parameters and result flags sharing common names

Move successful archive step to parallel stage instead of post

161544a

[CI] Append a space to FINN_DOCKER_EXTRA to avoid malformed docker co…

dd7806e

…mmands Jenkins unexpectedly trims trailing spaces from env variables. This leads to badly formed inputs for docker. Appending an extra space solves this issues and causes no further problems.

Merge branch 'dev' into testing/jenkins-integration

011a818

[CI] Address PR comments

56b155f

johnnoel added 5 commits October 9, 2023 11:09

Merge branch 'dev' into testing/jenkins-integration

1ec9e60

[CI] exclude bnn_pynq from quicktest

b2e0473

[CI] fixing linting, lingering line left behind after resolving merge…

98e94f7

… conflict

[NB] make all output paths absolute in advanced notebook

5eb535a

[CI] remove reference to unused hack script

99e9b73

jmonks-amd force-pushed the testing/jenkins-integration branch 4 times, most recently from 46767dd to 1787ec5 Compare October 17, 2023 14:14

[Tests] fix end2end bnn_pynq cnv transpose

07e3b39

jmonks-amd force-pushed the testing/jenkins-integration branch from 1787ec5 to 07e3b39 Compare October 17, 2023 14:27

[Tests] Disabling end2end_bnn_pynq U250 tests failing due to routing …

1bf20d5

…on 2022.2 tools

jmonks-amd force-pushed the testing/jenkins-integration branch from 8d66d9f to 1bf20d5 Compare October 18, 2023 14:52

[CI] Use virtual env instead of Conda for Jenkins testing

4f51ed6

auphelia approved these changes Oct 24, 2023

View reviewed changes

auphelia merged commit 69bd72f into dev Oct 24, 2023
2 checks passed

auphelia deleted the testing/jenkins-integration branch October 24, 2023 09:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Testing/jenkins integration #862

Testing/jenkins integration #862

fionnodonohoe-xlnx commented Jul 24, 2023 •

edited

Loading

smtalds commented Oct 6, 2023 •

edited

Loading

auphelia commented Oct 24, 2023

Testing/jenkins integration #862

Testing/jenkins integration #862

Conversation

fionnodonohoe-xlnx commented Jul 24, 2023 • edited Loading

smtalds commented Oct 6, 2023 • edited Loading

auphelia commented Oct 24, 2023

fionnodonohoe-xlnx commented Jul 24, 2023 •

edited

Loading

smtalds commented Oct 6, 2023 •

edited

Loading