Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run End to End test suites as different jobs and fix timeouts #3241

Merged
merged 3 commits into from
Aug 19, 2024
Merged

Run End to End test suites as different jobs and fix timeouts #3241

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b0c0d6a
Reduce delay for password-autofill test
alessandroboron Aug 14, 2024
2d72b91
Fix test due to DOM changes
alessandroboron Aug 14, 2024
17fba95
Split tests in multiple jobs
alessandroboron Aug 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 50 additions & 26 deletions .github/workflows/end-to-end.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,11 @@ on:
workflow_dispatch:

jobs:
end-to-end-tests:
name: End to end Tests
build-end-to-end-tests:
name: Build End to End Tests
runs-on: macos-14-xlarge

timeout-minutes: 30

steps:
- name: Check out the code
uses: actions/checkout@v3
Expand Down Expand Up @@ -48,26 +49,59 @@ jobs:
ONLY_ACTIVE_ARCH=NO \
| tee xcodebuild.log

- name: Install Maestro
run: |
export MAESTRO_VERSION=1.36.0; curl -Ls "https://get.maestro.mobile.dev" | bash
- name: Store Binary
uses: actions/upload-artifact@v4
with:
name: duckduckgo-ios-app
path: DerivedData/Build/Products/Debug-iphonesimulator/DuckDuckGo.app

- name: Upload logs when workflow failed
uses: actions/upload-artifact@v4
if: failure() || cancelled()
with:
name: BuildLogs
path: |
xcodebuild.log
DerivedData/Logs/Test/*.xcresult
retention-days: 7

- name: Release tests
run: |
export PATH="$PATH":"$HOME/.maestro/bin"; maestro cloud --apiKey ${{ secrets.MAESTRO_CLOUD_API_KEY }} -e ONBOARDING_COMPLETED=true --fail-on-timeout=true --fail-on-cancellation=true --timeout=150 --ios-version=17 --include-tags=release DerivedData/Build/Products/Debug-iphonesimulator/DuckDuckGo.app .maestro/
end-to-end-tests:
Copy link
Contributor Author

@alessandroboron alessandroboron Aug 15, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently I have setup the concurrency matrix in this way

strategy:
      matrix:
        test-tag: [release, privacy, securityTest, adClick]
      max-parallel: 1 # Uncomment this line to run tests sequentially. 
      fail-fast: false

and use ${{ matrix.test-tag }} in the command below to pass the tag to the command

export PATH="$PATH":"$HOME/.maestro/bin"; maestro cloud --apiKey ${{ secrets.MAESTRO_CLOUD_API_KEY }} -e ONBOARDING_COMPLETED=true --fail-on-timeout=true --fail-on-cancellation=true --timeout=150 --ios-version=17 --include-tags=${{ matrix.test-tag }} DerivedData/Build/Products/Debug-iphonesimulator/DuckDuckGo.app .maestro/

If we decide to have run onboarding tests in CI we can also add onboarding-completed to the matrix

strategy:
    matrix:
      test-tag: [release, privacy, securityTest, adClick, onboarding]
      include:
        - test-tag: release
          onboarding-completed: true
        - test-tag: privacy
          onboarding-completed: true
        - test-tag: securityTest
          onboarding-completed: true
        - test-tag: adClick
          onboarding-completed: true
        - test-tag: onboarding
          onboarding-completed: false

and use variable ${{ matrix.onboarding-completed }} as below

export PATH="$PATH":"$HOME/.maestro/bin"; maestro cloud --apiKey ${{ secrets.MAESTRO_CLOUD_API_KEY }} -e ONBOARDING_COMPLETED=${{ matrix.onboarding-completed }}--fail-on-timeout=true --fail-on-cancellation=true --timeout=150 --ios-version=17 --include-tags=${{ matrix.test-tag }} DerivedData/Build/Products/Debug-iphonesimulator/DuckDuckGo.app .maestro/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually there’s no even need to modify the matrix because the onboarding tests overrides ONBOARDING_COMPLETED env variable in their flow 🎉

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't have time to test it out properly today, but the overall approach looks good to me 👍

name: End to end Tests
needs: build-end-to-end-tests
runs-on: macos-14-xlarge
timeout-minutes: 90
strategy:
matrix:
test-tag: [release, privacy, securityTest, adClick]
max-parallel: 1 # Uncomment this line to run tests sequentially.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you mean “Comment out” rather than “Uncomment” here? 🙂

fail-fast: false

- name: Privacy tests
run: |
export PATH="$PATH":"$HOME/.maestro/bin"; maestro cloud --apiKey ${{ secrets.MAESTRO_CLOUD_API_KEY }} -e ONBOARDING_COMPLETED=true --fail-on-timeout=true --fail-on-cancellation=true --timeout=150 --ios-version=17 --include-tags=privacy DerivedData/Build/Products/Debug-iphonesimulator/DuckDuckGo.app .maestro/
steps:

- name: Check out the code
uses: actions/checkout@v3 # Don't need submodules here as this is only for the tests folder

- name: Retrieve Binary
uses: actions/download-artifact@v4
with:
name: duckduckgo-ios-app
path: DerivedData/Build/Products/Debug-iphonesimulator/DuckDuckGo.app

- name: Security tests
- name: Install Maestro
run: |
export PATH="$PATH":"$HOME/.maestro/bin"; maestro cloud --apiKey ${{ secrets.MAESTRO_CLOUD_API_KEY }} -e ONBOARDING_COMPLETED=true --fail-on-timeout=true --fail-on-cancellation=true --timeout=150 --ios-version=17 --include-tags=securityTest DerivedData/Build/Products/Debug-iphonesimulator/DuckDuckGo.app .maestro/
export MAESTRO_VERSION=1.36.0; curl -Ls "https://get.maestro.mobile.dev" | bash

- name: Ad Click Detection Flow tests
- name: End to End tests
run: |
export PATH="$PATH":"$HOME/.maestro/bin"; maestro cloud --apiKey ${{ secrets.MAESTRO_CLOUD_API_KEY }} -e ONBOARDING_COMPLETED=true --fail-on-timeout=true --fail-on-cancellation=true --timeout=150 --ios-version=17 --include-tags=adClick DerivedData/Build/Products/Debug-iphonesimulator/DuckDuckGo.app .maestro/
export PATH="$PATH":"$HOME/.maestro/bin"; maestro cloud --apiKey ${{ secrets.MAESTRO_CLOUD_API_KEY }} -e ONBOARDING_COMPLETED=true --fail-on-timeout=true --fail-on-cancellation=true --timeout=150 --ios-version=17 --include-tags=${{ matrix.test-tag }} DerivedData/Build/Products/Debug-iphonesimulator/DuckDuckGo.app .maestro/

notify-failure:
name: Notify on failure
if: ${{ always() && contains(join(needs.*.result, ','), 'failure') && github.ref_name == 'main' }}
needs: [build-end-to-end-tests, end-to-end-tests]
runs-on: ubuntu-latest

steps:
- name: Create Asana task when workflow failed
if: ${{ failure() }}
run: |
Expand All @@ -76,13 +110,3 @@ jobs:
--header "Authorization: Bearer ${{ secrets.ASANA_ACCESS_TOKEN }}" \
--header "Content-Type: application/json" \
--data ' { "data": { "name": "GH Workflow Failure - End to end tests", "workspace": "${{ vars.GH_ASANA_WORKSPACE_ID }}", "projects": [ "${{ vars.GH_ASANA_IOS_APP_PROJECT_ID }}" ], "notes" : "The end to end workflow has failed. See https://github.com/duckduckgo/iOS/actions/runs/${{ github.run_id }}. For instructions on how to handle the failure(s), check https://app.asana.com/0/0/1206423571874502/f" } }'

- name: Upload logs when workflow failed
uses: actions/upload-artifact@v4
if: failure() || cancelled()
with:
name: BuildLogs
path: |
xcodebuild.log
DerivedData/Logs/Test/*.xcresult
retention-days: 7
2 changes: 1 addition & 1 deletion .maestro/data_clearing_tests/02_duckduckgo_settings.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ tags:
- pressKey: Enter

# Change settings
- tapOn: "Safe search: moderate"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems the webpage slightly changed and this copy wasn’t found anymore

- tapOn: "Safe search"
- tapOn: "Off"

# Fire Button - twice, just to be sure
Expand Down
8 changes: 0 additions & 8 deletions .maestro/release_tests/password-autofill.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,12 +45,8 @@ tags:
- tapOn: "Close"

# Validate standard form
- runFlow:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These delays were causing some random timeouts in the test so I removed few of them and ensure that the test still pass.

file: ../shared/delay.yaml
- assertVisible:
id: "searchEntry"
- runFlow:
file: ../shared/delay.yaml
- tapOn:
id: "searchEntry"
- runFlow:
Expand All @@ -71,11 +67,7 @@ tags:
# Validate multistep form
- tapOn:
id: "searchEntry"
- runFlow:
file: ../shared/delay.yaml
- inputText: "https://privacy-test-pages.site/autofill/autoprompt/3-multistep-form.html"
- runFlow:
file: ../shared/delay.yaml
- pressKey: Enter
- runFlow:
file: ../shared/delay.yaml
Expand Down
Loading