Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

iOS end to end workflow changes related to iOS 17 and iPad #6445

Merged
merged 1 commit into from
Aug 8, 2024

Conversation

niklasberglund
Copy link
Collaborator

@niklasberglund niklasberglund commented Jul 4, 2024

This PR changes how iOS end to end test workflows are set up because we ran into some issues. The test iPhone was changed. First it was changed to an iPad and then to another iPhone. The new iPhone is running iOS 17 which don't reinstall the app when launched from tests. So the workflow instead invoke multiple test jobs which run one test suite each. This way we have more control of what we want to do in between test suite runs in CI(uninstall app).

image

This PR also adds support for running tests on iPad, some minor changes were needed. There's a new configuration property TEST_DEVICE_IS_IPAD which should be set in UITests.xcconfig. Also it can now be configured in UITests.xcconfig whether app should be uninstalled in test suite teardown. This is by default true(1) for running locally, set to false(0) in CI.

The workflow file ios-end-to-end-tests.yml is now a reusable workflow which is reused by other thin workflows.
image
More information about this in ios/MullvadVPNUITests/README.md.

There are some things that should probably still be improved on:

  • Test device is currently specified by setting the environment variable TEST_DEVICE_UDID in ios-end-to-end-tests.yml and ios-end-to-end-tests-settings-migration.yml. This should be more dynamic to be easier to set up runner on other computers and to run tests on multiple devices.
  • Settings migration tests have not been updated. They might be easier to maintain if they also use the reusable workflow.

The changes in this PR can be tested by running the iOS end-to-end tests workflow https://github.com/mullvad/mullvadvpn-app/actions/workflows/ios-end-to-end-tests.yml and specifying the PR branch. The PR also introduces other workflows but unfortunately they won't show up in GitHub until they've been merged to main. It might be easiest to ask Niklas for a demo?

The ios/MullvadVPNUITests/README.md has been updated to reflect changes introduced in this PR and better describe the end to end tests, but all README changes are in a separate PR #6446


This change is Reviewable

@niklasberglund niklasberglund added the iOS Issues related to iOS label Jul 4, 2024
@niklasberglund niklasberglund changed the title Temporarily run tests on another device iOS end to end workflow changes related to iOS 17 and iPad Jul 4, 2024
@niklasberglund niklasberglund force-pushed the temporarily-run-tests-on-another-device branch 2 times, most recently from 84f90cd to abc5768 Compare July 4, 2024 16:03
@buggmagnet buggmagnet force-pushed the temporarily-run-tests-on-another-device branch 9 times, most recently from 9798586 to b341018 Compare July 29, 2024 11:31
@buggmagnet buggmagnet force-pushed the temporarily-run-tests-on-another-device branch from 1de01e9 to 1047d7d Compare July 29, 2024 12:19
Copy link
Collaborator

@pinkisemils pinkisemils left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 14 of 15 files at r1, 2 of 2 files at r6, all commit messages.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @niklasberglund)


.github/workflows/ios.yml line 59 at r6 (raw file):

        with:
          limit-access-to-actor: true
          detached: true

I don't think we need this in prod.


ios/MullvadVPNUITests/README.md line 51 at r6 (raw file):

* `MullvadVPNUITestsVerifyDNSSettingsChanged` - Verify custom DNS settings still changed
* `MullvadVPNUITestsChangeSettings` - Change all settings except custom DNS setting
* `MullvadVPNUITestsVerifySettingsChanged` - Verify all settings except custom DNS setting still changed

I need that trailing newline.


ios/MullvadVPNUITests/tests.json line 18 at r6 (raw file):

        ]
    }
}

This could also do with a newline at the end.

Copy link
Contributor

@buggmagnet buggmagnet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 1 of 15 files at r1.
Reviewable status: 14 of 16 files reviewed, 3 unresolved discussions (waiting on @niklasberglund and @pinkisemils)

@buggmagnet buggmagnet self-assigned this Jul 31, 2024
@buggmagnet buggmagnet force-pushed the temporarily-run-tests-on-another-device branch 3 times, most recently from 99cd75d to 372cf99 Compare July 31, 2024 11:25
@pinkisemils pinkisemils requested review from mojganii and acb-mv July 31, 2024 12:37
Copy link
Collaborator

@mojganii mojganii left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewable status: 1 of 16 files reviewed, 3 unresolved discussions (waiting on @buggmagnet, @niklasberglund, and @pinkisemils)

@buggmagnet buggmagnet force-pushed the temporarily-run-tests-on-another-device branch from 372cf99 to c60fbe6 Compare August 8, 2024 13:38
@buggmagnet buggmagnet dismissed pinkisemils’s stale review August 8, 2024 13:41

Reviewer is on vacation

@buggmagnet buggmagnet merged commit fce09af into main Aug 8, 2024
37 of 39 checks passed
@buggmagnet buggmagnet deleted the temporarily-run-tests-on-another-device branch August 8, 2024 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
iOS Issues related to iOS
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants