Skip to content

Commit

Permalink
build: use github actions for windows (electron#44136)
Browse files Browse the repository at this point in the history
* build: test windows runner

* build: try build windows on windows?

* build: take win/cross changes

* build: use bash as default shell always

* build: configure git for windows build tools

* build: bash as default

* build: configure windows correctly

* build: use sha1sum

* build: force windows cipd init and python3 existence

* just pain

* build: restore cache on windows

* build: use build-tools gclient

* build: sync gclient vars to build windows job

* build: output depshash for debugging

* build: past sam was a silly goose

* build: depshash logging

* build: force lf endings for lock and DEPS

* build: platform strings are hard

* build: checkout on windows host

* sup

* no check

* idk

* sigh

* ...

* no double checkout

* build: yolo some stuff

* build: run gn-check for windows on linux hosts for speed

* use container...

* cry ?

* build: e d

* e d

* no log

* fix toolchain on windows cross check

* build: use powershell to add mksnapshot_args

* build: enable x86 and arm64 windows builds too

* clean up

* maybe not needed

* build: keep action around for post step

* build: configure git global on win

* build: ia32 zip manifest

* build: no patch depot_tools for tests

* build: get arm64 windows closer to working

* build: windows tar is ass

* 32 bit on 32 bit

* maybe bash

* build: set up nodejs

* correct windows sharding

* fix some spec runner stuff

* fix windows tests

* overwrite -Force

* sigh

* screen res

* wat

* logs

* ... more logs

* line endings will be the death of me

* remove 1080p force thing

* vsctools + logging

* disable some fullscreen tests on GHA

* no progress

* run all CI

* install visual studio on arm64

* windows hax for non windows

* maybe arm sdk

* clean up depshash logic

* build: use single check per platform

* ensure clean args

* fix loop

* remove debug

* update default build image sha for dispatch

* plzzzz

* one more try

* arm64 vctools

* sad

* build: fix non-dispatch windows gn check

* chore: debug datadog-ci location

* chore: update build-tools for newer toolchain

* chore: set path for datadog-ci

* try this

* chore: fixup gn check

* fixup gn-check some more

* fixup windows gn check

* chore: fixup windows gn check

* test: use cmd for Windows testing

* fixup use cmd for testing on Windows

* fixup windows GN check

* fixup npm config arch for x86

* Can we set test files via powershell

* fixup to set test files via powershell

* fixup set test files via powershell

* Don't check cross instance cache disk space on Windows

* Use separate step to set env variables for testing

* fixup Use separate step to set env variables for testing

* fixup Use separate step to set env variables for testing

* fixup Use separate step to set env variables for testing (AGAIN)

* use powershell if in powershell

* fixup use powershell if in powershell

* chore: remove no longer needed changes to depot_tools

xref: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5669094
and https://chromium-review.googlesource.com/c/chromium/src/+/5844046

* chore: try using 7zip on Windows to extract tarball

* Revert "chore: try using 7zip on Windows to extract tarball"

This reverts commit c7432b6.

* test: debug failing tests on GHA windows

* fix: ftbfs when including simdjson in Node.js

(cherry picked from commit 48e44c4)

* chore: try to track down Windows testing hang

* use correct timeout

* try this

* see if this helps

* try to figure out why node is running

* shard tests to try to narrow down WOA lockup

* try to narrow down problem test

* Narrow down blocking test more

* do we need a combo to repro

* see if this cleans up the tests

* fixup navigator.usb test

* remove logging from problematic tests

* Revert "shard tests to try to narrow down WOA lockup"

This reverts commit a180658.

* remove logging

* debug keyboard test

* add timeout for Windows since arm64 sometimes hangs

* see if this helps

* put back original timeout

* try to use screenCapture to get screenshots of what is going on on WOA

* try using electron screencapture to debug WOA hang

* chore: turn off privacy experience

* run screenshot on both shards

* fixup screencap

* try to narrow down hanging spec

* chore: cleanup servers left open

* cleanup tests

* Revert "try to narrow down hanging spec"

This reverts commit a0f959f.

* cleanup test debugging

* fixup extensions spec

* cleanup unneeded items

* run wtf with 2 shards instead of 6

* Revert "run wtf with 2 shards instead of 6"

This reverts commit ca2d282.

* debug windows version on woa

* dump more info

* Get detailed CPU info

* revert debugging

* use same args as AppVeyor WOA for GHA WOA

* fixup use same args as AppVeyor WOA for GHA WOA

* fixup use same args as AppVeyor WOA for GHA WOA

* try to track down which tests trigger hang

* one or more of these combinations should hang

* break up web contents spec to find hang

* further break down api-web-contents to find hang

* test: ensure all webContents are closed

* test: fix require is not defined error

* see if api-web-contents spec is now good

* test: ensure all webContents are closed

* Revert "try to track down which tests trigger hang"

This reverts commit 07298d6.

* chore: use alternate location for windows toolchain

* Reapply "try to track down which tests trigger hang"

This reverts commit 0321f76.

* try to narrow down problem test

* fix TEST_SHARD env var

* no, really fix TEST_SHARD env var

* see if this fixes it

* test: cleanup any remaining windows and webcontents

* see if new cleanup helps

* dont destroy webcontents for now

* fixup dont destroy webcontents for now

* Only cleanup right before process.exit

* see if this fixes the hang

* actually destroy webcontents

* Revert "Reapply "try to track down which tests trigger hang""

This reverts commit cdee7de.

* see if this helps

* Revert "see if this helps"

This reverts commit 9a15a69.

* Is it all about the web contents?

* it is all about the webcontents

but which one?

* Narrow down problem webcontents test

* try to speed up git install on WOA

* disable problematic test on WOA

* remove debugging

* remove debugging from choco installs

* Revert "disable problematic test on WOA"

This reverts commit e060fb0.

* Revert "remove debugging"

This reverts commit f18dd8b.

* run against all the tests in the failing shard

* don't run visibility tests first

* remove debugging

* 3 is a magic number

* Revert "3 is a magic number"

This reverts commit 36b91cc.

* match what Appveyor runs exactly

* Revert "match what Appveyor runs exactly"

This reverts commit 7260dd4.

* chore: sort files alphabetically

* find out what spec is leaving stuff open

* chore: Checkout PR HEAD commit

 instead of merge commit

* try using app.exit instead of process.exit

* test: cleanup BrowserWindows and webContents

* Revert "chore: sort files alphabetically"

This reverts commit d9e217f.

* chore: use win32 to match process.platform

Needed for build-tools to download from PRs

* chore: cache yarn dir

* fixup cache yarn

* fixup use win32 to match process.platform

* fixup use win32 to match process.platform

* fixup cache yarn

* Add debugging for WOA hang

* Add debugging for failing keyboard lock test

* Revert "Add debugging for WOA hang"

This reverts commit 8df03d5.

* try using process.kill

* add more debugging to keyboard.lock test

* Revert "Add debugging for failing keyboard lock test"

* remove debugging

* test: disable keyboard.lock on Windows

* test: disable fullscreen tests on Windows

* test: only force test suite exit on WOA

* fixup test: only force test suite exit on WOA

* cleanup tests

* extract yarn caching/install to action

* try using bash to run windows tests

* remove left over debugging

* standardize on 'win' for Windows builds

* use 'x86' for arch for manifest files

* fixup try using bash to run windows tests

* fixup use 'x86' for arch for manifest files

* standardize on 'win' for Windows builds

* fixup use 'x86' for arch for manifest files

* fixup try using bash to run windows tests

---------

Co-authored-by: John Kleinschmidt <[email protected]>
Co-authored-by: Charles Kerr <[email protected]>
  • Loading branch information
3 people authored Dec 12, 2024
1 parent a5b4339 commit be1a3dc
Show file tree
Hide file tree
Showing 42 changed files with 657 additions and 328 deletions.
3 changes: 3 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
# `git apply` and friends don't understand CRLF, even on windows. Force those
# files to be checked out with LF endings even if core.autocrlf is true.
*.patch text eol=lf
DEPS text eol=lf
yarn.lock text eol=lf
script/zip_manifests/*.manifest text eol=lf
patches/**/.patches merge=union

# Source code and markdown files should always use LF as line ending.
Expand Down
18 changes: 12 additions & 6 deletions .github/actions/build-electron/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,10 @@ inputs:
description: 'Target arch'
required: true
target-platform:
description: 'Target platform'
description: 'Target platform, should be linux, win, macos'
required: true
artifact-platform:
description: 'Artifact platform, should be linux, darwin or mas'
description: 'Artifact platform, should be linux, win, darwin or mas'
required: true
step-suffix:
description: 'Suffix for build steps'
Expand Down Expand Up @@ -71,7 +71,7 @@ runs:
cd src
e build --target electron:electron_dist_zip -j $NUMBER_OF_NINJA_PROCESSES
if [ "${{ inputs.is-asan }}" != "true" ]; then
target_os=${{ inputs.target-platform == 'linux' && 'linux' || 'mac'}}
target_os=${{ inputs.target-platform == 'macos' && 'mac' || inputs.target-platform }}
if [ "${{ inputs.artifact-platform }}" = "mas" ]; then
target_os="${target_os}_mas"
fi
Expand All @@ -82,7 +82,7 @@ runs:
run: |
cd src
e build --target electron:electron_mksnapshot -j $NUMBER_OF_NINJA_PROCESSES
gn desc out/Default v8:run_mksnapshot_default args > out/Default/mksnapshot_args
ELECTRON_DEPOT_TOOLS_DISABLE_LOG=1 e d gn desc out/Default v8:run_mksnapshot_default args > out/Default/mksnapshot_args
# Remove unused args from mksnapshot_args
SEDOPTION="-i"
if [ "`uname`" = "Darwin" ]; then
Expand All @@ -91,7 +91,7 @@ runs:
sed $SEDOPTION '/.*builtins-pgo/d' out/Default/mksnapshot_args
sed $SEDOPTION '/--turbo-profiling-input/d' out/Default/mksnapshot_args
if [ "`uname`" = "Linux" ]; then
if [ "${{ inputs.target-platform }}" = "linux" ]; then
if [ "${{ inputs.target-arch }}" = "arm" ]; then
electron/script/strip-binaries.py --file $PWD/out/Default/clang_x86_v8_arm/mksnapshot
electron/script/strip-binaries.py --file $PWD/out/Default/clang_x86_v8_arm/v8_context_snapshot_generator
Expand All @@ -105,7 +105,13 @@ runs:
fi
e build --target electron:electron_mksnapshot_zip -j $NUMBER_OF_NINJA_PROCESSES
(cd out/Default; zip mksnapshot.zip mksnapshot_args gen/v8/embedded.S)
if [ "${{ inputs.target-platform }}" = "win" ]; then
cd out/Default
powershell Compress-Archive -update mksnapshot_args mksnapshot.zip
powershell Compress-Archive -update gen/v8/embedded.S mksnapshot.zip
else
(cd out/Default; zip mksnapshot.zip mksnapshot_args gen/v8/embedded.S)
fi
- name: Generate Cross-Arch Snapshot (arm/arm64) ${{ inputs.step-suffix }}
shell: bash
if: ${{ (inputs.target-arch == 'arm' || inputs.target-arch == 'arm64') && inputs.target-platform == 'linux' }}
Expand Down
70 changes: 41 additions & 29 deletions .github/actions/checkout/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@ inputs:
description: 'Whether to generate and persist a SAS token for the item in the cache'
required: false
default: 'false'
use-cache:
description: 'Whether to persist the cache to the shared drive'
required: false
default: 'true'
runs:
using: "composite"
steps:
Expand All @@ -13,31 +17,27 @@ runs:
run: |
echo "GIT_CACHE_PATH=$(pwd)/git-cache" >> $GITHUB_ENV
- name: Install Dependencies
shell: bash
run: |
cd src/electron
node script/yarn install --frozen-lockfile
uses: ./src/electron/.github/actions/install-dependencies
- name: Install Build Tools
uses: ./src/electron/.github/actions/install-build-tools
- name: Get Depot Tools
shell: bash
run: |
git clone --depth=1 https://chromium.googlesource.com/chromium/tools/depot_tools.git
sed -i '/ninjalog_uploader_wrapper.py/d' ./depot_tools/autoninja
# Remove swift-format dep from cipd on macOS until we send a patch upstream.
cd depot_tools
git apply --3way ../src/electron/.github/workflows/config/gclient.diff
if [[ ! -d depot_tools ]]; then
git clone --depth=1 https://chromium.googlesource.com/chromium/tools/depot_tools.git
# Ensure depot_tools does not update.
test -d depot_tools && cd depot_tools
touch .disable_auto_update
# Ensure depot_tools does not update.
test -d depot_tools && cd depot_tools
touch .disable_auto_update
fi
- name: Add Depot Tools to PATH
shell: bash
run: echo "$(pwd)/depot_tools" >> $GITHUB_PATH
- name: Generate DEPS Hash
shell: bash
run: |
node src/electron/script/generate-deps-hash.js && cat src/electron/.depshash-target
echo "DEPSHASH=v1-src-cache-$(shasum src/electron/.depshash | cut -f1 -d' ')" >> $GITHUB_ENV
node src/electron/script/generate-deps-hash.js
echo "DEPSHASH=v1-src-cache-$(cat src/electron/.depshash)" >> $GITHUB_ENV
- name: Generate SAS Key
if: ${{ inputs.generate-sas-token == 'true' }}
shell: bash
Expand All @@ -54,18 +54,23 @@ runs:
id: check-cache
shell: bash
run: |
cache_path=/mnt/cross-instance-cache/$DEPSHASH.tar
echo "Using cache key: $DEPSHASH"
echo "Checking for cache in: $cache_path"
if [ ! -f "$cache_path" ] || [ `du $cache_path | cut -f1` = "0" ]; then
if [[ "${{ inputs.use-cache }}" == "false" ]]; then
echo "Not using cache this time..."
echo "cache_exists=false" >> $GITHUB_OUTPUT
echo "Cache Does Not Exist for $DEPSHASH"
else
echo "cache_exists=true" >> $GITHUB_OUTPUT
echo "Cache Already Exists for $DEPSHASH, Skipping.."
cache_path=/mnt/cross-instance-cache/$DEPSHASH.tar
echo "Using cache key: $DEPSHASH"
echo "Checking for cache in: $cache_path"
if [ ! -f "$cache_path" ] || [ `du $cache_path | cut -f1` = "0" ]; then
echo "cache_exists=false" >> $GITHUB_OUTPUT
echo "Cache Does Not Exist for $DEPSHASH"
else
echo "cache_exists=true" >> $GITHUB_OUTPUT
echo "Cache Already Exists for $DEPSHASH, Skipping.."
fi
fi
- name: Check cross instance cache disk space
if: steps.check-cache.outputs.cache_exists == 'false'
if: steps.check-cache.outputs.cache_exists == 'false' && inputs.use-cache == 'true'
shell: bash
run: |
# if there is less than 20 GB free space then creating the cache might fail so exit early
Expand All @@ -81,13 +86,17 @@ runs:
if: steps.check-cache.outputs.cache_exists == 'false'
shell: bash
run: |
gclient config \
e d gclient config \
--name "src/electron" \
--unmanaged \
${GCLIENT_EXTRA_ARGS} \
"$GITHUB_SERVER_URL/$GITHUB_REPOSITORY"
ELECTRON_USE_THREE_WAY_MERGE_FOR_PATCHES=1 gclient sync --with_branch_heads --with_tags -vvvvv
if [ "$TARGET_OS" != "" ]; then
echo "target_os=['$TARGET_OS']" >> ./.gclient
fi
ELECTRON_USE_THREE_WAY_MERGE_FOR_PATCHES=1 e d gclient sync --with_branch_heads --with_tags -vv
if [ "${{ inputs.is-release }}" != "true" && -n "${{ env.PATCH_UP_APP_CREDS }}" ]; then
# Re-export all the patches to check if there were changes.
python3 src/electron/script/export_all_patches.py src/electron/patches/config.json
Expand Down Expand Up @@ -128,13 +137,13 @@ runs:
# https://dawn-review.googlesource.com/c/dawn/+/83901
# TODO: maybe better to always leave out */.git/HEAD file for all targets ?
- name: Delete .git directories under src to free space
if: steps.check-cache.outputs.cache_exists == 'false'
if: ${{ steps.check-cache.outputs.cache_exists == 'false' && inputs.use-cache == 'true' }}
shell: bash
run: |
cd src
( find . -type d -name ".git" -not -path "./third_party/angle/*" -not -path "./third_party/dawn/*" -not -path "./electron/*" ) | xargs rm -rf
- name: Minimize Cache Size for Upload
if: steps.check-cache.outputs.cache_exists == 'false'
if: ${{ steps.check-cache.outputs.cache_exists == 'false' && inputs.use-cache == 'true' }}
shell: bash
run: |
rm -rf src/android_webview
Expand All @@ -145,17 +154,20 @@ runs:
rm -rf src/third_party/angle/third_party/VK-GL-CTS/src
rm -rf src/third_party/swift-toolchain
rm -rf src/third_party/swiftshader/tests/regres/testlists
cp src/electron/.github/actions/checkout/action.yml ./
rm -rf src/electron
mkdir -p src/electron/.github/actions/checkout
mv action.yml src/electron/.github/actions/checkout
- name: Compress Src Directory
if: steps.check-cache.outputs.cache_exists == 'false'
if: ${{ steps.check-cache.outputs.cache_exists == 'false' && inputs.use-cache == 'true' }}
shell: bash
run: |
echo "Uncompressed src size: $(du -sh src | cut -f1 -d' ')"
tar -cf $DEPSHASH.tar src
echo "Compressed src to $(du -sh $DEPSHASH.tar | cut -f1 -d' ')"
cp ./$DEPSHASH.tar /mnt/cross-instance-cache/
- name: Persist Src Cache
if: steps.check-cache.outputs.cache_exists == 'false'
if: ${{ steps.check-cache.outputs.cache_exists == 'false' && inputs.use-cache == 'true' }}
shell: bash
run: |
final_cache_path=/mnt/cross-instance-cache/$DEPSHASH.tar
Expand Down
11 changes: 10 additions & 1 deletion .github/actions/install-build-tools/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,15 @@ runs:
- name: Install Build Tools
shell: bash
run: |
export BUILD_TOOLS_SHA=eeb1a11392e4cec08fd926c93b31ab556dc0c23b
if [ "$(expr substr $(uname -s) 1 10)" == "MSYS_NT-10" ]; then
git config --global core.filemode false
git config --global core.autocrlf false
git config --global branch.autosetuprebase always
fi
export BUILD_TOOLS_SHA=bf2a839205d569be99e0b23ede5d8a0d5041a777
npm i -g @electron/build-tools
e auto-update disable
if [ "$(expr substr $(uname -s) 1 10)" == "MSYS_NT-10" ]; then
e d cipd.bat --version
cp "C:\Python37\python.exe" "C:\Python37\python3.exe"
fi
21 changes: 21 additions & 0 deletions .github/actions/install-dependencies/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
name: 'Install Dependencies'
description: 'Installs yarn depdencies using cache when available'
runs:
using: "composite"
steps:
- name: Get yarn cache directory path
shell: bash
id: yarn-cache-dir-path
run: echo "dir=$(node src/electron/script/yarn cache dir)" >> $GITHUB_OUTPUT
- uses: actions/cache@0c45773b623bea8c8e75f6c82b208c3cf94ea4f9
id: yarn-cache
with:
path: ${{ steps.yarn-cache-dir-path.outputs.dir }}
key: ${{ runner.os }}-yarn-${{ hashFiles('src/electron/yarn.lock') }}
restore-keys: |
${{ runner.os }}-yarn-
- name: Install Dependencies
shell: bash
run: |
cd src/electron
node script/yarn install --frozen-lockfile --prefer-offline
Loading

0 comments on commit be1a3dc

Please sign in to comment.