Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Rel-Eng]: Adds script to automatically generate GHCup metadata for Cabal releases #9411

Merged
merged 6 commits into from
Dec 16, 2023
Merged

[Rel-Eng]: Adds script to automatically generate GHCup metadata for Cabal releases #9411

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
9acc813
[Rel-Eng]: Add initial version of the release metadata script
arjunkathuria Nov 6, 2023
15d669d
[Rel-Eng]: Change top shebang from '/bin/bash' to '/usr/bin/env bash'
arjunkathuria Nov 11, 2023
cc3a82c
[Rel-Eng]: Add script to download cabal-install binaries for a release
arjunkathuria Nov 11, 2023
d7a7871
[Rel-Eng]: Integrate the release binary downloading script into GHCup…
arjunkathuria Nov 11, 2023
2499ce0
[Rel-Eng]: Add release-scripts binary-downloads to gitignore
arjunkathuria Nov 11, 2023
d5c49d4
[Rel-Eng]: Change from '/bin/bash' to '/usr/bin/env bash' in the down…
arjunkathuria Dec 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,3 +88,8 @@ bench.html

# I'm unsure how to ignore these generated golden files
cabal-testsuite/PackageTests/NewUpdate/RejectFutureIndexStates/cabal.out

## Release Scripts

# ignore the downloaded binary files
scripts/release/binary-downloads/
125 changes: 125 additions & 0 deletions scripts/release/create-release-metadata-for-ghcup.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
#!/usr/bin/env bash

# This script, when passed the cabal release number as the first and only argument
# generates the metadata in the correct format to be useable as is by GHCup
# for eg:-
# $ create-release-metadata-for-ghcup.sh 3.10.2.0 or "3.10.2.0"

# Note:- Please run ./download-cabal-install-release-binaries.sh before running this script.
set -eu
set -o pipefail

RELEASE=$1
## FixMe:// What dir to use here?

if [ -d "binary-downloads/cabal-install-${RELEASE}-binaries" ]; then
echo "binary downloads folder for release ${RELEASE} found, starting generating GHCup metadata..."
else
echo "The binary downloads for release ${RELEASE} not found."
echo "Please run the script to download them first."
fi

cd "binary-downloads/cabal-install-${RELEASE}-binaries"

cat <<EOF > /dev/stdout
$RELEASE:
viTags:
- Latest
viChangeLog: https://github.com/haskell/cabal/blob/master/release-notes/cabal-install-$RELEASE.md
viPostInstall: *cabal-${RELEASE//./}-post-install
viArch:
A_64:
Linux_UnknownLinux:
unknown_versioning: &cabal-${RELEASE//./}-64
dlUri: https://downloads.haskell.org/~cabal/cabal-install-$RELEASE/cabal-install-$RELEASE-x86_64-linux-alpine3_12.tar.xz
dlSubdir: cabal-install-$RELEASE
dlHash: $(sha256sum "cabal-install-$RELEASE-x86_64-linux-alpine3_12.tar.xz" | awk '{ print $1 }')
Linux_Alpine:
unknown_versioning: &cabal-${RELEASE//./}-64
Linux_CentOS:
unknown_versioning: &cabal-${RELEASE//./}-64-centos7
dlUri: https://downloads.haskell.org/~cabal/cabal-install-$RELEASE/cabal-install-$RELEASE-x86_64-linux-centos7.tar.xz
dlSubdir: cabal-install-$RELEASE
dlHash: $(sha256sum "cabal-install-$RELEASE-x86_64-linux-centos7.tar.xz" | awk '{ print $1 }')
Linux_Debian:
' ( >= 9 && < 10)': &cabal-${RELEASE//./}-64-debian
dlUri: https://downloads.haskell.org/~cabal/cabal-install-$RELEASE/cabal-install-$RELEASE-x86_64-linux-deb9.tar.xz
dlSubdir: cabal-install-$RELEASE
dlHash: $(sha256sum "cabal-install-$RELEASE-x86_64-linux-deb9.tar.xz" | awk '{ print $1 }')
' ( == 10 && < 11)':
dlUri: https://downloads.haskell.org/~cabal/cabal-install-$RELEASE/cabal-install-$RELEASE-x86_64-linux-deb10.tar.xz
dlSubdir: cabal-install-$RELEASE
dlHash: $(sha256sum "cabal-install-$RELEASE-x86_64-linux-deb10.tar.xz" | awk '{ print $1 }')
' ( >= 11)':
dlUri: https://downloads.haskell.org/~cabal/cabal-install-$RELEASE/cabal-install-$RELEASE-x86_64-linux-deb11.tar.xz
dlSubdir: cabal-install-$RELEASE
dlHash: $(sha256sum "cabal-install-$RELEASE-x86_64-linux-deb11.tar.xz" | awk '{ print $1 }')
unknown_versioning: &cabal-${RELEASE//./}-64-debian
Linux_Fedora:
'>= 33':
dlUri: https://downloads.haskell.org/~cabal/cabal-install-$RELEASE/cabal-install-$RELEASE-x86_64-linux-fedora33.tar.xz
dlSubdir: cabal-install-$RELEASE
dlHash: $(sha256sum "cabal-install-$RELEASE-x86_64-linux-fedora33.tar.xz" | awk '{ print $1 }')
unknown_versioning: &cabal-${RELEASE//./}-64-centos7
Linux_Ubuntu:
'< 20': &cabal-${RELEASE//./}-64-ubuntu18
dlUri: https://downloads.haskell.org/~cabal/cabal-install-$RELEASE/cabal-install-$RELEASE-x86_64-linux-ubuntu18_04.tar.xz
dlSubdir: cabal-install-$RELEASE
dlHash: $(sha256sum "cabal-install-$RELEASE-x86_64-linux-ubuntu18_04.tar.xz" | awk '{ print $1 }')
'>= 20': &cabal-${RELEASE//./}-64-ubuntu20
dlUri: https://downloads.haskell.org/~cabal/cabal-install-$RELEASE/cabal-install-$RELEASE-x86_64-linux-ubuntu20_04.tar.xz
dlSubdir: cabal-install-$RELEASE
dlHash: $(sha256sum "cabal-install-$RELEASE-x86_64-linux-ubuntu20_04.tar.xz" | awk '{ print $1 }')
unknown_versioning: *cabal-${RELEASE//./}-64-ubuntu18
Linux_Mint:
'< 20': *cabal-${RELEASE//./}-64-ubuntu18
'>= 20': *cabal-${RELEASE//./}-64-ubuntu20
unknown_versioning: *cabal-${RELEASE//./}-64-ubuntu18
Darwin:
unknown_versioning:
dlUri: https://downloads.haskell.org/~cabal/cabal-install-$RELEASE/cabal-install-$RELEASE-x86_64-darwin.tar.xz
dlSubdir: cabal-install-$RELEASE
dlHash: $(sha256sum "cabal-install-$RELEASE-x86_64-darwin.tar.xz" | awk '{ print $1 }')
Windows:
unknown_versioning:
dlUri: https://downloads.haskell.org/~cabal/cabal-install-$RELEASE/cabal-install-$RELEASE-x86_64-windows.zip
dlSubdir: cabal-install-$RELEASE
dlHash: $(sha256sum "cabal-install-$RELEASE-x86_64-windows.zip" | awk '{ print $1 }')
FreeBSD:
unknown_versioning:
dlUri: https://downloads.haskell.org/~cabal/cabal-install-$RELEASE/cabal-install-$RELEASE-x86_64-freebsd.tar.xz
dlSubdir: cabal-install-$RELEASE
dlHash: $(sha256sum "cabal-install-$RELEASE-x86_64-freebsd.tar.xz" | awk '{ print $1 }')
A_32:
Linux_UnknownLinux:
unknown_versioning: &cabal-${RELEASE//./}-32
dlUri: https://downloads.haskell.org/~cabal/cabal-install-$RELEASE/cabal-install-$RELEASE-i386-linux-alpine3_12.tar.xz
dlSubdir: cabal-install-$RELEASE
dlHash: $(sha256sum "cabal-install-$RELEASE-i386-linux-alpine3_12.tar.xz" | awk '{ print $1 }')
Linux_Alpine:
unknown_versioning: *cabal-${RELEASE//./}-32
Linux_Debian:
'( >= 9 )':
dlUri: https://downloads.haskell.org/~cabal/cabal-install-$RELEASE/cabal-install-$RELEASE-i386-linux-deb9.tar.xz
dlSubdir: cabal-install-$RELEASE
dlHash: $(sha256sum "cabal-install-$RELEASE-i386-linux-deb9.tar.xz" | awk '{ print $1 }')
unknown_versioning: *cabal-${RELEASE//./}-32
A_ARM64:
Darwin:
unknown_versioning:
dlUri: https://downloads.haskell.org/~cabal/cabal-install-$RELEASE/cabal-install-$RELEASE-aarch64-darwin.tar.xz
dlSubdir: cabal-install-$RELEASE
dlHash: $(sha256sum "cabal-install-$RELEASE-aarch64-darwin.tar.xz" | awk '{ print $1 }')
Linux_Debian:
'( >= 10 && < 11)': &cabal-31020-arm64
dlUri: https://downloads.haskell.org/~cabal/cabal-install-$RELEASE/cabal-install-$RELEASE-aarch64-linux-deb10.tar.xz
dlSubdir: cabal-install-$RELEASE
dlHash: $(sha256sum "cabal-install-$RELEASE-aarch64-linux-deb10.tar.xz" | awk '{ print $1 }')
'( >= 11)':
dlUri: https://downloads.haskell.org/~cabal/cabal-install-$RELEASE/cabal-install-$RELEASE-aarch64-linux-deb11.tar.xz
dlSubdir: cabal-install-$RELEASE
dlHash: $(sha256sum "cabal-install-$RELEASE-aarch64-linux-deb11.tar.xz" | awk '{ print $1 }')
unknown_versioning: *cabal-${RELEASE//./}-arm64
Linux_UnknownLinux:
unknown_versioning: *cabal-${RELEASE//./}-arm64
EOF
42 changes: 42 additions & 0 deletions scripts/release/download-cabal-install-release-binaries.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
#!/usr/bin/env bash

# A script to download the release binary files for a given cabal release
# from upstream "downlods.haskell.org".
# It accepts the first and only argument as the release number.
#
# useage:-
# $ download-cabal-install-release-binaries.sh "3.10.1.0"
#
# This was initally made to be used with ./create-release-metadata-for-ghcup.sh

set -eu
set -o pipefail

RELEASE=$1

echo "RELEASE: $RELEASE"

for com in wget sha256sum ; do
command -V ${com} >/dev/null 2>&1
done

[ ! -d "binary-downloads/cabal-install-${RELEASE}-binaries" ]

mkdir -p "binary-downloads/cabal-install-${RELEASE}-binaries"

cd "binary-downloads/cabal-install-${RELEASE}-binaries"

## Download release files
echo "Downloading form: \"https://downloads.haskell.org/~cabal/cabal-install-${RELEASE}/\""
wget --no-parent -r --reject "index.html*" --no-directories "https://downloads.haskell.org/~cabal/cabal-install-${RELEASE}/"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the wrong way around and would mean we download twice: once manually and once after we uploaded (why?).

The artifacts are built on gitlab CI. The release manager will have to download them to their local machine in order to upload them to downloads.haskell.org. And they want to perform the yaml step after they downloaded from gitlab, not after they uploaded to the download server.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. Since we don't seem to do github binary releases for cabal-install, i suppose i'd have to download them from upstream links like https://downloads.haskell.org/~cabal/cabal-install-3.10.1.0/ ?

could've mentioned here when i asked to confirm.

What's the correct url here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chreekat might know

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🕯️ * summons @chreekat * 🕯️

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👻

Copy link
Contributor Author

@arjunkathuria arjunkathuria Nov 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think that this is better suited as its own issue/PR pair?
The scope of this particular PR was to make a script that generates metadata for GHCup for a cabal release.

I think we can iterate on the how-to and where-to download parts that make releases easier as its own rel-eng issue as it would evolve into its own discussion/interface requirements.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're asking for the URL to download artifacts from gitlab.

That's why I suggest talking to a past/current Cabal release manager. I imagine they already have a workflow for this. While I can point to release pipelines and explain in general how to find the artifacts, it would be better to know what the actual current process is. Maybe @Kleidukos can help.

Do you think that this is better suited as its own issue/PR pair?
The scope of this particular PR was to make a script that generates metadata for GHCup for a cabal release.

Yeah, I totally agree with that.

To actually automate it, it would be a lot easier to run the script in the GitLab pipeline that generates the bindists in the first place. That's what I can help with. https://github.com/haskell/cabal/blob/master/.gitlab-ci.yml is the entrypoint.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the best course of action is to purge gitlab CI.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may want to re-purpose the tools of the GHC release engineering team:

Copy link
Contributor Author

@arjunkathuria arjunkathuria Nov 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, they do it in a couple of steps.

  1. get the job ids from a pipeline id.
  2. try to get artifacts from these job ids and error if anyone of those failed.
  3. If successful, generate metadata from these artifacts.

step 3. is mostly done here.

i suppose i can try repurposing the logic in steps 1. and 2. from their python scripts in the shell script here.
If that's not required or if that needs to be a separate issue/PR pair or a broader discussion, please do let me know.


## Verify that sha256 sums of downloaded files match the ones mentioned in the upstream SHA256SUMS file
echo "verifying checksums for downloaded files..."

if sha256sum --check ./SHA256SUMS; then
echo "All checksums match!"
echo "Successfully downloaded binaries for release: ${RELEASE}"
else
echo "checksums of downloaded files do no match the ones listed in upstream SHA256SUMS file."
echo "please try deleting \"binary-downloads/cabal-install-${RELEASE}-binaries\" folder and downloading again."
fi
Loading