diff --git a/README.md b/README.md index e375b0a..82560ee 100644 --- a/README.md +++ b/README.md @@ -97,39 +97,38 @@ we've decided to leverage the advantages offered by revision control. As such, we've adopted a well defined branching model for each third-party package. First of all, we have a Delphix repository on github for each third-party -package that we build. Each repository has at least 2 branches: **master** and -**upstreams/master**. The **master** branch of the package is the one we build, -and contains Delphix changes. The **upstreams/master** branch is used to track +package that we build. Each repository has at least 2 branches: **6.0/stage** and +**upstreams/6.0/stage**. The **6.0/stage** branch of the package is the one we build, +and contains Delphix changes. The **upstreams/6.0/stage** branch is used to track the upstream version of the package. For packages that are not provided by -Ubuntu but are available on git, the **upstreams/master** branch usually just -tracks the **master** branch of the project. For packages that are provided by -Ubuntu, the **upstreams/master** branch instead tracks the source package that +Ubuntu but are available on git, the **upstreams/6.0/stage** branch usually just +tracks the **6.0/stage** branch of the project. For packages that are provided by +Ubuntu, the **upstreams/6.0/stage** branch instead tracks the source package that is maintained by Ubuntu (i.e. the branch contains the files obtained from `apt-get source `). This offers the advantage of using a version of the package tuned to work with our Ubuntu distribution. -When updating a package, we first check if the **upstreams/master** branch is +When updating a package, we first check if the **upstreams/6.0/stage** branch is up-to-date, by fetching the latest version of the upstream git repository or the Ubuntu source package. If changes are detected, we update the -**upstreams/master** branch and push the changes to GitHub. +**upstreams/6.0/stage** branch and push the changes to GitHub. -The second step is to check if the **master** branch is up-to-date with -**upstreams/master**. If it is already up-to-date, then we are done. If not, -then we attempt merging **upstreams/master** into **master**. +The second step is to check if the **6.0/stage** branch is up-to-date with +**upstreams/6.0/stage**. If it is already up-to-date, then we are done. If not, +then we attempt merging **upstreams/6.0/stage** into **6.0/stage**. If the merge is successful, then we push the changes to a staging branch on -GitHub, called **projects/auto-update/master/merging**. The intent is for +GitHub, called **projects/auto-update/6.0/stage/merging**. The intent is for a different system to fetch those changes, build them, and then launch tests. See [Scripts > sync-with-upstream.sh](#sync-with-upstreamsh) below. Once the merge has been tested, [Scripts > push-merge.sh](#push-mergesh) is -called on the original VM to push the changes to the **master** branch on +called on the original VM to push the changes to the **6.0/stage** branch on GitHub. -Note that the example above targets the **master** branch, but the same -workflow could apply to other branches, like **6.0/stage**, although it is -not currently in use. +Note that the example above targets the **6.0/stage** branch, but the same +workflow could apply to other branches. ## Scripts @@ -172,8 +171,8 @@ Usage: ``` This checks if a package has updates in the upstream project that haven't been -pulled into the **upstreams/master** branch, or if the **upstreams/master** -branch has commits that haven't been merged into the **master** branch. +pulled into the **upstreams/6.0/stage** branch, or if the **upstreams/6.0/stage** +branch has commits that haven't been merged into the **6.0/stage** branch. If updates are available, the file `/update-available` will be created. @@ -189,19 +188,19 @@ Usage: This script has 2 tasks: 1. Check if the upstream project has updates that are not pulled into the -**upstreams/master** branch of the package, and if so then update that branch +**upstreams/6.0/stage** branch of the package, and if so then update that branch and push changes to GitHub. -2. Merge **upstreams/master** into **master** and push the changes to a staging -branch on GitHub, called **projects/auto-update/master/merging**. Another +2. Merge **upstreams/6.0/stage** into **6.0/stage** and push the changes to a staging +branch on GitHub, called **projects/auto-update/6.0/stage/merging**. Another system should use that branch to build the package, and then run the appropriate integration tests. After testing has been completed, `push-merge.sh ` should be called on -the same system to push the merge to the **master** branch. +the same system to push the merge to the **6.0/stage** branch. Note that the DRYRUN environment variable must be set when running this script. If DRYRUN is set to "true", then changes are not pushed to GitHub in step 1, -and staged changes are pushed to **projects/auto-update/master/merging-dryrun** +and staged changes are pushed to **projects/auto-update/6.0/stage/merging-dryrun** in step 2 instead of the non-dryrun branch. The intention is that when testing changes to the logic we want to be able to run most of the logic, but without affecting the production branches. @@ -215,8 +214,8 @@ Usage: This must be called on a system that has previously called `sync-with-upstream.sh` for the same package. It will push the merge that was -previously prepared by `sync-with-upstream.sh` to the production **master** -branch, after checking that the **master** branch hasn't been modified since +previously prepared by `sync-with-upstream.sh` to the production **6.0/stage** +branch, after checking that the **6.0/stage** branch hasn't been modified since `sync-with-upstream.sh` was called. Like for `sync-with-upstream.sh`, the DRYRUN environment variable must be set @@ -245,10 +244,10 @@ of some of the scripts defined above. revision defined. If not set, it will be auto-generated from the timestamp. Applies to [buildpkg.sh](#buildpkgsh). -* **DEFAULT_GIT_BRANCH**: The product branch that is being built or updated is - typically stored in the file `branch.config`, however it can be overridden via - DEFAULT_GIT_BRANCH. It can either be set to a development branch, such as - "master" or "6.0/stage", or a release tag, such as "release/6.0.6.0". +* **DEFAULT_GIT_BRANCH**: The product branch that is being built or updated + and must be set via DEFAULT_GIT_BRANCH. It can either be set to a + development branch, such as "6.0/stage", or a release tag, such as + "release/6.0.6.0". The product branch is used in multiple instances. When running [setup.sh](#setupsh), it will determine what linux-package-mirror link to use when fetching packages from apt (although those links can be @@ -383,7 +382,7 @@ The `fetch()` hook is optional, as a default is provided and should be used. It is called when fetching the source code of the package to build or to update. The repository is cloned into `/repo` and checked out as branch **repo-HEAD**. If we are performing a package update, then we also -fetch the **upstreams/master** branch into **upstream-HEAD**. The default +fetch the **upstreams/6.0/stage** branch into **upstream-HEAD**. The default should only be overridden when not fetching the package source from git. #### Prepare (hook) @@ -402,7 +401,7 @@ The `update_upstream()` hook should only be defined for third party packages that can be auto-updated. It is responsible for fetching the latest upstream source code on top of branch **upstream-HEAD** of our fetched repository in `/repo`. Note that any changes should be rebased on top of -the **upstreams/master** branch. If changes are detected, file +the **upstreams/6.0/stage** branch. If changes are detected, file `/upstream-updated` should be created. #### Merge With Upstream (hook) @@ -552,8 +551,8 @@ UPSTREAM_GIT_BRANCH="" Note that steps 3 to 5 are most useful when getting a third party package from an Ubuntu source package. When the third party package is fetched from git, -you may simply fork the upstream repository and add an **upstreams/master** -branch that points to the **master** branch; you can then update +you may simply fork the upstream repository and add an **upstreams/6.0/stage** +branch that points to the **6.0/stage** branch; you can then update `DEFAULT_PACKAGE_GIT_URL` in config.sh to your forked git repository and skip to step 6. @@ -591,8 +590,8 @@ DEFAULT_PACKAGE_GIT_URL="https://github.com//" #### Step 5. Push to your developer repository Next step is to push the upstream code to the newly created repository to your -developer repository. You should push the initial commit to both the **master** -branch and the **upstreams/master** branch. +developer repository. You should push the initial commit to both the **6.0/stage** +branch and the **upstreams/6.0/stage** branch. #### Step 6. Build the package @@ -618,7 +617,7 @@ build the package. For a package that doesn't have a `debian` metadata directory already defined in its source tree, you'll need to create it, and push the changes to the -**master** branch of your developer repository. See +**6.0/stage** branch of your developer repository. See [Common Steps > Creating debian metadirectory](#creating-debian-metadirectory) for more details. @@ -729,9 +728,9 @@ is to create an official repository for it. [Delphix Open-Source Policy](https://docs.delphix.com/en/ip-strategy/outbound-open-source) if you haven't already, and provide the necessary info so that a `github.com/delphix/` repository can be created for it. You'll need - to push the **master** branch from your developer repository, as well as the - **upstreams/master** branch if it is a third-party package. Note that if you - have modified **master** (i.e. it diverges from **upstreams/master**), you + to push the **6.0/stage** branch from your developer repository, as well as the + **upstreams/6.0/stage** branch if it is a third-party package. Note that if you + have modified **6.0/stage** (i.e. it diverges from **upstreams/6.0/stage**), you should submit your changes for review before pushing them. 1. If this is a third-party package that is to be auto-updated by Delphix diff --git a/branch.config b/branch.config deleted file mode 100644 index 01007e6..0000000 --- a/branch.config +++ /dev/null @@ -1,14 +0,0 @@ -# -# Copyright (c) 2019 by Delphix. All rights reserved. -# - -# -# The "BRANCH" parameter tracks the upstream branch of linux-pkg. It is -# used to determine which branch of the linux package mirror will be used for -# the build if DEFAULT_GIT_BRANCH is not set. DEFAULT_GIT_BRANCH is set when -# linux pkg is built by the linux-pkg-build Jenkins jobs. The -# DEFAULT_GIT_BRANCH parameter should be updated by the release lead on -# branching. -# - -DEFAULT_GIT_BRANCH="6.0/patch" diff --git a/buildpkg.sh b/buildpkg.sh index 010b049..9d4cda3 100755 --- a/buildpkg.sh +++ b/buildpkg.sh @@ -67,13 +67,13 @@ shift $((OPTIND - 1)) PACKAGE=$1 logmust check_package_exists "$PACKAGE" +check_env DEFAULT_GIT_BRANCH # # DEFAULT_REVISION & DEFAULT_GIT_BRANCH will be set if called from buildlist.sh. # If the script is called manually, we set it here. # DEFAULT_REVISION="${DEFAULT_REVISION:-$(default_revision)}" -logmust determine_default_git_branch echo "" echo_bold "====================================================================" diff --git a/checkupdates.sh b/checkupdates.sh index 1ccf18a..4d1904e 100755 --- a/checkupdates.sh +++ b/checkupdates.sh @@ -53,7 +53,6 @@ logmust check_package_exists "$PACKAGE" # If the script is called manually, we set it here. # DEFAULT_REVISION="${DEFAULT_REVISION:-$(default_revision)}" -logmust determine_default_git_branch logmust load_package_config "$PACKAGE" logmust create_workdir diff --git a/default-package-config.sh b/default-package-config.sh index 9b6035e..bcd04a9 100644 --- a/default-package-config.sh +++ b/default-package-config.sh @@ -345,6 +345,7 @@ function kernel_update_upstream() { local local_upstream_commit local_upstream_commit=$(git rev-parse upstream-HEAD) [[ -z "${local_upstream_commit}" ]] && die "could not find upstream-HEAD's commit" + check_env DEFAULT_GIT_BRANCH echo "note: upstreams/${DEFAULT_GIT_BRANCH} commit: ${local_upstream_commit}" if [[ "${upstream_tag_commit}" == "${local_upstream_commit}" ]]; then diff --git a/lib/common.sh b/lib/common.sh index 4599fd2..a92c1ec 100644 --- a/lib/common.sh +++ b/lib/common.sh @@ -144,32 +144,6 @@ function run_setup_if_needed() { echo_bold "------------------------------------------------------------" } -# -# Determine DEFAULT_GIT_BRANCH. If it is unset, default to the branch set in -# branch.config. -# -function determine_default_git_branch() { - - [[ -n "$DEFAULT_GIT_BRANCH" ]] && return - - echo "DEFAULT_GIT_BRANCH is not set." - if ! source "$TOP/branch.config" 2>/dev/null; then - die "No branch.config file found in repo root." - fi - - if [[ -z "$DEFAULT_GIT_BRANCH" ]]; then - die "$DEFAULT_GIT_BRANCH parameter was not sourced" \ - "from branch.config. Ensure branch.config is" \ - "properly formatted with e.g." \ - "DEFAULT_GIT_BRANCH=''" - fi - - echo "Defaulting DEFAULT_GIT_BRANCH to branch" \ - "$DEFAULT_GIT_BRANCH set in branch.config." - - export DEFAULT_GIT_BRANCH -} - function is_release_branch() { check_env DEFAULT_GIT_BRANCH [[ "$DEFAULT_GIT_BRANCH" == release/* ]] diff --git a/package-lists/build/main.pkgs b/package-lists/build/main.pkgs index 4b7cce6..d183be1 100644 --- a/package-lists/build/main.pkgs +++ b/package-lists/build/main.pkgs @@ -23,6 +23,7 @@ libkdumpfile make-jpkg makedumpfile masking +misc-debs nfs-utils performance-diagnostics ptools diff --git a/packages/adoptopenjdk/config.sh b/packages/adoptopenjdk/config.sh index eb67c63..e18a105 100755 --- a/packages/adoptopenjdk/config.sh +++ b/packages/adoptopenjdk/config.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # -# Copyright 2018, 2022 Delphix +# Copyright 2018, 2023 Delphix # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -21,13 +21,13 @@ PACKAGE_DEPENDENCIES="make-jpkg" case $(dpkg-architecture -q DEB_HOST_ARCH 2>/dev/null || echo "none") in amd64) - _tarfile="OpenJDK8U-jdk_x64_linux_hotspot_8u345b01.tar.gz" - _tarfile_sha256="ed6c9db3719895584fb1fd69fc79c29240977675f26631911c5a1dbce07b7d58" + _tarfile="OpenJDK8U-jdk_x64_linux_hotspot_8u362b09.tar.gz" + _tarfile_sha256="1486a792fb224611ce0cd0e83d4aacd3503b56698549f8e9a9f0a6ebb83bdba1" _jdk_path="/usr/lib/jvm/adoptopenjdk-java8-jdk-amd64" ;; arm64) - _tarfile="OpenJDK8U-jdk_aarch64_linux_hotspot_8u345b01.tar.gz" - _tarfile_sha256="c1965fb24dded7d7944e2da36cd902adf3b7b1d327aaa21ea507cff00a5a0090" + _tarfile="OpenJDK8U-jdk_aarch64_linux_hotspot_8u362b09.tar.gz" + _tarfile_sha256="9290a8beefd7a94f0eb030f62d402411a852100482b9c5b63714bacc57002c2a" _jdk_path="/usr/lib/jvm/adoptopenjdk-java8-jdk-arm64" ;; *) ;; diff --git a/packages/misc-debs/config.sh b/packages/misc-debs/config.sh new file mode 100644 index 0000000..9ad948b --- /dev/null +++ b/packages/misc-debs/config.sh @@ -0,0 +1,73 @@ +#!/usr/bin/env bash +# +# Copyright 2021, 2023 Delphix +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# shellcheck disable=SC2034 + +DEFAULT_PACKAGE_GIT_URL=none +SKIP_COPYRIGHTS_CHECK=true + +# +# IMPORTANT NOTE +# -------------- +# +# Debian packages (debs) that are not built from source by linux-pkg can be +# added to this "meta-package". As a general rule, pre-built debs should only +# be added here when they have been fetched from a trusted third-party +# package archive. +# +# Here are some valid reasons for adding new debs here: +# - There are bugs with a recent version of a package provided by Ubuntu and +# we want to pin an older version of that package. +# - Ubuntu provides a version of a package that is too old, and the package's +# maintainers provide a more recent version of the package. Note that in this +# case, you may also look into adding the maintainer's archive to the +# linux-package-mirror PPAs list. +# +# To add a new deb here, upload that deb to the linux-pkg/misc-debs directory +# in artifcatory and note the deb's SHA256. Be explicit on where this deb +# was fetched from and why it was added to this list. +# +# When removing debs from this list, you should not remove them from artifactory +# as they would used when rebuilding older releases. +# + +function fetch() { + logmust cd "$WORKDIR/artifacts" + + local debs=( + "tar_1.30+dfsg-7ubuntu0.20.04.3_amd64.deb a3b1212570b10c192b3d49cffb68f909146d4b72a7104f213f4426c7e41d0e49" + ) + + local url="http://artifactory.delphix.com/artifactory/linux-pkg/misc-debs" + + echo "Fetched debs:" >BUILD_INFO + local entry + for entry in "${debs[@]}"; do + local deb sha256 + deb=$(echo "$entry" | awk '{print $1}') + sha256=$(echo "$entry" | awk '{print $2}') + [[ -n "$deb" && -n "$sha256" ]] || die "Invalid entry '$entry'" + + logmust fetch_file_from_artifactory "$url/$deb" "$sha256" + + echo "$entry" >>BUILD_INFO + done +} + +function build() { + return + # Nothing to do, all the logic is done in fetch(). +} diff --git a/push-merge.sh b/push-merge.sh index d2293c0..1006e48 100755 --- a/push-merge.sh +++ b/push-merge.sh @@ -18,6 +18,7 @@ TOP="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" source "$TOP/lib/common.sh" +check_env DEFAULT_GIT_BRANCH logmust check_running_system logmust run_setup_if_needed @@ -56,7 +57,6 @@ fi logmust check_package_exists "$PACKAGE" DEFAULT_REVISION="${DEFAULT_REVISION:-$(default_revision)}" -logmust determine_default_git_branch logmust load_package_config "$PACKAGE" if [[ ! -d "$WORKDIR/repo" ]]; then diff --git a/setup.sh b/setup.sh index 48e50d3..9881811 100755 --- a/setup.sh +++ b/setup.sh @@ -19,8 +19,7 @@ TOP="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" source "$TOP/lib/common.sh" logmust check_running_system - -logmust determine_default_git_branch +check_env DEFAULT_GIT_BRANCH # # Update the sources.list file to point to our internal package mirror. If no diff --git a/sync-with-upstream.sh b/sync-with-upstream.sh index def9ab9..568052a 100755 --- a/sync-with-upstream.sh +++ b/sync-with-upstream.sh @@ -18,6 +18,7 @@ TOP="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" source "$TOP/lib/common.sh" +check_env DEFAULT_GIT_BRANCH logmust check_running_system logmust run_setup_if_needed @@ -53,8 +54,6 @@ PACKAGE=$1 logmust check_package_exists "$PACKAGE" -logmust determine_default_git_branch - merging_ref="refs/heads/projects/auto-update/$DEFAULT_GIT_BRANCH/merging" if [[ "$DRYRUN" == 'true' ]]; then merging_ref="${merging_ref}-dryrun"