Skip to content

Commit

Permalink
Merge branch 'main' into feature/29-create-changelog
Browse files Browse the repository at this point in the history
  • Loading branch information
@alee-ntap
alee-ntap committed Dec 1, 2023
2 parents b6503b5 + 7f7b4a8 commit b09292a
Show file tree
Hide file tree
Showing 6 changed files with 254 additions and 74 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/build_pkg.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,10 @@ on:
message:
type: string
default: ""
tag_suffix:
git_filter:
type: string
default: ""
tag_mangel:
git_tag_match:
type: string
default: ""
jobs:
Expand Down
55 changes: 11 additions & 44 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,56 +7,23 @@ It contains:
- `container/`: This folder contains container-related files that define images and configurations.
- `*.containerfile`: These files define container images.
- `apt/`: This folder contains apt configuration files.
- `bin/`: This folder contains executable scripts installs into the container. Note: Also available via bind mount in the workflow for enable the possibility of workflow development tests.
- `bin/`: This folder contains executable scripts installs into the container.
> Note: Also available via bind mount in the workflow for enable the possibility of workflow development tests.
- `conf/`: This folder contains containers configurations on installed packages list and native container setup.
- `misc/`: This folder contains miscellaneous files related to the project.
- `scripts/`: This folder contains scripts for it's github actions workflow.

## Local Development

By following these steps, you will be able to build the container image for your desired architecture and generate packages using the containerized environment. This enables efficient local development, allowing you to iterate on your code, test packages, and validate changes effectively.

### Build container

To build the container, follow the steps below:

Install `podman` on your system: https://podman.io/docs/installation

Update unqualified search in `/etc/containers/registries.conf`:
## Versioning

```
unqualified-search-registries = ["docker.io"]
```
In the following documentation, you can find more information about how the versioning of Garden Linux packages works:
* [docs/versioning.md](docs/versioning.md)

On amd64 architecture:
```
podman build --build-arg arch=amd64 -f container/build.containerfile -t build:amd64 .
```

On arm64 architecture:
```
podman build --build-arg arch=arm64v8 -f container/build.containerfile -t build:arm64v8 .
```

Note: You do not need to use `sudo` as the build should work just fine with unprivileged containers.

### Build a package

To build a package, perform the following steps inside the package repository:

Create a .build directory:
```
mkdir .build
```
## Local Development

On amd64 architecture:
```
podman run --rm -v $PWD:/input -v $PWD/.build:/output build:amd64 build
```
In the following documentation, you can find more information about how to build Garden Linux packages locally on your machine:
* [docs/local_build.md](docs/local_build.md)

On arm64 architecture:
```
podman run --rm -v $PWD:/input -v $PWD/.build:/output build:arm64v8 build
```
## Workflow

Alternatively, you can use pre-built containers from the GitHub Container Registry (ghcr) by accessing the following link: https://github.com/gardenlinux/package-build/pkgs/container/package-build. These pre-built containers can be used as an alternative to building the container locally.
In the following documentation, you can find more information about the Github Action and workflows used to build Garden Linux packages.
* [docs/workflow.md](docs/workflow.md)
80 changes: 52 additions & 28 deletions container/bin/build_source
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@ main() (
debian_source="$(yq -r '.jobs.build_pkg.with.debian_source // "'$debian_source'"' < "$inputs_file")"
build_options="terse $(yq -r '.jobs.build_pkg.with.build_options // ""' < "$inputs_file")"
build_profiles="$(yq -r '.jobs.build_pkg.with.build_profiles // ""' < "$inputs_file")"
tag_suffix="$(yq -r '.jobs.build_pkg.with.tag_suffix // ""' < "$inputs_file")"
tag_mangle="$(yq -r '.jobs.build_pkg.with.tag_mangle // ""' < "$inputs_file")"
git_filter="$(yq -r '.jobs.build_pkg.with.git_filter // "(.*)"' < "$inputs_file")"
git_tag_match="$(yq -r '.jobs.build_pkg.with.git_tag_match // "(.*)"' < "$inputs_file")"

# Define Build Options
export DEB_BUILD_OPTIONS="$build_options"
Expand Down Expand Up @@ -104,8 +104,8 @@ apt_source() (

git_source() (
IFS='#' read -r url ref <<< "$1"
git clone --bare --branch "$ref" --depth 1 "$url" src.git
GIT_DIR=src.git git archive --prefix src/ "$ref" > orig.tar
git clone --depth 1 --bare --branch "$(get_latest_tag)" --depth 1 "$url" src.git
GIT_DIR=src.git git archive --prefix src/ "$(get_latest_tag)" > orig.tar
rm -rf src.git
tar -x < orig.tar
)
Expand Down Expand Up @@ -162,22 +162,51 @@ apply_patches() (
)

get_sources() (
#local source=$1
local source_type=$1

# What kind of source do we have?
if [[ "$source" = "git+"* ]]; then
# What kind of source type do we have?
if [[ "$source_type" = "git+"* ]]; then
git_source "${source#git+}"
elif [[ "$source" = "native" ]]; then
elif [[ "$source_type" = "native" ]]; then
native_source
else
apt_source "$source"
apt_source "$source_type"
fi
)

get_upstream-version() (
IFS='#' read -r url ref <<< "${source#git+}"
git ls-remote --tags --refs $url "$tag_suffix"* | awk -F/ '{ print $3 }' | \
grep -Ev "$tag_mangle" | tail -1 | sed -e s/"$tag_suffix"//
get_tag_version() (
local tag
if [ -z $PACKAGE_VERSION ]; then
tag=$(get_latest_tag)
else
tag=$(get_tag_by_version ${PACKAGE_VERSION#gardenlinux.*})
fi
echo $tag
)

get_latest_tag() (
IFS='#' read -r url ref <<< "${source_type#git+}"
echo $(git ls-remote --tags --refs $url | sed "s#.*\/##" | \
grep -E "$git_filter" | | sort -V | tail -n1)
#grep -E "$git_filter" | sed -E "s#$git_tag_match#\1#" | sort -V | tail -n1)
)

get_upstream_version() (
local source_type=$1

# What kind of source type do we have?
if [[ "$source_type" = "git+"* ]]; then
get_latest_tag
else
native_source
fi

# Append revision -0 on fetched latest git tagged version
# use -0 to keep it less than revision -1 from debian
# this allows us to use debian version in ther future
latest_tag=$(get_latest_tag)
package_version="${latest_tag}-0"
echo $package_version
)

get_version() (
Expand All @@ -186,21 +215,16 @@ get_version() (
# If version is undefined, let's simply use
# the version of the upstream package and add
# the git tag to it.
if [ -z "$version" ]; then
if [ -f /input/debian/changelog ]; then
package_version=$(dpkg-parsechangelog --show-field Version)
else
package_version=$(get_upstream-version)
fi
commit_hash=$(git -C /input rev-parse HEAD)

# No version has been provided. Check if there is already a Garden Linux version 0
# if not, the current package will be the first one. Otherwise, simply use the commit hash.
if git -C /input tag | grep -q "gardenlinux/${package_version}gardenlinux0"; then
version="${package_version}gardenlinux~${commit_hash}"
else
version="${package_version}gardenlinux0"
fi

commit_hash=$(git -C /input rev-parse HEAD)
package_version=$(get_upstream_version $source_type)

# No version has been provided. Check if there is already a Garden Linux version 0
# if not, the current package will be the first one. Otherwise, simply use the commit hash.
if (git -C /input rev-parse --abbrev-ref HEAD | grep -qv "main") || (git -C /input tag | grep -q "gardenlinux/${package_version}gardenlinux0"); then
version="${package_version}gardenlinux~${commit_hash}"
else
version="${package_version}gardenlinux0"
fi

# Print the version accordingly
Expand Down
47 changes: 47 additions & 0 deletions docs/local_build.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Local Development
By following these steps, you will be able to build the container image for your desired architecture and generate packages using the containerized environment. This enables efficient local development, allowing you to iterate on your code, test packages, and validate changes effectively.

## Build container

To build the container, follow the steps below:

Install `podman` on your system: https://podman.io/docs/installation

Update unqualified search in `/etc/containers/registries.conf`:

```
unqualified-search-registries = ["docker.io"]
```

On amd64 architecture:
```
podman build --build-arg arch=amd64 -f container/build.containerfile -t build:amd64 .
```

On arm64 architecture:
```
podman build --build-arg arch=arm64v8 -f container/build.containerfile -t build:arm64v8 .
```

Note: You do not need to use `sudo` as the build should work just fine with unprivileged containers.

## Build a package

To build a package, perform the following steps inside the package repository:

Create a .build directory:
```
mkdir .build
```

On amd64 architecture:
```
podman run --rm -v $PWD:/input -v $PWD/.build:/output build:amd64 build
```

On arm64 architecture:
```
podman run --rm -v $PWD:/input -v $PWD/.build:/output build:arm64v8 build
```

Alternatively, you can use pre-built containers from the GitHub Container Registry (ghcr) by accessing the following link: https://github.com/gardenlinux/package-build/pkgs/container/package-build. These pre-built containers can be used as an alternative to building the container locally.
38 changes: 38 additions & 0 deletions docs/versioning.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Versioning

This chapter describes how the versioning of Garden Linux packages work that have not been mirrored by other sources. Thereby, there are three types of version schemas used:

- [Commit Hash Version](#commit-hash-version)
- [First Upstream Version](#first-upstream-version)
- [Git Tag Version](#git-tag-version)

## Commit Hash Version
**Example:** `gardenlinux/3.2.2-2gardenlinux~3618de3e65de043ca91cbf3f112eaf353a2c9c68`
**Published:** `false`

Whenever there is a build triggered in a given package repo that is not based on a provided Git Tag, the pipeline tries to evaluate if there was already a released build for the given upstream version of the corresponding package by checking the Git tags of the corresponding package repo.

If there was already such a build, this versioning schema applies to the package build then.

This way, a developer can test changes to the package repo like providing new patches for it. Once the resulting package fulfills the expectation, a developer could then tag its latest commit, so that the latest version is properly versioned and could be published to Github releases as described in chapter [Git Tag Version](#git-tag-version).

Packages that are using this kind of versioning are only built but not published. In order to access those packages, the corresponding package build (Github Action) must be opened. It contains the source and binary package of the given built package in its artifact section.

## First Upstream Version
**Example:** `gardenlinux/3.2.2-2gardenlinux0`
**Published:** `true`

If the package pipeline is triggered without providing a Git Tag and it turns out that there was not already a released build for the given upstream version, then the pipeline will automatically add a Git Tag and publish the resulting package to the Github releases. Thereby, the pipeline checks the Git tags of the given package repo to determine if such a version has already been released or not.

In order to provide a proper version schema, the pipeline will add the version suffix `gardenlinux0` to such packages. This way, a developer can see that the package was automatically built and updated by the packaging pipeline.

This mechanism allows to automatically update Garden Linux packages whenever there is an update in the corresponding upstream version without requiring a developer to add a Git tag. This reduces the general maintenance efforts because a developer doesn't need to manually tag a package to get the latest upstream version into the Garden Linux releases.

## Git Tag Version
**Example:** `gardenlinux/3.2.2-2gardenlinux1`, `gardenlinux/3.2.2-2gardenlinux2`, `gardenlinux/3.2.2-2gardenlinux3`, ...
**Published:** `true`

If there is already a released built for the given upstream version in the Garden Linux package releases but a developer needs to provide updates to the package itself (e.g. via new patches), the developer can release such an update by tagging the last functional Git commit with the following schema:
* `{UPSTREAM VERSION}gardenlinux{1|2|3|...}`

As long as the developer does not tag such a package, the versioning from chapter [Commit Hash Version](#commit-hash-version) applies to the built package. Such packages are considered to be used for testing only whereas tagged versions are published to the Garden Linux package releases.
104 changes: 104 additions & 0 deletions docs/workflow.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
# Gardenlinux build-package workflows

This repository contains the Gardenlinux workflows for building and deploying containers and provide a resuable workflow for other packages to build and depolying.

- [Workflow](#workflows)
- [Workflow Detail](#workflow-details)
- [Build Container](#build-container)
- [Workflow Configuration](#workflow-configuration)
- [Job Details](#job-details)
- [Build Package](#build-package)
- [Workflow Configuration](#workflow-configuration-1)
- [Job Details](#job-details-1)
- [Usage](#usage)
- [Contributing](#contributing)

## Workflows

The following workflows are defined in this repository:

- `build_container.yml`: This workflow builds container images based on the provided .containererfile and pushes it to a container registry.
- `build_pkg.yml`: This is reusable workflow builds a package based on the provided input configurations and pushes it to a package repository.

## Workflow Details

### Build Container

The `build_container.yml` workflow is responsible for building container images and pushing it to a container registry.

#### Workflow Configuration

The `container/` folder in this project contains all it's configurations. The `build_container.yml` workflow is triggered on every push to the repository.

#### Job Details

The `build` job within the `build_container.yml` workflow performs the following steps:
- Checks out the repository.
- Sets up the binfmt using a privileged Podman container.
- Builds the container images based on the provided containerfiles, considering the `host` and `target` matrix values that builds amd64:amd64, arm64v8:amd64, amd64:arm64v8 and arm64v8:arm64v8 images.
- Publishes the built container images to the container registry.

### Build Package

The `build_pkg.yml` workflow is a reusable workflow that can be called by other packages. It builds a package based on the provided input configurations and pushes it to a package repository.

#### Workflow Configuration

The `build_pkg.yml` workflow can be triggered by calling it with specific input parameters.

Input Parameters available in `build_pkg.yml`:
| Parameter | Type | Description |
| --------- | ---- | ------------|
|`repository`|**Type:** string<br>**Default:** `${{ github.repository }}`| The repository to build the package from.|
|`ref`|**Type:** string<br>**Default:** `${{ github.sha }}`| The ref (commit or branch) to build the package from.|
|`build_container`|**Type:** string<br>**Default:** `ghcr.io/gardenlinux/package-build`| The container image used for building the package.|
|`dependencies`|**Type:** string| Comma-separated list of repositories and tags to fetch dependencies from.|
|`source`|**Type:** string| The source name of the package. There are three values that one can choose from:<br>- Debian Source Package: `{SOURCE PACKAGE NAME}`<br>- Git Source: `git+{GIT URL}`<br>- Native Build: `native`|
|`debian_source`|**Type:** string| Defines from which source the `debian/` directory should be used. There are three values that one can choose from:<br>- Debian Source Package: `{SOURCE PACKAGE NAME}`<br>- Git Source: `git+{GIT URL}`<br>- Native Build: `native`|
|`email`|**Type:** string<br>**Default:** `[email protected]`| The Debian package maintainer email.|
|`maintainer`|**Type:** string<br>**Default:** `Garden Linux Builder`| The Debian package maintainer full name.|
|`distribution`|**Type:** string<br>**Default:** `gardenlinux`| The target distribution for the package.|
|`message`|**Type:** string<br>**Default:** `Rebuild for Garden Linux.`| The changelog entry message for the package build.|
|`build_option`|**Type:** string| Additional build options for the package build. Build option `terse` is always set.|
|`build_profiles`|**Type:** string| Additional build profiles for the package build.|


#### Job Details

The `build_pkg.yml` workflow consists of the following jobs:
- `source`: This job retrieves the source package, sets up the build environment, and builds the source package.
- `packages`: This job builds the binary packages for the specified architecture.
- `publish`: This job publishes the drafted release.
- `cleanup`: This job deletes the drafted release in case of failure.

#### Usage

To use the `build_pkg.yml` workflow defined in this repository, follow these steps:

1. Create a new workflow file (e.g., `build.yaml`) in the `.github/workflows/` folder of your package-project.
2. In the new workflow file, include the following content, eg: htop:

```
name: project
on: push
jobs:
htop:
uses: gardenlinux/package-build/.github/workflows/build_pkg.yml@branchname
```
Replace `project` to other package name of your package-project. Replace `branchname` with the specific branch or tag of the build_pkg.yaml workflow you want to use. The default is `main` branch.

3. Call the gardenlinux/package-build/.github/workflows/build_pkg.yml@branchname workflow with the specific input parameters under `with:` specific to your project. such as `email`, `maintainer`, `distribution`, `message` and others mentioned in the Workflow Configuration section. The default values will be use if wihout provide any input parameters.
```
uses: gardenlinux/package-build/.github/workflows/build_pkg.yml@branchname
with:
email: your@address
maintainer: "Your Name"
```
4. Commit and push the new workflow file to your project's repository.
5. The `build_pkg.yml` workflow will be triggered based on the provided inputs.

> Note: Make sure to adjust and use the matches branch name on your package-project in the uses field to match the desired branch name of the `build_pkg.yaml@branchname`` workflow.
## Contributing

Contributions to this repository are welcome. If you encounter any issues or have suggestions for improvements, please open an issue or submit a pull request.

0 comments on commit b09292a

Please sign in to comment.