multi-kernel, cross-compiling, bash based Hook & (default+foreign) kernels build (incl GHA matrix) #205

rpardini · 2024-03-22T23:50:57Z

multi-kernel, cross-compiling, bash based Hook & (default+external) kernels build (incl GHA matrix)

Original RFC below; please check commits for the (many) updates done after the first drop. Original RFC kept for reference, below the line.

RFC: multi-kernel, cross-compiling, bash based Hook & (default+external) kernels build (incl GHA matrix)

very early stage RFC

This is a rewrite of the build system.
The produced default artifacts (aarch64/x86_64) should be equivalent, save for an updated 5.10.213+ kernel and arm64 fixes.
It's missing, at least, documentation and linters, possibly more, that I removed and intend to rewrite.
But since it's a large-ish change, I'd like to collect some feedback before continuing.

Main topics

Makefile build replaced with bash. I'm not too proud of the bash, I will definitely clean it up a lot -- it's the concepts I'd need confirmation on.
Replaces the emulated Alpine kernel build with a Debian based cross-compiling build
- Much faster building. Emulating x86_64 on arm64 is very slow and vice-versa.
Replaces kernel .config's with the defconfig versions, via Kbuild's make savedefconfig
Replaces Git-SHA1-based image versioning ("current revision") with content-based hashing.
- This way, there's much higher cache reuse, and new versions are pushed only when components actually changed (caveat emptor)
- Should allow people to develop Hook without having to build a kernel, depending on CI frequency and luck.
Introduces multiple "flavors" of hook. Instead of restricted to 2 hardcoded flavors (x86_64 and aarch64, built from source), we can now define multiple flavors, each with an ID and version/configure/build methods.
- the hook-default-amd64 and hook-default-arm64 kernels are equivalent to the two original.
- the armbian- prefixed kernels are actually Armbian kernels for more exotic arm64 SBCs, or Armbian's generic UEFI kernels for both arches. Those are very fast to "build" since Armbian publishes their .deb packages in OCI images, and here we
  just download and massage them into the format required by Linuxkit.
hook.yaml is replaced with hook.template.yaml which is templated via a limited-var invocation of envsubst; only the kernel image and the arch is actually different per-flavor.
Introduced a distributed GitHub Actions build workflow. The bash build system produces JSON objects that drive the matrix stages:
- One matrix is per-arch, and builds all the containers whose source is hosted in this repo (bootkit, docker, mdev)
- Second matrix is per-flavor(/kernel), and builds the kernel
- Third matrix, depending on the other two, is per-flavor(/kernel), and builds Hook itself (via LinuxKit) and prepares a .tar.gz into GH artifacts
Auto-updating of the kernel via kernel.org's JSON endpoint (ofc only works for LTS or recent-enough stable kernels). Could opt-out/use a fixed version.
Auto updating of Armbian kernels via OCI tag listing via skopeo. Can opt-out/use a fixed version.
DTB-producing Kernel builds (aarch64) produce a dtbs.tar.gz artifact together with the initrd and vmlinuz.

Flavors (/kernels)

Hook's own kernels

ID	Current version	Description
`hook-default-arm64`	5.10.213	Hook's own aarch64 kernel
`hook-default-amd64`	5.10.213	Hook's own x86_64 kernel

Armbian kernels

External kernels, taken from Armbian's OCI repos. Those are "exotic" kernels for certain SoC's.
- edge: release candidates or stable but rarely LTS, more aggressive patching
- current: LTS kernels, stable-ish patching

ID	Current version	Description
`armbian-bcm2711-current`	6.6.22	bcm2711 (Broadcom) current, from RaspberryPi Foundation with many Armbian fixes for CNCF-landscape projects; for the RaspberryPi 3b+/4b/5
`armbian-meson64-edge`	6.7.10	meson64 (Amlogic) edge Khadas VIM3/3L, Radxa Zero/2, LibreComputer Potatos, and many more
`armbian-rockchip64-edge`	6.7.10	rockchip64 (Rockchip) edge, for many rk356x/3399 SoCs. Not for rk3588!
`armbian-uefi-arm64-edge`	6.8.1	Armbian generic edge UEFI kernel
`armbian-uefi-x86-edge`	6.8.1	Armbian generic edge UEFI kernel

Proof of working-ness?

In my fork:

https://github.com/rpardini/tinkerbell-hook/actions/runs/8396946361 A full build workflow, with all misses:
- Default kernels build (in GHA default runners) in 15-21 minutes each
- Armbian kernels are done in less than 2 minutes each
- LK containers in 1 minute for x86_64, 4 minutes for arm64 (done under qemu)
- Most Hook builds in around 1 minute
- Full build done in <24m.
https://github.com/rpardini/tinkerbell-hook/actions/runs/8396643610/job/22998447592 A full build run with all cache hits
- Everything done in <4 minutes
https://github.com/rpardini/tinkerbell-hook/releases/tag/20240322-2128 artifacts for testing (initramfs+vmlinuz+dtbs artifact for each flavor)
https://github.com/rpardini?tab=packages&repo_name=tinkerbell-hook OCI packages in ghcr.io (quay.io had a meltdown today)

Future possibilities:

it would be fairly simple to add Debian/Ubuntu kernels as well as Armbian firmware.
Many, many more Armbian kernels could be added, but save for Allwinner and the Rockchip -rkr vendor kernel, I think they might be too niche.
Users should have an easy time adding it themselves if they need, though.
Better support for u-boot's "pxelinux" booting requires changes outside of Hook (namely in Smee/ipxedust) which I'll PR eventually.
Certain arm64 SoCs require changes in iPXE (nap.h) -- same, I'll PR those to ipxedust repo.
All these Hook flavors are used in a "showcase" Helm chart based on stack that I will also PR to the charts repo.

TO-DO

Find a better name for "flavor". Naming things is hard. Hats? Swords?
Update README.md and CONTRIBUTING.md; For now all I have is really
- bash build.sh config-kernel <flavor> & follow instructions to configure kernel; only works for default flavors
- bash build.sh build-kernel <flavor> builds the kernel
- bash build.sh build <flavor> builds Hook with that kernel
Restore golang & shellcheck linting
Update to Linuxkit 1.2.0 and new linuxkit pkgs. I tried, but there's some incompatible changes that we need to figure out.
Consider using actuated for native arm64 building? -- https://actuated.dev/blog/arm-ci-cncf-ampere

Thanks for reading this far. I'm looking forward to your feedback!

chrisdoherty4 · 2024-03-23T00:10:04Z

Thanks @rpardini. Awesome OP. I've got a busy week so probably won't get to it until next weekend.

rpardini · 2024-03-25T18:11:08Z

won't get to it until next weekend.

No problem, meanwhile I keep on working -- I just added a 2nd drop.

2nd drop of rpardini's take on multi-hook

bump linuxkit/openntpd and remove bind
- now it works, which allows us to use RTC-defient hardware (eg RPi)
introduce build-run-qemu and run-qemu to run linuxkit run qemu
- works on both amd64 & arm64 as long as KVM is available
- still mostly Debian/Ubuntu host only, needs a Fedora treatment
- with TINK_SERVER & MAC env variables set: run a complete test
  - without: run just basic Linuxkit services & kernel, enough to test certain things
refactor a lot of the bash into functions, handle shortcircuits, etc
better bash logging
introduce DO_PUSH checking to avoid pushing on local builds
check for Docker working & having buildx available before starting

- `showcase` is a chart that, based on values.yaml dictionaries: - generates Tinkerbell CRs (Hardware/Template/Workflow) for both standard (UEFI) & exotic (supported by Armbian) devices - generates download/process jobs for multiple Hook flavors (see tinkerbell/hook#205) - generates download/process jobs for a few OS images (Ubuntu Cloud Images, Armbian, etc) - should be independent of how one deployed Tinkerbell itself (stack chart, individual components, etc) - A few features: - validates values.yaml for common mistakes; arch must match, etc. - validates & handles rootDisk differences (re-invents "formatPartition()" a bit) - avoids re-downloading Hooks and Images that are already on disk, even if Job re-runs - allows easy way to use - custom Hooks - custom Kernel cmdline parameters at both the Hook & device level - for example `acpi=off` at Hook level and `console=ttyS0` at board level - custom OS images for deployment - reboot or kexec to finish deployment - different partition numbers for OS image's rootfs (some images have ESP, some have a separate `/boot`, etc) - control if growpart and/or ssh/user setup is done during provisioning or not - conversion of OS images (`qemu-to-raw-gzip` and `xz-to-gz`) - has a "merge" mechanism with a common way to set parameters like net gateway, UEFI, etc (also easy to override per-device) - default values have everything `enabled: false` thus showcase should produce nothing by default. - Hooks & Images can be forced `enabled: true` in values.yaml, or - `enabled: true` Devices automatically enable their Hook & Image - Probably missing: - More validations - Currently pointing to my Tinkerbell Actions, which I haven't PR'ed yet - How to use: - Clone it, edit the values.yaml to your liking, and deploy. Signed-off-by: Ricardo Pardini <[email protected]>

rpardini · 2024-03-31T10:37:14Z

Pushed: forced initial no-offset-limit NTP sync via busybox (fixes RaspberryPi & others without an RTC), support (WiP) for rk3588 devices & a fix for DockerHub rate limits being hit pulling linuxkit/* pkgs during Hook Linuxkit build.

Also, an initial PR for the showcase chart, which demonstrates much of what is being done here, in the charts repo: tinkerbell/charts#89

jacobweinstock · 2024-03-31T14:40:38Z

Hey @rpardini. Thanks for this. I'm playing around with it and the x86_64 build works great. I have some non-technical concerns though. It's not clear how to cross compile HookOS for aarch64. Also, it seems like there are some envs involved for different build.sh commands but it isnt clear what all the env options available are and how to use them properly.

As this is a significant change from the status quo and in order for this to land and be maintainable we're going to need to be able to understand this better. Can you provide docs for all functionality? Most users arent going to need to or want to rebuild kernels so docs around all the options for building the final HookOS are the most important to me. Again, thanks for all this work! I think we really needed it and I want to see it land.

jacobweinstock · 2024-03-31T14:53:23Z

I think i found it. /build.sh linuxkit hook-default-arm64 works great! We'll need docs to explain this. Will also need docs around how to customize a kernel.

jacobweinstock · 2024-03-31T14:58:50Z

Also, i see in the RFC.md that you mention actuated. We have dedicated self hosted GitHub runners for both x86 and aarch64. They can be referenced in github actions via runs-on: [self-hosted, Linux, ARM64] and runs-on: [self-hosted, Linux, X64]

jacobweinstock · 2024-03-31T16:18:36Z

FYI, just got Hook 5.15 and 6.6 kernels built and booted into them! I'm liking this!

rpardini · 2024-03-31T16:23:23Z

Super thanks for the review!

Can you provide docs for all functionality?

... /build.sh linuxkit hook-default-arm64 ... We'll need docs to explain this. Will also need docs around how to customize a kernel.

Definitely! Docs are a big challenge. I hope to massage the RFC.md into README.md over time.
I'm not too sure about the general CLI interface though: it takes environment variables as well as a command and optional flavor/kernel-id.
I guess settling on some sane terminology also needed.

To customize a kernel: bash build.sh config-kernel hook-default-arm64 -- it will prepare a shell in Docker context where you can run make menuconfig -- it outputs some instructions, which evidently must be improved. Do you think we should have a more direct shortcut to make menuconfig && make savedefconfig & cp defconfig ... so all can done in a single step, too?

They can be referenced in github actions via runs-on: [self-hosted, Linux, ARM64] and runs-on: [self-hosted, Linux, X64]

Perfect. I'd like to keep the ability to cross-build on GH Hosted runners, for people without self-hosted runners.

What I propose is adding environment variables like RUNNER_ARM64="self-hosted, Linux, ARM64" and RUNNER_AMD64="self-hosted, Linux, X64" -- if found, the gha-matrix command will use them as values for a new runner: JSON field. If one or both are not set, it defaults to ubuntu-latest. Finally, the field is used in the GHA job matrix as runs-on: ${{matrix.runner}}. Then we can have a conditional step on the gh_org== tinkerbell to set the value; that way forks default to free runners. wdyt?

- `showcase` is a chart that, based on values.yaml dictionaries: - generates Tinkerbell CRs (Hardware/Template/Workflow) for both standard (UEFI) & exotic (supported by Armbian) devices - generates download/process jobs for multiple Hook flavors (see tinkerbell/hook#205) - generates download/process jobs for a few OS images (Ubuntu Cloud Images, Armbian, etc) - should be independent of how one deployed Tinkerbell itself (stack chart, individual components, etc) - A few features: - validates values.yaml for common mistakes; arch must match, etc. - validates & handles rootDisk differences (re-invents "formatPartition()" a bit) - avoids re-downloading Hooks and Images that are already on disk, even if Job re-runs - allows easy way to use - custom Hooks - custom Kernel cmdline parameters at both the Hook & device level - for example `acpi=off` at Hook level and `console=ttyS0` at board level - custom OS images for deployment - reboot or kexec to finish deployment - different partition numbers for OS image's rootfs (some images have ESP, some have a separate `/boot`, etc) - control if growpart and/or ssh/user setup is done during provisioning or not - conversion of OS images (`qemu-to-raw-gzip` and `xz-to-gz`) - has a "merge" mechanism with a common way to set parameters like net gateway, UEFI, etc (also easy to override per-device) - default values have everything `enabled: false` thus showcase should produce nothing by default. - Hooks & Images can be forced `enabled: true` in values.yaml, or - `enabled: true` Devices automatically enable their Hook & Image - Probably missing: - More validations - Currently pointing to my Tinkerbell Actions, which I haven't PR'ed yet - How to use: - Clone it, edit the values.yaml to your liking, and deploy. Signed-off-by: Ricardo Pardini <[email protected]>

rpardini · 2024-04-21T19:20:00Z

Another large drop. I didn't squash this time, so I might have missed some sign-off's.
I wrote some docs

Ref the GitHub actions runners: see the same workflow running against an org with self hosted arm64 runners, and another fork with just plain gh-hosted amd64 runners. Ended up with finer-grained control than proposed above.

rpardini · 2024-04-21T19:35:46Z

got Hook 5.15 and 6.6 kernels built and booted into them!

In the last drop I added hook-latest-lts-amd64 and hook-latest-lts-arm64, using 6.6; the defconfig was simply copied over from 5.10 and olddefconfig'ed. Works well in QEMU, but has some trouble on nvidia hardware (apparently ref nouveau / framebuffer). Could you push your 6.6 defconfig's if you have them?

jacobweinstock · 2024-04-22T21:53:14Z

build.sh

+declare -g HOOK_KERNEL_OCI_BASE="${HOOK_KERNEL_OCI_BASE:-"quay.io/tinkerbellrpardini/kernel-"}"
+declare -g HOOK_LK_CONTAINERS_OCI_BASE="${HOOK_LK_CONTAINERS_OCI_BASE:-"quay.io/tinkerbellrpardini/linuxkit-"}"


These need to default to quay.io/tinkerbell/. Also if there are new repos that don't exist in quay.io/tinkerbell i'll need to create them. Do we maybe want different kernels to be image tags?

Yes, those were mostly examples/leftovers from my two setups.
Ref quay.io I think you're facing the same dilemma as me: it creates new repos/images with "private" visibility by default?
We could yes twist this so the image is always the same, and only the tag changes across kernel flavors.

I changed the default -- but didn't change the naming scheme.

jacobweinstock · 2024-04-24T19:02:36Z

Hey @rpardini , I believe i have found the issue with Docker not starting with Linuxkit v1.2.0. We need to enable cgroups v2 in the DinD container.

  - name: hook-docker
    image: "${HOOK_CONTAINER_DOCKER_IMAGE}"
    capabilities:
      - all
    net: host
    pid: host
    mounts:
      - type: cgroup2
        options: [ "rw", "nosuid", "noexec", "nodev", "relatime" ]
        destination: /sys/fs/cgroup

jacobweinstock · 2024-04-25T21:08:55Z

.github/workflows/build-all-matrix.yaml

+  # Apart from the quay/ghcr coordinates above (used for both pulling & pushing), we might also want to
+  # log in to DockerHub (with a read-only token) so we aren't hit by rate limits when pulling the linuxkit pkgs.
+  # To do so, set the secret DOCKER_USERNAME and DOCKER_PASSWORD in the repo secrets, and set the below to yes.
+  LOGIN_TO_DOCKERHUB: "${{ github.repository_owner == 'rpardini' && 'yes' || 'no' }}"


Sorry, the DockerHub login is ok, but the check for rpardini will need to be removed.

Definitely. I just wanted to exemplify the conditional -- not everyone will have it I guess.
Since I added LK caching (to GHA cache), the need for this is lower as well.

Removed conditional for this PR.

jacobweinstock · 2024-04-27T03:00:45Z

.github/workflows/build-all-matrix.yaml

+      - name: Docker Login to DockerHub # read-only token, required to be able to pull all the linuxkit pkgs without getting rate limited.
+        if: ${{ env.LOGIN_TO_DOCKERHUB == 'yes' }}
+        uses: docker/login-action@v3
+        with: { registry: "docker.io", username: "${{ secrets.DOCKER_USERNAME }}", password: "${{ secrets.DOCKER_PASSWORD }}" }


FYI, I've added dockerhub read only access creds for a Tinkerbell org account. DOCKERHUB_USERNAME and DOCKERHUB_PASSWORD

Set the default to use them and removed the conditional.

…es unexpectedly Signed-off-by: Ricardo Pardini <[email protected]>

…DOCKER_BUILDX_PROGRESS_TYPE` - We've too much debug logging for a normal user; hide it. - DOCKER_BUILDX_PROGRESS_TYPE defaults to `plain` if DEBUG=yes or under GitHub actions - otherwise defaults to `tty` unless `DOCKER_BUILDX_PROGRESS_TYPE` is set Signed-off-by: Ricardo Pardini <[email protected]>

…ary's keys - I still hate the stringified-dict syntax Signed-off-by: Ricardo Pardini <[email protected]>

- Slightly better syntax IMHO Signed-off-by: Ricardo Pardini <[email protected]>

- turns out I randomly called those "flavors", "flavours" or "kernels"; consolidate on "inventory" Signed-off-by: Ricardo Pardini <[email protected]>

… working afterwards - rmdir -> rm -rf Signed-off-by: Ricardo Pardini <[email protected]>

- otherwise it overwrites the default ones during deployment Signed-off-by: Ricardo Pardini <[email protected]>

- Remove references to Nix from CONTRIBUTING.md too - this needs more work, editing and polishing Signed-off-by: Ricardo Pardini <[email protected]>

…ettings - reword some comments - quay.io/tinkerbell is the official base - remove 'dev' tag from CI default - use fixed - 'yes' for DockerHub login; - 'ARM64' tagged self-hosted runners for arm64 lk containers Signed-off-by: Ricardo Pardini <[email protected]>

rpardini · 2024-04-30T21:03:10Z

Hey @jacobweinstock -- I've fixed the DCOs, cleaned up a few commit messages, changed the default OCI coordinates & CI params to Tinkerbell org, and rebased onto main.

I've not touched the existing CI workflows, though; feel free to push to my branch if fixes are needed before merge.

Thanks so much for the reviews!

jacobweinstock

Let's do this!!! Thanks, @rpardini for all the hard work!

- `showcase` is a chart that, based on values.yaml dictionaries: - generates Tinkerbell CRs (Hardware/Template/Workflow) for both standard (UEFI) & exotic (supported by Armbian) devices - generates download/process jobs for multiple Hook flavors (see tinkerbell/hook#205) - generates download/process jobs for a few OS images (Ubuntu Cloud Images, Armbian, etc) - should be independent of how one deployed Tinkerbell itself (stack chart, individual components, etc) - A few features: - validates values.yaml for common mistakes; arch must match, etc. - validates & handles rootDisk differences (re-invents "formatPartition()" a bit) - avoids re-downloading Hooks and Images that are already on disk, even if Job re-runs - allows easy way to use - custom Hooks - custom Kernel cmdline parameters at both the Hook & device level - for example `acpi=off` at Hook level and `console=ttyS0` at board level - custom OS images for deployment - reboot or kexec to finish deployment - different partition numbers for OS image's rootfs (some images have ESP, some have a separate `/boot`, etc) - control if growpart and/or ssh/user setup is done during provisioning or not - conversion of OS images (`qemu-to-raw-gzip` and `xz-to-gz`) - has a "merge" mechanism with a common way to set parameters like net gateway, UEFI, etc (also easy to override per-device) - default values have everything `enabled: false` thus showcase should produce nothing by default. - Hooks & Images can be forced `enabled: true` in values.yaml, or - `enabled: true` Devices automatically enable their Hook & Image - Probably missing: - More validations - Currently pointing to my Tinkerbell Actions, which I haven't PR'ed yet - How to use: - Clone it, edit the values.yaml to your liking, and deploy. Signed-off-by: Ricardo Pardini <[email protected]>

- `showcase` is a chart that, based on values.yaml dictionaries: - generates Tinkerbell CRs (Hardware/Template/Workflow) for both standard (UEFI) & exotic (supported by Armbian) devices - generates download/process jobs for multiple Hook flavors (see tinkerbell/hook#205) - generates download/process jobs for a few OS images (Ubuntu Cloud Images, Armbian, etc) - should be independent of how one deployed Tinkerbell itself (stack chart, individual components, etc) - A few features: - validates values.yaml for common mistakes; arch must match, etc. - validates & handles rootDisk differences (re-invents "formatPartition()" a bit) - avoids re-downloading Hooks and Images that are already on disk, even if Job re-runs - allows easy way to use - custom Hooks - custom Kernel cmdline parameters at both the Hook & device level - for example `acpi=off` at Hook level and `console=ttyS0` at board level - custom OS images for deployment - reboot or kexec to finish deployment - different partition numbers for OS image's rootfs (some images have ESP, some have a separate `/boot`, etc) - control if growpart and/or ssh/user setup is done during provisioning or not - conversion of OS images (`qemu-to-raw-gzip` and `xz-to-gz`) - has a "merge" mechanism with a common way to set parameters like net gateway, UEFI, etc (also easy to override per-device) - default values have everything `enabled: false` thus showcase should produce nothing by default. - Hooks & Images can be forced `enabled: true` in values.yaml, or - `enabled: true` Devices automatically enable their Hook & Image - Probably missing: - More validations - Currently pointing to my Tinkerbell Actions, which I haven't PR'ed yet - How to use: - Clone it, edit the values.yaml to your liking, and deploy. Signed-off-by: Ricardo Pardini <[email protected]> Squashed commits: - showcase: not all images are ubuntu - showcase: introduce insecureRegistries/tinkWorkerImage/tinkWorkerImagePrefix - tinkWorkerImage defaults to quay.io/tinkerbellrpardini/tink-worker:v0.11.0-alpha-rpardini3 - showcase: introduce grpcTLS/grpcInsecureTLS - showcase: generateJobs default to true, allow disabling Jobs - showcase: introduce image.doRestoreGRUBNormalcy for GRUB+EFI entry juggling and enable it for fatso images - showcase: introduce image.doAddEFIBootEntry for efibootmgr juggling and enable it for fatso images - showcase: WiP: nanopct6 (ipxe won't load, even with DTB) - showcase: use rpardini's fork of waitdaemon at quay.io - showcase: bump Hook to 20240604-0609 (further mdev by-id fixes) - showcase: bump Hook to 20240603-0515 (mdev by-id fixes); bump fatso images to v1008 - showcase: bump Hooks; bump images; add rpi4b armbian cloud image; add odroidhc4 example - showcase: fix default userdata script - showcase: bump all armsurvivors Armbian images to 24.05.20-armsurvivors-184; add t95z-cloud-edge image; use qcow2-to-img conversion where relevant - showcase: add support for `xz-qcow2-to-img-gz` image conversion - showcase: bump Hook to 20240520-0941 (finally working armbian's again!) - showcase: bump Hook to 20240520-0729 - showcase: bump Hook to 20240519-1935 (new scheme) - showcase: switch to 20240519-1405 Hook, with new (old?) naming convention - showcase: add demo t95z (meson64) - showcase: bump fatso - showcase: bump images - showcase: add ntp stuff - showcase: bump images, reorg - showcase: add image's 'doInjectHegelCloudInit'; set device's `userData` on the Hardware - add configuration hegelURL - some examples - showcase: bump Hook version; add `latest-lts-xxx` Hooks; add more examples - showcase: bump fatso images - showcase: allow to override hook's defaults via `hookOverride: {}` on the device - showcase: bump hook versions; fix example - showcase: bump hook versions; add examples for peg/pegk and skipDownload - showcase: hook: allow skipping hook download via `skipDownload` - showcase: devices: show `imgstat` before `boot` - showcase: bump fatso images version to 1001; add local variant examples - showcase: HACK: change default Hook amd64 bootMode to reboot (this needs override per-device) - showcase: example 'fatso' baremetal images with download-only and local example - showcase: bump orangepi3b image - showcase: introduce 'download-only' and 'local' non-conversions - showcase: reduce WAIT_SECONDS to zero - showcase: introduce doFixResolvConf at image level - showcase: install pkgs with DEBIAN_FRONTEND=noniteractive, no suggests, no recommends; 180s timeout - showcase: default netplan to device `e*` (not `en*`) so it matches `eth0` too - showcase: add `rock-5b-edge-trixie-uefi-dtb` image for Rock-5b - showcase: add armbian cloud-k8s images for uefi arm64 and amd64 - showcase: bump hook to rpardini's 20240404-2216; add `armbian-rk3588-edge` hook definition (but no board example yet) - showcase: VM examples - showcase: add R58X 3588 example & UEFI cloud image - showcase: validate hookRef not unset - showcase: 0.0.2: disable kexec on arm64, it doesn't really work (yet) - showcase: add README

- `showcase` is a chart that, based on values.yaml dictionaries: - generates Tinkerbell CRs (Hardware/Template/Workflow) for both standard (UEFI) & exotic (supported by Armbian) devices - generates download/process jobs for multiple Hook flavors (see tinkerbell/hook#205) - generates download/process jobs for a few OS images (Ubuntu Cloud Images, Armbian, etc) - should be independent of how one deployed Tinkerbell itself (stack chart, individual components, etc) - A few features: - validates values.yaml for common mistakes; arch must match, etc. - validates & handles rootDisk differences (re-invents "formatPartition()" a bit) - avoids re-downloading Hooks and Images that are already on disk, even if Job re-runs - allows easy way to use - custom Hooks - custom Kernel cmdline parameters at both the Hook & device level - for example `acpi=off` at Hook level and `console=ttyS0` at board level - custom OS images for deployment - reboot or kexec to finish deployment - different partition numbers for OS image's rootfs (some images have ESP, some have a separate `/boot`, etc) - control if growpart and/or ssh/user setup is done during provisioning or not - conversion of OS images (`qemu-to-raw-gzip` and `xz-to-gz`) - has a "merge" mechanism with a common way to set parameters like net gateway, UEFI, etc (also easy to override per-device) - default values have everything `enabled: false` thus showcase should produce nothing by default. - Hooks & Images can be forced `enabled: true` in values.yaml, or - `enabled: true` Devices automatically enable their Hook & Image - Probably missing: - More validations - Currently pointing to my Tinkerbell Actions, which I haven't PR'ed yet - How to use: - Clone it, edit the values.yaml to your liking, and deploy. Signed-off-by: Ricardo Pardini <[email protected]> Squashed commits: - showcase: fix insecure-registries empty bug; add odroidm1; bump hooks - showcase: not all images are ubuntu - showcase: introduce insecureRegistries/tinkWorkerImage/tinkWorkerImagePrefix - tinkWorkerImage defaults to quay.io/tinkerbellrpardini/tink-worker:v0.11.0-alpha-rpardini3 - showcase: introduce grpcTLS/grpcInsecureTLS - showcase: generateJobs default to true, allow disabling Jobs - showcase: introduce image.doRestoreGRUBNormalcy for GRUB+EFI entry juggling and enable it for fatso images - showcase: introduce image.doAddEFIBootEntry for efibootmgr juggling and enable it for fatso images - showcase: WiP: nanopct6 (ipxe won't load, even with DTB) - showcase: use rpardini's fork of waitdaemon at quay.io - showcase: bump Hook to 20240604-0609 (further mdev by-id fixes) - showcase: bump Hook to 20240603-0515 (mdev by-id fixes); bump fatso images to v1008 - showcase: bump Hooks; bump images; add rpi4b armbian cloud image; add odroidhc4 example - showcase: fix default userdata script - showcase: bump all armsurvivors Armbian images to 24.05.20-armsurvivors-184; add t95z-cloud-edge image; use qcow2-to-img conversion where relevant - showcase: add support for `xz-qcow2-to-img-gz` image conversion - showcase: bump Hook to 20240520-0941 (finally working armbian's again!) - showcase: bump Hook to 20240520-0729 - showcase: bump Hook to 20240519-1935 (new scheme) - showcase: switch to 20240519-1405 Hook, with new (old?) naming convention - showcase: add demo t95z (meson64) - showcase: bump fatso - showcase: bump images - showcase: add ntp stuff - showcase: bump images, reorg - showcase: add image's 'doInjectHegelCloudInit'; set device's `userData` on the Hardware - add configuration hegelURL - some examples - showcase: bump Hook version; add `latest-lts-xxx` Hooks; add more examples - showcase: bump fatso images - showcase: allow to override hook's defaults via `hookOverride: {}` on the device - showcase: bump hook versions; fix example - showcase: bump hook versions; add examples for peg/pegk and skipDownload - showcase: hook: allow skipping hook download via `skipDownload` - showcase: devices: show `imgstat` before `boot` - showcase: bump fatso images version to 1001; add local variant examples - showcase: HACK: change default Hook amd64 bootMode to reboot (this needs override per-device) - showcase: example 'fatso' baremetal images with download-only and local example - showcase: bump orangepi3b image - showcase: introduce 'download-only' and 'local' non-conversions - showcase: reduce WAIT_SECONDS to zero - showcase: introduce doFixResolvConf at image level - showcase: install pkgs with DEBIAN_FRONTEND=noniteractive, no suggests, no recommends; 180s timeout - showcase: default netplan to device `e*` (not `en*`) so it matches `eth0` too - showcase: add `rock-5b-edge-trixie-uefi-dtb` image for Rock-5b - showcase: add armbian cloud-k8s images for uefi arm64 and amd64 - showcase: bump hook to rpardini's 20240404-2216; add `armbian-rk3588-edge` hook definition (but no board example yet) - showcase: VM examples - showcase: add R58X 3588 example & UEFI cloud image - showcase: validate hookRef not unset - showcase: 0.0.2: disable kexec on arm64, it doesn't really work (yet) - showcase: add README

rpardini mentioned this pull request Mar 22, 2024

ARM64 server gets stuck on "bpfilter write fail: -32" when activating Linuxkit (Alpine). #198

Closed

rpardini force-pushed the multi-hook branch from 76d095a to 52f87ed Compare March 25, 2024 18:07

chrisdoherty4 marked this pull request as ready for review March 28, 2024 19:27

rpardini mentioned this pull request Mar 31, 2024

tinkerbell/charts: introduce showcase chart tinkerbell/charts#89

Open

chrisdoherty4 assigned jacobweinstock and chrisdoherty4 and unassigned jacobweinstock Apr 2, 2024

jacobweinstock force-pushed the multi-hook branch from 9bedc43 to 328f5e8 Compare April 2, 2024 15:18

rpardini force-pushed the multi-hook branch 2 times, most recently from 77e3230 to 13b9bfd Compare April 21, 2024 19:15

jacobweinstock reviewed Apr 22, 2024

View reviewed changes

jacobweinstock reviewed Apr 25, 2024

View reviewed changes

jacobweinstock mentioned this pull request Apr 26, 2024

Kernel upgrade #188

Closed

3 tasks

jacobweinstock reviewed Apr 27, 2024

View reviewed changes

rpardini added 9 commits April 30, 2024 22:48

build: lk-containers: sort files before hashing them, lest hash chang…

1993c45

…es unexpectedly Signed-off-by: Ricardo Pardini <[email protected]>

build: split inventory; derive inventory array from inventory diction…

7574c20

…ary's keys - I still hate the stringified-dict syntax Signed-off-by: Ricardo Pardini <[email protected]>

build: inventory: introduce define_id() to define kernels/flavours

8c937fe

- Slightly better syntax IMHO Signed-off-by: Ricardo Pardini <[email protected]>

build: hopefully better nomenclature "inventory" instead of "kernels"

19bac9a

- turns out I randomly called those "flavors", "flavours" or "kernels"; consolidate on "inventory" Signed-off-by: Ricardo Pardini <[email protected]>

build: fix: avoid leftover 'run' LK out files preventing 'build' from…

0c1ea1a

… working afterwards - rmdir -> rm -rf Signed-off-by: Ricardo Pardini <[email protected]>

build: introduce FORCE_OUTPUT_ID & use it for the latest-lts id's

cecee05

- otherwise it overwrites the default ones during deployment Signed-off-by: Ricardo Pardini <[email protected]>

docs: update README.md for the new bash build system

cf69b9e

- Remove references to Nix from CONTRIBUTING.md too - this needs more work, editing and polishing Signed-off-by: Ricardo Pardini <[email protected]>

rpardini force-pushed the multi-hook branch from 169314f to ab6b609 Compare April 30, 2024 20:52

rpardini changed the title ~~RFC: multi-kernel, cross-compiling, bash based Hook & (default+external) kernels build (incl GHA matrix)~~ multi-kernel, cross-compiling, bash based Hook & (default+foreign) kernels build (incl GHA matrix) Apr 30, 2024

jacobweinstock approved these changes Apr 30, 2024

View reviewed changes

jacobweinstock added the ready-to-merge Signal to Mergify to merge the PR. label Apr 30, 2024

jacobweinstock merged commit df70921 into tinkerbell:main Apr 30, 2024
4 of 6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

multi-kernel, cross-compiling, bash based Hook & (default+foreign) kernels build (incl GHA matrix) #205

multi-kernel, cross-compiling, bash based Hook & (default+foreign) kernels build (incl GHA matrix) #205

rpardini commented Mar 22, 2024 •

edited

Loading

chrisdoherty4 commented Mar 23, 2024

rpardini commented Mar 25, 2024

rpardini commented Mar 31, 2024

jacobweinstock commented Mar 31, 2024 •

edited

Loading

jacobweinstock commented Mar 31, 2024 •

edited

Loading

jacobweinstock commented Mar 31, 2024 •

edited

Loading

jacobweinstock commented Mar 31, 2024

rpardini commented Mar 31, 2024

rpardini commented Apr 21, 2024

rpardini commented Apr 21, 2024

jacobweinstock Apr 22, 2024 •

edited

Loading

rpardini Apr 27, 2024

rpardini Apr 30, 2024

jacobweinstock commented Apr 24, 2024

jacobweinstock Apr 25, 2024 •

edited

Loading

rpardini Apr 27, 2024

rpardini Apr 30, 2024

jacobweinstock Apr 27, 2024

rpardini Apr 27, 2024

rpardini Apr 30, 2024

rpardini commented Apr 30, 2024

jacobweinstock left a comment

		declare -g HOOK_KERNEL_OCI_BASE="${HOOK_KERNEL_OCI_BASE:-"quay.io/tinkerbellrpardini/kernel-"}"
		declare -g HOOK_LK_CONTAINERS_OCI_BASE="${HOOK_LK_CONTAINERS_OCI_BASE:-"quay.io/tinkerbellrpardini/linuxkit-"}"

multi-kernel, cross-compiling, bash based Hook & (default+foreign) kernels build (incl GHA matrix) #205

multi-kernel, cross-compiling, bash based Hook & (default+foreign) kernels build (incl GHA matrix) #205

Conversation

rpardini commented Mar 22, 2024 • edited Loading

multi-kernel, cross-compiling, bash based Hook & (default+external) kernels build (incl GHA matrix)

RFC: multi-kernel, cross-compiling, bash based Hook & (default+external) kernels build (incl GHA matrix)

Main topics

Flavors (/kernels)

Hook's own kernels

Armbian kernels

Proof of working-ness?

Future possibilities:

TO-DO

chrisdoherty4 commented Mar 23, 2024

rpardini commented Mar 25, 2024

2nd drop of rpardini's take on multi-hook

rpardini commented Mar 31, 2024

jacobweinstock commented Mar 31, 2024 • edited Loading

jacobweinstock commented Mar 31, 2024 • edited Loading

jacobweinstock commented Mar 31, 2024 • edited Loading

jacobweinstock commented Mar 31, 2024

rpardini commented Mar 31, 2024

rpardini commented Apr 21, 2024

rpardini commented Apr 21, 2024

jacobweinstock Apr 22, 2024 • edited Loading

Choose a reason for hiding this comment

rpardini Apr 27, 2024

Choose a reason for hiding this comment

rpardini Apr 30, 2024

Choose a reason for hiding this comment

jacobweinstock commented Apr 24, 2024

jacobweinstock Apr 25, 2024 • edited Loading

Choose a reason for hiding this comment

rpardini Apr 27, 2024

Choose a reason for hiding this comment

rpardini Apr 30, 2024

Choose a reason for hiding this comment

jacobweinstock Apr 27, 2024

Choose a reason for hiding this comment

rpardini Apr 27, 2024

Choose a reason for hiding this comment

rpardini Apr 30, 2024

Choose a reason for hiding this comment

rpardini commented Apr 30, 2024

jacobweinstock left a comment

Choose a reason for hiding this comment

rpardini commented Mar 22, 2024 •

edited

Loading

jacobweinstock commented Mar 31, 2024 •

edited

Loading

jacobweinstock commented Mar 31, 2024 •

edited

Loading

jacobweinstock commented Mar 31, 2024 •

edited

Loading

jacobweinstock Apr 22, 2024 •

edited

Loading

jacobweinstock Apr 25, 2024 •

edited

Loading