diff --git a/.github/CODE_OF_CONDUCT.md b/.github/CODE_OF_CONDUCT.md new file mode 100644 index 0000000..f763e15 --- /dev/null +++ b/.github/CODE_OF_CONDUCT.md @@ -0,0 +1,250 @@ +# Contributor Covenant Code of Conduct + +## Our Pledge + +We as members, contributors, and leaders pledge to make participation in our +community a harassment-free experience for everyone, regardless of age, body +size, visible or invisible disability, ethnicity, sex characteristics, gender +identity and expression, level of experience, education, socio-economic status, +nationality, personal appearance, race, religion, or sexual identity +and orientation. + +We pledge to act and interact in ways that contribute to an open, welcoming, +diverse, inclusive, and healthy community. + +## Our Standards + +Examples of behavior that contributes to a positive environment for our +community include: + +- Demonstrating empathy and kindness toward others +- Being respectful of differing opinions, viewpoints, and experiences +- Giving and gracefully accepting constructive feedback +- Accepting responsibility and apologizing to those affected by our mistakes, + and learning from the experience +- Focusing on what is best not just for us as individuals, but for the + overall community + +Examples of unacceptable behavior include: + +- The use of sexualized language or imagery, and sexual attention or + advances of any kind +- Trolling, insulting or derogatory comments, and personal or political attacks +- Public or private harassment +- Publishing others' private information, such as a physical or email + address, without their explicit permission +- Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Enforcement Responsibilities + +Community leaders are responsible for clarifying and enforcing our standards of +acceptable behavior and will take appropriate and fair corrective action in +response to any behavior that they deem inappropriate, threatening, offensive, +or harmful. + +Community leaders have the right and responsibility to remove, edit, or reject +comments, commits, code, wiki edits, issues, and other contributions that are +not aligned to this Code of Conduct, and will communicate reasons for moderation +decisions when appropriate. + +## Scope + +This Code of Conduct applies within all community spaces, and also applies when +an individual is officially representing the community in public spaces. +Examples of representing our community include using an official e-mail address, +posting via an official social media account, or acting as an appointed +representative at an online or offline event. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported to the community leaders responsible for enforcement at contact@buhl.casa. +All complaints will be reviewed and investigated promptly and fairly. + +All community leaders are obligated to respect the privacy and security of the +reporter of any incident. + +## Enforcement Guidelines + +Community leaders will follow these Community Impact Guidelines in determining +the consequences for any action they deem in violation of this Code of Conduct: + +### 1. Correction + +**Community Impact**: Use of inappropriate language or other behavior deemed +unprofessional or unwelcome in the community. + +**Consequence**: A private, written warning from community leaders, providing +clarity around the nature of the violation and an explanation of why the +behavior was inappropriate. A public apology may be requested. + +### 2. Warning + +**Community Impact**: A violation through a single incident or series +of actions. + +**Consequence**: A warning with consequences for continued behavior. No +interaction with the people involved, including unsolicited interaction with +those enforcing the Code of Conduct, for a specified period of time. This +includes avoiding interactions in community spaces as well as external channels +like social media. Violating these terms may lead to a temporary or +permanent ban. + +### 3. Temporary Ban + +**Community Impact**: A serious violation of community standards, including +sustained inappropriate behavior. + +**Consequence**: A temporary ban from any sort of interaction or public +communication with the community for a specified period of time. No public or +private interaction with the people involved, including unsolicited interaction +with those enforcing the Code of Conduct, is allowed during this period. +Violating these terms may lead to a permanent ban. + +### 4. Permanent Ban + +**Community Impact**: Demonstrating a pattern of violation of community +standards, including sustained inappropriate behavior, harassment of an +individual, or aggression toward or disparagement of classes of individuals. + +**Consequence**: A permanent ban from any sort of public interaction within +the community. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], +version 2.0, available at +https://www.contributor-covenant.org/version/2/0/code_of_conduct.html. + +Community Impact Guidelines were inspired by [Mozilla's code of conduct +enforcement ladder](https://github.com/mozilla/diversity). + +[homepage]: https://www.contributor-covenant.org + +For answers to common questions about this code of conduct, see the FAQ at +https://www.contributor-covenant.org/faq. Translations are available at +https://www.contributor-covenant.org/translations. +We as members, contributors, and leaders pledge to make participation in our +community a harassment-free experience for everyone, regardless of age, body +size, visible or invisible disability, ethnicity, sex characteristics, gender +identity and expression, level of experience, education, socio-economic status, +nationality, personal appearance, race, religion, or sexual identity +and orientation. + +We pledge to act and interact in ways that contribute to an open, welcoming, +diverse, inclusive, and healthy community. + +## Our Standards + +Examples of behavior that contributes to a positive environment for our +community include: + +- Demonstrating empathy and kindness toward other people +- Being respectful of differing opinions, viewpoints, and experiences +- Giving and gracefully accepting constructive feedback +- Accepting responsibility and apologizing to those affected by our mistakes, + and learning from the experience +- Focusing on what is best not just for us as individuals, but for the + overall community + +Examples of unacceptable behavior include: + +- The use of sexualized language or imagery, and sexual attention or + advances of any kind +- Trolling, insulting or derogatory comments, and personal or political attacks +- Public or private harassment +- Publishing others' private information, such as a physical or email + address, without their explicit permission +- Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Enforcement Responsibilities + +Community leaders are responsible for clarifying and enforcing our standards of +acceptable behavior and will take appropriate and fair corrective action in +response to any behavior that they deem inappropriate, threatening, offensive, +or harmful. + +Community leaders have the right and responsibility to remove, edit, or reject +comments, commits, code, wiki edits, issues, and other contributions that are +not aligned to this Code of Conduct, and will communicate reasons for moderation +decisions when appropriate. + +## Scope + +This Code of Conduct applies within all community spaces, and also applies when +an individual is officially representing the community in public spaces. +Examples of representing our community include using an official e-mail address, +posting via an official social media account, or acting as an appointed +representative at an online or offline event. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported to the community leaders responsible for enforcement at contact@buhl.casa. +All complaints will be reviewed and investigated promptly and fairly. + +All community leaders are obligated to respect the privacy and security of the +reporter of any incident. + +## Enforcement Guidelines + +Community leaders will follow these Community Impact Guidelines in determining +the consequences for any action they deem in violation of this Code of Conduct: + +### 1. Correction + +**Community Impact**: Use of inappropriate language or other behavior deemed +unprofessional or unwelcome in the community. + +**Consequence**: A private, written warning from community leaders, providing +clarity around the nature of the violation and an explanation of why the +behavior was inappropriate. A public apology may be requested. + +### 2. Warning + +**Community Impact**: A violation through a single incident or series +of actions. + +**Consequence**: A warning with consequences for continued behavior. No +interaction with the people involved, including unsolicited interaction with +those enforcing the Code of Conduct, for a specified period of time. This +includes avoiding interactions in community spaces as well as external channels +like social media. Violating these terms may lead to a temporary or +permanent ban. + +### 3. Temporary Ban + +**Community Impact**: A serious violation of community standards, including +sustained inappropriate behavior. + +**Consequence**: A temporary ban from any sort of interaction or public +communication with the community for a specified period of time. No public or +private interaction with the people involved, including unsolicited interaction +with those enforcing the Code of Conduct, is allowed during this period. +Violating these terms may lead to a permanent ban. + +### 4. Permanent Ban + +**Community Impact**: Demonstrating a pattern of violation of community +standards, including sustained inappropriate behavior, harassment of an +individual, or aggression toward or disparagement of classes of individuals. + +**Consequence**: A permanent ban from any sort of public interaction within +the community. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], +version 2.0, available at +https://www.contributor-covenant.org/version/2/0/code_of_conduct.html. + +Community Impact Guidelines were inspired by [Mozilla's code of conduct +enforcement ladder](https://github.com/mozilla/diversity). + +[homepage]: https://www.contributor-covenant.org + +For answers to common questions about this code of conduct, see the FAQ at +https://www.contributor-covenant.org/faq. Translations are available at +https://www.contributor-covenant.org/translations. diff --git a/.github/labels.yaml b/.github/labels.yaml new file mode 100644 index 0000000..262ff7b --- /dev/null +++ b/.github/labels.yaml @@ -0,0 +1,11 @@ +--- +- name: renovate/container + color: "027fa0" +- name: type/patch + color: "ffeC19" +- name: type/minor + color: "ff9800" +- name: type/major + color: "f6412d" +- name: hold + color: "ee0701" diff --git a/.github/renovate.json5 b/.github/renovate.json5 new file mode 100644 index 0000000..4ca27d3 --- /dev/null +++ b/.github/renovate.json5 @@ -0,0 +1,50 @@ +{ + $schema: "https://docs.renovatebot.com/renovate-schema.json", + extends: [ + "config:recommended", + "docker:enableMajor", + ":dependencyDashboard", + ":disableRateLimiting", + ":timezone(America/New_York)", + ":semanticCommits", + ], + dependencyDashboardTitle: "Renovate Dashboard 🤖", + suppressNotifications: ["prEditedNotification", "prIgnoreNotification"], + packageRules: [ + { + addLabels: ["renovate/container", "type/major"], + additionalBranchPrefix: "{{parentDir}}-", + commitMessageExtra: " ( {{currentVersion}} → {{newVersion}} )", + commitMessagePrefix: "feat({{parentDir}})!: ", + commitMessageTopic: "{{depName}}", + labels: ["app/{{parentDir}}"], + matchDatasources: ["docker"], + matchFileNames: ["apps/**/Dockerfile"], + matchUpdateTypes: ["major"], + }, + { + addLabels: ["renovate/container", "type/minor"], + additionalBranchPrefix: "{{parentDir}}-", + commitMessageExtra: "( {{currentVersion}} → {{newVersion}} )", + commitMessageTopic: "{{depName}}", + labels: ["app/{{parentDir}}"], + matchDatasources: ["docker"], + matchFileNames: ["apps/**/Dockerfile"], + matchUpdateTypes: ["minor"], + semanticCommitScope: "{{parentDir}}", + semanticCommitType: "feat", + }, + { + addLabels: ["renovate/container", "type/patch"], + additionalBranchPrefix: "{{parentDir}}-", + commitMessageExtra: "( {{currentVersion}} → {{newVersion}} )", + commitMessageTopic: "{{depName}}", + labels: ["app/{{parentDir}}"], + matchDatasources: ["docker"], + matchFileNames: ["apps/**/Dockerfile"], + matchUpdateTypes: ["patch"], + semanticCommitScope: "{{parentDir}}", + semanticCommitType: "fix", + }, + ], +} diff --git a/.github/workflows/boltz.yaml b/.github/workflows/boltz.yaml deleted file mode 100644 index 416ca13..0000000 --- a/.github/workflows/boltz.yaml +++ /dev/null @@ -1,206 +0,0 @@ -name: Build and push Boltz - -on: - workflow_dispatch: - inputs: - image_tag: - description: "Tag for the image" - required: false - default: "latest" - -env: - REGISTRY: ghcr.io - IMAGE_NAME: boltz - -jobs: - build-boltz: - runs-on: ubuntu-latest - permissions: - contents: read - packages: write - steps: - - name: Checkout repository - uses: actions/checkout@v4 - - - name: Set repository owner - id: set_repo_owner_lower - shell: bash - run: | - echo "REPOSITORY_OWNER=$(echo '${{ github.repository_owner }}' | tr '[:upper:]' '[:lower:]')" >> $GITHUB_ENV - - # - name: Set up QEMU - # uses: docker/setup-qemu-action@v3 - # with: - # platforms: arm64,amd64 - - - name: Login to the container registry - uses: docker/login-action@v3 - with: - registry: ${{ env.REGISTRY }} - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - - - name: Extract image metadata - id: meta - uses: docker/metadata-action@v4 - with: - images: | - ${{ env.REGISTRY }}/${{ env.REPOSITORY_OWNER }}/${{ env.IMAGE_NAME }} - flavor: latest=auto - tags: type=raw,value=latest,enable={{is_default_branch}} - - - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v3 - - - name: Build and push Docker image - uses: docker/build-push-action@v6 - with: - push: true - #platforms: arm64,amd64 - context: ./${{ env.IMAGE_NAME }}/python3.11-slim - tags: ${{ env.REGISTRY }}/${{ env.REPOSITORY_OWNER }}/${{ env.IMAGE_NAME }}:${{ github.event.inputs.image_tag }} - build-args: BASE_IMAGE=python:3.11-slim - # build-boltz-cu118: - # #runs-on: ubuntu-latest - # runs-on: big-storage - # permissions: - # contents: read - # packages: write - # steps: - # - name: Checkout repository - # uses: actions/checkout@v4 - # - # - name: Set repository owner - # id: set_repo_owner_lower - # shell: bash - # run: | - # echo "REPOSITORY_OWNER=$(echo '${{ github.repository_owner }}' | tr '[:upper:]' '[:lower:]')" >> $GITHUB_ENV - # - # # - name: Set up QEMU - # # uses: docker/setup-qemu-action@v3 - # # with: - # # platforms: arm64,amd64 - # - # - name: Login to the container registry - # uses: docker/login-action@v3 - # with: - # registry: ${{ env.REGISTRY }} - # username: ${{ github.actor }} - # password: ${{ secrets.GITHUB_TOKEN }} - # - # - name: Extract image metadata - # id: meta - # uses: docker/metadata-action@v4 - # with: - # images: | - # ${{ env.REGISTRY }}/${{ env.REPOSITORY_OWNER }}/${{ env.IMAGE_NAME }} - # flavor: latest=auto - # tags: type=raw,value=latest,enable={{is_default_branch}} - # - # - name: Set up Docker Buildx - # uses: docker/setup-buildx-action@v3 - # - # - name: Build and push Docker image - # uses: docker/build-push-action@v6 - # with: - # push: true - # #platforms: arm64,amd64 - # context: ./${{ env.IMAGE_NAME }}/11.8.0-cudnn8-runtime-ubuntu22.04 - # tags: ${{ env.REGISTRY }}/${{ env.REPOSITORY_OWNER }}/${{ env.IMAGE_NAME }}:${{ github.event.inputs.image_tag }}-cu118 - # build-args: BASE_IMAGE=nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu22.04 - # build-boltz-cu121: - # #runs-on: ubuntu-latest - # runs-on: big-storage - # permissions: - # contents: read - # packages: write - # steps: - # - name: Checkout repository - # uses: actions/checkout@v4 - # - # - name: Set repository owner - # id: set_repo_owner_lower - # shell: bash - # run: | - # echo "REPOSITORY_OWNER=$(echo '${{ github.repository_owner }}' | tr '[:upper:]' '[:lower:]')" >> $GITHUB_ENV - # - # # - name: Set up QEMU - # # uses: docker/setup-qemu-action@v3 - # # with: - # # platforms: arm64,amd64 - # - # - name: Login to the container registry - # uses: docker/login-action@v3 - # with: - # registry: ${{ env.REGISTRY }} - # username: ${{ github.actor }} - # password: ${{ secrets.GITHUB_TOKEN }} - # - # - name: Extract image metadata - # id: meta - # uses: docker/metadata-action@v4 - # with: - # images: | - # ${{ env.REGISTRY }}/${{ env.REPOSITORY_OWNER }}/${{ env.IMAGE_NAME }} - # flavor: latest=auto - # tags: type=raw,value=latest,enable={{is_default_branch}} - # - # - name: Set up Docker Buildx - # uses: docker/setup-buildx-action@v3 - # - # - name: Build and push Docker image - # uses: docker/build-push-action@v6 - # with: - # push: true - # #platforms: arm64,amd64 - # context: ./${{ env.IMAGE_NAME }}/12.1.0-cudnn8-runtime-ubuntu22.04 - # tags: ${{ env.REGISTRY }}/${{ env.REPOSITORY_OWNER }}/${{ env.IMAGE_NAME }}:${{ github.event.inputs.image_tag }}-cu121 - # build-args: BASE_IMAGE=nvidia/cuda:12.1.0-cudnn8-runtime-ubuntu22.04 - build-boltz-cu124: - #runs-on: ubuntu-latest - runs-on: big-storage - permissions: - contents: read - packages: write - steps: - - name: Checkout repository - uses: actions/checkout@v4 - - - name: Set repository owner - id: set_repo_owner_lower - shell: bash - run: | - echo "REPOSITORY_OWNER=$(echo '${{ github.repository_owner }}' | tr '[:upper:]' '[:lower:]')" >> $GITHUB_ENV - - # - name: Set up QEMU - # uses: docker/setup-qemu-action@v3 - # with: - # platforms: arm64,amd64 - - - name: Login to the container registry - uses: docker/login-action@v3 - with: - registry: ${{ env.REGISTRY }} - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - - - name: Extract image metadata - id: meta - uses: docker/metadata-action@v4 - with: - images: | - ${{ env.REGISTRY }}/${{ env.REPOSITORY_OWNER }}/${{ env.IMAGE_NAME }} - flavor: latest=auto - tags: type=raw,value=latest,enable={{is_default_branch}} - - - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v3 - - - name: Build and push Docker image - uses: docker/build-push-action@v6 - with: - push: true - #platforms: arm64,amd64 - context: ./${{ env.IMAGE_NAME }}/12.4.1-cudnn-runtime-ubuntu22.04 - tags: ${{ env.REGISTRY }}/${{ env.REPOSITORY_OWNER }}/${{ env.IMAGE_NAME }}:${{ github.event.inputs.image_tag }}-cu124 - build-args: BASE_IMAGE=nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04 diff --git a/.github/workflows/build-images.yaml b/.github/workflows/build-images.yaml new file mode 100644 index 0000000..5ab2de7 --- /dev/null +++ b/.github/workflows/build-images.yaml @@ -0,0 +1,351 @@ +--- +# yaml-language-server: $schema=https://json.schemastore.org/github-workflow.json +name: "Image Build" + +on: + workflow_call: + inputs: + appsToBuild: + required: false + type: string + default: "" + channelsToBuild: + required: false + type: string + default: "" + pushImages: + required: false + default: false + type: boolean + sendNotifications: + required: false + default: false + type: boolean + force: + required: false + default: true + type: boolean + description: Force rebuild + secrets: + BOT_APP_ID: + description: The App ID of the GitHub App + required: true + BOT_APP_PRIVATE_KEY: + description: The private key of the GitHub App + required: true + +jobs: + prepare: + name: Prepare to Build + runs-on: ubuntu-latest + outputs: + matrices: ${{ steps.prepare-matrices.outputs.matrices }} + steps: + - name: Lowercase repository owner + shell: bash + run: echo "LOWERCASE_REPO_OWNER=${GITHUB_REPOSITORY_OWNER,,}" >> $GITHUB_ENV + + - name: Generate Token + uses: actions/create-github-app-token@v1 + id: app-token + with: + app-id: "${{ secrets.BOT_APP_ID }}" + private-key: "${{ secrets.BOT_APP_PRIVATE_KEY }}" + + - name: Checkout + uses: actions/checkout@v4 + with: + token: "${{ steps.app-token.outputs.token }}" + + - name: Setup Python + uses: actions/setup-python@v5 + with: + python-version: 3.x + cache: pip + + - name: Install Python Requirements + shell: bash + run: pip install --requirement requirements.txt && pip freeze + + - name: Prepare Matrices + id: prepare-matrices + env: + TOKEN: ${{ steps.app-token.outputs.token }} + shell: bash + run: | + if [[ -z "${{ inputs.appsToBuild }}" ]]; then + matrices=$(python ./scripts/actions/prepare-matrices.py "all" "${{ inputs.pushImages }}" "${{ inputs.force }}") + else + if [[ -z "${{ inputs.channelsToBuild }}" ]]; then + matrices=$(python ./scripts/actions/prepare-matrices.py "${{ inputs.appsToBuild }}" "${{ inputs.pushImages }}" "${{ inputs.force }}") + else + matrices=$(python ./scripts/actions/prepare-matrices.py "${{ inputs.appsToBuild }}" "${{ inputs.pushImages }}" "${{ inputs.force }}" "${{ inputs.channelsToBuild }}") + fi + fi + echo "matrices=${matrices}" >> $GITHUB_OUTPUT + echo "${matrices}" + + build-platform-images: + name: Build/Test ${{ matrix.image.name }} (${{ matrix.image.platform }}) + needs: prepare + runs-on: ubuntu-latest + if: ${{ toJSON(fromJSON(needs.prepare.outputs.matrices).imagePlatforms) != '[]' && toJSON(fromJSON(needs.prepare.outputs.matrices).imagePlatforms) != '' }} + strategy: + fail-fast: false + matrix: + image: + ["${{ fromJSON(needs.prepare.outputs.matrices).imagePlatforms }}"] + permissions: + contents: read + packages: write + steps: + - name: Lowercase repository owner + shell: bash + run: echo "LOWERCASE_REPO_OWNER=${GITHUB_REPOSITORY_OWNER,,}" >> $GITHUB_ENV + + - name: Log Matrix Input + shell: bash + run: | + cat << EOF + ${{ toJSON(matrix.image)}} + EOF + + - name: Validate Matrix Input + shell: bash + run: | + if [[ -z "${{ matrix.image.name }}" ]]; then + echo "image.name is empty" + exit 1 + fi + if [[ -z "${{ matrix.image.version }}" ]]; then + echo "image.version is empty" + exit 1 + fi + if [[ -z "${{ matrix.image.context }}" ]]; then + echo "image.context is empty" + exit 1 + fi + if [[ -z "${{ matrix.image.dockerfile }}" ]]; then + echo "image.dockerfile is empty" + exit 1 + fi + if [[ -z "${{ matrix.image.platform }}" ]]; then + echo "image.platform is empty" + exit 1 + fi + if [[ -z "${{ matrix.image.tests_enabled }}" ]]; then + echo "image.tests_enabled is empty" + exit 1 + fi + echo "${{ matrix.image.name }}" | grep -E "[a-zA-Z0-9_\.\-]+" || "Image Name is invalid" + echo "${{ matrix.image.version }}" | grep -E "[a-zA-Z0-9_\.\-]+" || "Image Version is invalid" + + - name: Generate Token + uses: actions/create-github-app-token@v1 + id: app-token + with: + app-id: "${{ secrets.BOT_APP_ID }}" + private-key: "${{ secrets.BOT_APP_PRIVATE_KEY }}" + + - name: Checkout + uses: actions/checkout@v4 + with: + token: "${{ steps.app-token.outputs.token }}" + + - name: Setup Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Login to GitHub Container Registry + uses: docker/login-action@v3 + with: + registry: ghcr.io + username: "${{ github.actor }}" + password: "${{ secrets.GITHUB_TOKEN }}" + + - name: Setup Goss + if: ${{ matrix.image.tests_enabled }} + uses: e1himself/goss-installation-action@v1 + with: + version: latest + + - name: Prepare Build Outputs + id: prepare-build-outputs + shell: bash + run: | + if [[ "${{ inputs.pushImages }}" == "true" ]]; then + image_name="ghcr.io/${{ env.LOWERCASE_REPO_OWNER }}/${{ matrix.image.name }}" + outputs="type=image,name=${image_name},push-by-digest=true,name-canonical=true,push=true" + else + image_name="ghcr.io/${{ env.LOWERCASE_REPO_OWNER }}/${{ matrix.image.name }}:zztesting" + outputs="type=docker,name=${image_name},push=false" + fi + echo "image_name=${image_name}" >> $GITHUB_OUTPUT + echo "outputs=${outputs}" >> $GITHUB_OUTPUT + + - name: Build Image + uses: docker/build-push-action@v6 + id: build + with: + build-args: |- + VERSION=${{ matrix.image.version }} + REVISION=${{ github.sha }} + CHANNEL=${{ matrix.image.channel }} + # TODO: Use ${{ matrix.image.context }}, requires updates to all dockerfiles :-( + context: . + platforms: ${{ matrix.image.platform }} + file: ${{ matrix.image.dockerfile }} + outputs: ${{ steps.prepare-build-outputs.outputs.outputs }} + cache-from: type=gha + cache-to: type=gha,mode=max + labels: |- + org.opencontainers.image.title=${{ matrix.image.name }} + org.opencontainers.image.url=https://ghcr.io/${{ env.LOWERCASE_REPO_OWNER }}/${{ matrix.image.name }} + org.opencontainers.image.version=${{ matrix.image.version }} + org.opencontainers.image.revision=${{ github.sha }} + org.opencontainers.image.vendor=${{ env.LOWERCASE_REPO_OWNER }} + + - name: Run Goss Tests + id: dgoss + if: ${{ matrix.image.tests_enabled }} + env: + CONTAINER_RUNTIME: docker + GOSS_FILE: ${{ matrix.image.goss_config }} + GOSS_OPTS: --retry-timeout 60s --sleep 2s --color --format documentation + GOSS_SLEEP: 2 + GOSS_FILES_STRATEGY: cp + CONTAINER_LOG_OUTPUT: goss_container_log_output + shell: bash + run: | + if [[ '${{ inputs.pushImages }}' == 'true' ]]; then + image_name="${{ steps.prepare-build-outputs.outputs.image_name }}@${{ steps.build.outputs.digest }}" + else + image_name="${{ steps.prepare-build-outputs.outputs.image_name }}" + fi + dgoss run ${image_name} ${{ matrix.image.goss_args }} + + - name: Export Digest + id: export-digest + if: ${{ inputs.pushImages }} + shell: bash + run: | + mkdir -p /tmp/${{ matrix.image.name }}/digests + digest="${{ steps.build.outputs.digest }}" + echo "${{ matrix.image.name }}" > "/tmp/${{ matrix.image.name }}/digests/${digest#sha256:}" + + - name: Upload Digest + if: ${{ inputs.pushImages}} + uses: actions/upload-artifact@v4 + with: + name: ${{ matrix.image.name }}-${{ matrix.image.target_os }}-${{ matrix.image.target_arch }} + path: /tmp/${{ matrix.image.name }}/* + if-no-files-found: error + retention-days: 1 + + merge: + name: Merge ${{ matrix.image.name }} + runs-on: ubuntu-latest + needs: ["prepare", "build-platform-images"] + # Always run merge, as the prior matrix is all or nothing. We test for prior step failure + # in the "Test Failed Bit" step. This ensures if one app fails, others can still complete. + if: ${{ always() && inputs.pushImages && toJSON(fromJSON(needs.prepare.outputs.matrices).images) != '[]' && toJSON(fromJSON(needs.prepare.outputs.matrices).images) != '' }} + strategy: + matrix: + image: ["${{ fromJSON(needs.prepare.outputs.matrices).images }}"] + fail-fast: false + steps: + - name: Lowercase repository owner + shell: bash + run: echo "LOWERCASE_REPO_OWNER=${GITHUB_REPOSITORY_OWNER,,}" >> $GITHUB_ENV + + - name: Download Digests + uses: actions/download-artifact@v4 + with: + pattern: "${{ matrix.image.name }}-{linux,darwin}-{amd64,arm64}" + merge-multiple: true + path: /tmp/${{ matrix.image.name }} + + - name: Ensure all platforms were built + id: ensure-platforms + shell: bash + run: | + EXPECTED_COUNT=$(cat << EOF | jq ". | length" + ${{ toJSON(matrix.image.platforms) }} + EOF + ) + ACTUAL_COUNT=$(ls -1 /tmp/${{ matrix.image.name }}/digests | wc -l) + if [[ $EXPECTED_COUNT != $ACTUAL_COUNT ]]; then + echo "Expected $EXPECTED_COUNT platforms, but only found $ACTUAL_COUNT" + echo "Expected: ${{ toJSON(matrix.image.platforms) }}" + echo "Actual: $(cat /tmp/${{ matrix.image.name }}/digests/*)" + exit 1 + fi + + - name: Setup Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Login to GitHub Container Registry + uses: docker/login-action@v3 + with: + registry: ghcr.io + username: "${{ github.actor }}" + password: "${{ secrets.GITHUB_TOKEN }}" + + - name: Log Files + working-directory: /tmp/${{ matrix.image.name }}/digests + shell: bash + run: | + ls -la + cat * + + - name: Merge Manifests + id: merge + working-directory: /tmp/${{ matrix.image.name }}/digests + env: + TAGS: ${{ toJSON(matrix.image.tags) }} + shell: bash + run: | + docker buildx imagetools create $(jq -cr '. | map("-t ghcr.io/${{ env.LOWERCASE_REPO_OWNER }}/${{matrix.image.name}}:" + .) | join(" ")' <<< "$TAGS") \ + $(printf 'ghcr.io/${{ env.LOWERCASE_REPO_OWNER }}/${{ matrix.image.name }}@sha256:%s ' *) + + - name: Inspect image + id: inspect + shell: bash + run: | + docker buildx imagetools inspect ghcr.io/${{ env.LOWERCASE_REPO_OWNER }}/${{ matrix.image.name }}:${{ matrix.image.tags[0] }} + + - name: Build successful + id: build-success + if: ${{ always() && steps.merge.outcome == 'success' && steps.inspect.outcome == 'success' }} + shell: bash + run: | + echo "message=🎉 ${{ matrix.image.name }} (${{ matrix.image.tags[0] }})" >> $GITHUB_OUTPUT + echo "color=0x00FF00" >> $GITHUB_OUTPUT + + - name: Build failed + id: build-failed + if: ${{ always() && (steps.merge.outcome == 'failure' || steps.inspect.outcome == 'failure') }} + shell: bash + run: | + echo "message=💥 ${{ matrix.image.name }} (${{ matrix.image.tags[0] }})" >> $GITHUB_OUTPUT + echo "color=0xFF0000" >> $GITHUB_OUTPUT + + - name: Send Discord Webhook + uses: sarisia/actions-status-discord@v1 + if: ${{ always() && inputs.sendNotifications == 'true' }} + with: + webhook: ${{ secrets.DISCORD_WEBHOOK }} + title: ${{ steps.build-failed.outputs.message || steps.build-success.outputs.message }} + color: ${{ steps.build-failed.outputs.color || steps.build-success.outputs.color }} + username: GitHub Actions + + # Summarize matrix https://github.community/t/status-check-for-a-matrix-jobs/127354/7 + build_success: + name: Build matrix success + runs-on: ubuntu-latest + needs: ["prepare", "merge"] + if: ${{ always() }} + steps: + - name: Check build matrix status + if: ${{ (inputs.appsToBuild != '' && inputs.appsToBuild != '[]') && (needs.merge.result != 'success' && needs.merge.result != 'skipped' && needs.prepare.result != 'success') }} + shell: bash + run: exit 1 diff --git a/.github/workflows/get-changed-images.yaml b/.github/workflows/get-changed-images.yaml new file mode 100644 index 0000000..f6426e3 --- /dev/null +++ b/.github/workflows/get-changed-images.yaml @@ -0,0 +1,47 @@ +--- +# yaml-language-server: $schema=https://json.schemastore.org/github-workflow.json +name: "Get Changed Images" + +on: + workflow_call: + outputs: + addedOrModified: + description: "Whether any files were added or modified" + value: ${{ jobs.get-changed-images.outputs.addedOrModified }} + addedOrModifiedImages: + description: "The images that were added or modified" + value: ${{ jobs.get-changed-images.outputs.addedOrModifiedImages }} + +jobs: + get-changed-images: + name: Get Changed Images + runs-on: ubuntu-latest + outputs: + addedOrModified: "${{ steps.changed-files.outputs.all_changed_and_modified_files_count > 0 }}" + addedOrModifiedImages: "${{ steps.changed-containers.outputs.addedOrModifiedImages }}" + steps: + - name: Checkout + uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Get Changed Files + id: changed-files + uses: tj-actions/changed-files@v45 + with: + files: apps/** + dir_names: true + dir_names_max_depth: 2 + json: true + escape_json: false + + - if: ${{ steps.changed-files.outputs.all_changed_and_modified_files_count > 0 }} + name: Determine changed images + id: changed-containers + shell: bash + run: | + IMAGES=$( \ + echo '${{ steps.changed-files.outputs.all_changed_and_modified_files }}' \ + | jq --raw-output 'map(sub("^apps/"; "")) | join(",")' \ + ) + echo "addedOrModifiedImages=${IMAGES}" >> $GITHUB_OUTPUT diff --git a/.github/workflows/label-sync.yaml b/.github/workflows/label-sync.yaml new file mode 100644 index 0000000..dd94d49 --- /dev/null +++ b/.github/workflows/label-sync.yaml @@ -0,0 +1,49 @@ +--- +# yaml-language-server: $schema=https://json.schemastore.org/github-workflow.json +name: "Label Sync" + +on: + workflow_dispatch: + push: + branches: ["main"] + paths: + - .github/labels.yaml + - .github/workflows/label-sync.yaml + - apps/** + schedule: + - cron: "0 0 * * *" # Every day at midnight + +jobs: + label-sync: + name: Label Sync + runs-on: ubuntu-latest + steps: + - name: Generate Token + uses: actions/create-github-app-token@v1 + id: app-token + with: + app-id: "${{ secrets.BOT_APP_ID }}" + private-key: "${{ secrets.BOT_APP_PRIVATE_KEY }}" + + - name: Checkout + uses: actions/checkout@v4 + with: + token: "${{ steps.app-token.outputs.token }}" + + - name: Setup Homebrew + uses: Homebrew/actions/setup-homebrew@master + + - name: Setup Workflow Tools + shell: bash + run: brew install go-task + + - name: Append app labels to the labels config file + shell: bash + run: task append-app-labels --force + + - name: Sync Labels + uses: EndBug/label-sync@v2 + with: + token: "${{ steps.app-token.outputs.token }}" + config-file: .github/labels.yaml + delete-other-labels: true diff --git a/.github/workflows/pr-validate.yaml b/.github/workflows/pr-validate.yaml new file mode 100644 index 0000000..837304b --- /dev/null +++ b/.github/workflows/pr-validate.yaml @@ -0,0 +1,29 @@ +--- +# yaml-language-server: $schema=https://json.schemastore.org/github-workflow.json +name: "Pull Request: Validate" + +on: + pull_request: + branches: ["main"] + types: ["opened", "synchronize", "reopened"] + +concurrency: + group: ${{ github.head_ref }}-pr-validate + cancel-in-progress: true + +jobs: + simple-checks: + uses: RareCompute/containers/.github/workflows/simple-checks.yaml@main + + get-changed-images: + uses: RareCompute/containers/.github/workflows/get-changed-images.yaml@main + + build-images: + needs: ["simple-checks", "get-changed-images"] + if: ${{ needs.get-changed-images.outputs.addedOrModified == 'true' }} + uses: RareCompute/containers/.github/workflows/build-images.yaml@main + secrets: inherit + with: + appsToBuild: "${{ needs.get-changed-images.outputs.addedOrModifiedImages }}" + pushImages: false + sendNotifications: false diff --git a/.github/workflows/qlora.yaml b/.github/workflows/qlora.yaml deleted file mode 100644 index d8127ec..0000000 --- a/.github/workflows/qlora.yaml +++ /dev/null @@ -1,208 +0,0 @@ -# NOTE: Unfortunately, as far as I know, these need to either be -# separate jobs or we need to use our own CI runner. Github -# Actions doesn't provide enough disk space by default. - -name: Build and push QLoRA - -on: - workflow_dispatch: - inputs: - image_tag: - description: "Tag for the image" - required: false - default: "latest" - -env: - REGISTRY: ghcr.io - IMAGE_NAME: qlora - -jobs: - build-qlora: - runs-on: ubuntu-latest - permissions: - contents: read - packages: write - steps: - - name: Checkout repository - uses: actions/checkout@v4 - - - name: Set repository owner - id: set_repo_owner_lower - shell: bash - run: | - echo "REPOSITORY_OWNER=$(echo '${{ github.repository_owner }}' | tr '[:upper:]' '[:lower:]')" >> $GITHUB_ENV - - # - name: Set up QEMU - # uses: docker/setup-qemu-action@v3 - # with: - # platforms: arm64,amd64 - - - name: Login to the container registry - uses: docker/login-action@v3 - with: - registry: ${{ env.REGISTRY }} - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - - - name: Extract image metadata - id: meta - uses: docker/metadata-action@v4 - with: - images: | - ${{ env.REGISTRY }}/${{ env.REPOSITORY_OWNER }}/${{ env.IMAGE_NAME }} - flavor: latest=auto - tags: type=raw,value=latest,enable={{is_default_branch}} - - - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v3 - - - name: Build and push Docker image - uses: docker/build-push-action@v6 - with: - push: true - #platforms: arm64,amd64 - context: ./${{ env.IMAGE_NAME }}/python3.11-slim - tags: ${{ env.REGISTRY }}/${{ env.REPOSITORY_OWNER }}/${{ env.IMAGE_NAME }}:${{ github.event.inputs.image_tag }} - build-args: BASE_IMAGE=python:3.11-slim - # build-qlora-cu118: - # #runs-on: ubuntu-latest - # runs-on: big-storage - # permissions: - # contents: read - # packages: write - # steps: - # - name: Checkout repository - # uses: actions/checkout@v4 - # - # - name: Set repository owner - # id: set_repo_owner_lower - # shell: bash - # run: | - # echo "REPOSITORY_OWNER=$(echo '${{ github.repository_owner }}' | tr '[:upper:]' '[:lower:]')" >> $GITHUB_ENV - # - # # - name: Set up QEMU - # # uses: docker/setup-qemu-action@v3 - # # with: - # # platforms: arm64,amd64 - # - # - name: Login to the container registry - # uses: docker/login-action@v3 - # with: - # registry: ${{ env.REGISTRY }} - # username: ${{ github.actor }} - # password: ${{ secrets.GITHUB_TOKEN }} - # - # - name: Extract image metadata - # id: meta - # uses: docker/metadata-action@v4 - # with: - # images: | - # ${{ env.REGISTRY }}/${{ env.REPOSITORY_OWNER }}/${{ env.IMAGE_NAME }} - # flavor: latest=auto - # tags: type=raw,value=latest,enable={{is_default_branch}} - # - # - name: Set up Docker Buildx - # uses: docker/setup-buildx-action@v3 - # - # - name: Build and push Docker image - # uses: docker/build-push-action@v6 - # with: - # push: true - # #platforms: arm64,amd64 - # context: ./${{ env.IMAGE_NAME }}/11.8.0-cudnn8-runtime-ubuntu22.04 - # tags: ${{ env.REGISTRY }}/${{ env.REPOSITORY_OWNER }}/${{ env.IMAGE_NAME }}:${{ github.event.inputs.image_tag }} - # build-args: BASE_IMAGE=nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu22.04 - # build-qlora-cu121: - # runs-on: ubuntu-latest - # permissions: - # contents: read - # packages: write - # steps: - # - name: Checkout repository - # uses: actions/checkout@v4 - # - # - name: Set repository owner - # id: set_repo_owner_lower - # shell: bash - # run: | - # echo "REPOSITORY_OWNER=$(echo '${{ github.repository_owner }}' | tr '[:upper:]' '[:lower:]')" >> $GITHUB_ENV - # - # # - name: Set up QEMU - # # uses: docker/setup-qemu-action@v3 - # # with: - # # platforms: arm64,amd64 - # - # - name: Login to the container registry - # uses: docker/login-action@v3 - # with: - # registry: ${{ env.REGISTRY }} - # username: ${{ github.actor }} - # password: ${{ secrets.GITHUB_TOKEN }} - # - # - name: Extract image metadata - # id: meta - # uses: docker/metadata-action@v4 - # with: - # images: | - # ${{ env.REGISTRY }}/${{ env.REPOSITORY_OWNER }}/${{ env.IMAGE_NAME }} - # flavor: latest=auto - # tags: type=raw,value=latest,enable={{is_default_branch}} - # - # - name: Set up Docker Buildx - # uses: docker/setup-buildx-action@v3 - # - # - name: Build and push Docker image - # uses: docker/build-push-action@v6 - # with: - # push: true - # #platforms: arm64,amd64 - # context: ./${{ env.IMAGE_NAME }}/12.1.0-cudnn8-runtime-ubuntu22.04 - # tags: ${{ env.REGISTRY }}/${{ env.REPOSITORY_OWNER }}/${{ env.IMAGE_NAME }}:${{ github.event.inputs.image_tag }}-cu121 - # build-args: BASE_IMAGE=nvidia/cuda:12.1.0-cudnn8-runtime-ubuntu22.04 - build-qlora-cu124: - runs-on: ubuntu-latest - permissions: - contents: read - packages: write - steps: - - name: Checkout repository - uses: actions/checkout@v4 - - - name: Set repository owner - id: set_repo_owner_lower - shell: bash - run: | - echo "REPOSITORY_OWNER=$(echo '${{ github.repository_owner }}' | tr '[:upper:]' '[:lower:]')" >> $GITHUB_ENV - - # - name: Set up QEMU - # uses: docker/setup-qemu-action@v3 - # with: - # platforms: arm64,amd64 - - - name: Login to the container registry - uses: docker/login-action@v3 - with: - registry: ${{ env.REGISTRY }} - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - - - name: Extract image metadata - id: meta - uses: docker/metadata-action@v4 - with: - images: | - ${{ env.REGISTRY }}/${{ env.REPOSITORY_OWNER }}/${{ env.IMAGE_NAME }} - flavor: latest=auto - tags: type=raw,value=latest,enable={{is_default_branch}} - - - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v3 - - - name: Build and push Docker image - uses: docker/build-push-action@v6 - with: - push: true - #platforms: arm64,amd64 - context: ./${{ env.IMAGE_NAME }}/12.4.1-cudnn-runtime-ubuntu22.04 - tags: ${{ env.REGISTRY }}/${{ env.REPOSITORY_OWNER }}/${{ env.IMAGE_NAME }}:${{ github.event.inputs.image_tag }}-cu124 - build-args: BASE_IMAGE=nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04 diff --git a/.github/workflows/release-on-merge.yaml b/.github/workflows/release-on-merge.yaml new file mode 100644 index 0000000..9e28dce --- /dev/null +++ b/.github/workflows/release-on-merge.yaml @@ -0,0 +1,41 @@ +--- +# yaml-language-server: $schema=https://json.schemastore.org/github-workflow.json +name: Release on Merge + +concurrency: + group: container-release + cancel-in-progress: false + +on: + push: + branches: ["main"] + paths: + - "apps/**" + - "scripts/templates/**" + - "!apps/**/metadata.yaml" + - "!apps/**/README.md" + +jobs: + simple-checks: + uses: RareCompute/containers/.github/workflows/simple-checks.yaml@main + + get-changed-images: + needs: ["simple-checks"] + uses: RareCompute/containers/.github/workflows/get-changed-images.yaml@main + + build-images: + needs: ["simple-checks", "get-changed-images"] + if: ${{ needs.get-changed-images.outputs.addedOrModified == 'true' }} + uses: RareCompute/containers/.github/workflows/build-images.yaml@main + secrets: inherit + with: + appsToBuild: "${{ needs.get-changed-images.outputs.addedOrModifiedImages }}" + pushImages: true + sendNotifications: true + + render-readme: + name: Render Readme + needs: build-images + if: ${{ always() && needs.build-images.result != 'failure' }} + uses: ./.github/workflows/render-readme.yaml + secrets: inherit diff --git a/.github/workflows/release-scheduled.yaml b/.github/workflows/release-scheduled.yaml new file mode 100644 index 0000000..60fdcb7 --- /dev/null +++ b/.github/workflows/release-scheduled.yaml @@ -0,0 +1,48 @@ +--- +# yaml-language-server: $schema=https://json.schemastore.org/github-workflow.json +name: Scheduled Release + +concurrency: + group: container-release + cancel-in-progress: false + +on: + workflow_dispatch: + inputs: + appsToBuild: + description: App(s) to build + required: false + type: string + default: all + force: + description: Force rebuild + type: boolean + default: false + required: true + schedule: + - cron: "0 * * * *" + +jobs: + simple-checks: + name: Simple Checks + uses: RareCompute/containers/.github/workflows/simple-checks.yaml@main + + build-images: + name: Build Images + needs: simple-checks + uses: RareCompute/containers/.github/workflows/build-images.yaml@main + secrets: inherit + permissions: + contents: read + packages: write + with: + appsToBuild: ${{ inputs.appsToBuild }} + force: ${{ inputs.force == true }} + pushImages: true + sendNotifications: true + + render-readme: + name: Render Readme + needs: build-images + uses: ./.github/workflows/render-readme.yaml + secrets: inherit diff --git a/.github/workflows/render-readme.yaml b/.github/workflows/render-readme.yaml new file mode 100644 index 0000000..9ed0e2a --- /dev/null +++ b/.github/workflows/render-readme.yaml @@ -0,0 +1,55 @@ +--- +# yaml-language-server: $schema=https://json.schemastore.org/github-workflow.json +name: "Render Readme" + +on: + workflow_call: + secrets: + BOT_APP_ID: + description: The App ID of the GitHub App + required: true + BOT_APP_PRIVATE_KEY: + description: The private key of the GitHub App + required: true + +jobs: + render-readme: + name: Render README + runs-on: ubuntu-latest + steps: + - name: Generate Token + uses: actions/create-github-app-token@v1 + id: app-token + with: + app-id: "${{ secrets.BOT_APP_ID }}" + private-key: "${{ secrets.BOT_APP_PRIVATE_KEY }}" + + - name: Checkout + uses: actions/checkout@v4 + with: + token: "${{ steps.app-token.outputs.token }}" + + - name: Setup Python + uses: actions/setup-python@v5 + with: + python-version: 3.x + cache: pip + + - name: Install Python Requirements + shell: bash + run: pip install --requirement requirements.txt && pip freeze + + - name: Render README + env: + GITHUB_TOKEN: "${{ steps.app-token.outputs.token }}" + shell: bash + run: python ./scripts/actions/render-readme.py + + - name: Commit Changes + shell: bash + run: | + git config --global user.name "bot-ross" + git config --global user.email "98030736+bot-ross[bot]@users.noreply.github.com" + git add ./README.md + git commit -m "chore: render README.md" || echo "No changes to commit" + git push origin || echo "No changes to push" diff --git a/.github/workflows/renovate.yaml b/.github/workflows/renovate.yaml new file mode 100644 index 0000000..6228457 --- /dev/null +++ b/.github/workflows/renovate.yaml @@ -0,0 +1,63 @@ +--- +# yaml-language-server: $schema=https://json.schemastore.org/github-workflow.json +name: "Renovate" + +on: + workflow_dispatch: + inputs: + dryRun: + description: Dry Run + default: "false" + required: false + logLevel: + description: Log Level + default: debug + required: false + version: + description: Renovate version + default: latest + required: false + schedule: + - cron: "0 * * * *" # Every hour + push: + branches: ["main"] + paths: + - .github/renovate.json5 + - .github/renovate/**.json5 + +concurrency: + group: ${{ github.workflow }}-${{ github.event.number || github.ref }} + cancel-in-progress: true + +env: + LOG_LEVEL: "${{ inputs.logLevel || 'debug' }}" + RENOVATE_AUTODISCOVER: true + RENOVATE_AUTODISCOVER_FILTER: "${{ github.repository }}" + RENOVATE_DRY_RUN: "${{ inputs.dryRun == true }}" + RENOVATE_PLATFORM: github + RENOVATE_PLATFORM_COMMIT: true + WORKFLOW_RENOVATE_VERSION: "${{ inputs.version || 'latest' }}" + +jobs: + renovate: + name: Renovate + runs-on: ubuntu-latest + steps: + - name: Generate Token + uses: actions/create-github-app-token@v1 + id: app-token + with: + app-id: "${{ secrets.BOT_APP_ID }}" + private-key: "${{ secrets.BOT_APP_PRIVATE_KEY }}" + + - name: Checkout + uses: actions/checkout@v4 + with: + token: "${{ steps.app-token.outputs.token }}" + + - name: Renovate + uses: renovatebot/github-action@v41.0.4 + with: + configurationFile: .github/renovate.json5 + token: "${{ steps.app-token.outputs.token }}" + renovate-version: "${{ env.WORKFLOW_RENOVATE_VERSION }}" diff --git a/.github/workflows/simple-checks.yaml b/.github/workflows/simple-checks.yaml new file mode 100644 index 0000000..6bfb476 --- /dev/null +++ b/.github/workflows/simple-checks.yaml @@ -0,0 +1,36 @@ +--- +# yaml-language-server: $schema=https://json.schemastore.org/github-workflow.json +name: "Simple Checks" + +on: + workflow_call: + +jobs: + metadata-validation: + name: Validate Image Metadata + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Get Changed Files + id: changed-files + uses: tj-actions/changed-files@v45 + with: + files_yaml: | + cue: + - metadata.rules.cue + - apps/**/metadata.yaml + + - name: Setup CUE + if: ${{ steps.changed-files.outputs.cue_any_changed == 'true' }} + uses: cue-lang/setup-cue@v1.0.1 + + # Run against all files to ensure they are tested if the cue schema is changed. + - name: Validate image metadata + if: ${{ steps.changed-files.outputs.cue_any_changed == 'true' }} + shell: bash + run: | + find ./apps/ -name metadata.yaml | xargs -I {} cue vet --schema '#Spec' {} ./metadata.rules.cue diff --git a/Taskfile.yaml b/Taskfile.yaml new file mode 100644 index 0000000..bf962db --- /dev/null +++ b/Taskfile.yaml @@ -0,0 +1,40 @@ +--- +version: "3" + +vars: + LABELS_CONFIG_FILE: "{{.ROOT_DIR}}/.github/labels.yaml" + PIP_REQUIREMENTS_FILE: "{{.ROOT_DIR}}/requirements.txt" + PYTHON_BIN: python3 + +env: + VIRTUAL_ENV: "{{.ROOT_DIR}}/.venv" + +tasks: + default: + cmd: task --list + silent: true + + venv: + desc: Set up a Python virtual environment + cmds: + - "{{.PYTHON_BIN}} -m venv {{.VIRTUAL_ENV}}" + - "{{.VIRTUAL_ENV}}/bin/python3 -m pip install --upgrade pip setuptools wheel" + - '{{.VIRTUAL_ENV}}/bin/python3 -m pip install --upgrade --requirement "{{.PIP_REQUIREMENTS_FILE}}"' + sources: + - "{{.PIP_REQUIREMENTS_FILE}}" + generates: + - "{{.VIRTUAL_ENV}}/pyvenv.cfg" + preconditions: + - test -f {{.PIP_REQUIREMENTS_FILE}} + + append-app-labels: + desc: Append app labels to the labels config file + cmds: + - for: { var: apps } + cmd: | + yq --inplace '. += [{"name": "app/{{.ITEM}}", "color": "0e8a16"}]' {{.LABELS_CONFIG_FILE}} + vars: + apps: + sh: ls --directory {{.ROOT_DIR}}/apps/*/ | xargs --max-args=1 basename + preconditions: + - sh: "[[ -z {{.CLI_FORCE}} ]]" diff --git a/apps/boltz/Dockerfile b/apps/boltz/Dockerfile new file mode 100644 index 0000000..eaeae83 --- /dev/null +++ b/apps/boltz/Dockerfile @@ -0,0 +1,61 @@ +FROM docker.io/library/python:3.12-slim-bookworm + +ARG TARGETPLATFORM +ARG VERSION +ARG CHANNEL + +ENV \ + PYTHONDONTWRITEBYTECODE=1 \ + PYTHONUNBUFFERED=1 \ + PIP_ROOT_USER_ACTION=ignore \ + PIP_NO_CACHE_DIR=1 \ + PIP_DISABLE_PIP_VERSION_CHECK=1 \ + PIP_BREAK_SYSTEM_PACKAGES=1 \ + CRYPTOGRAPHY_DONT_BUILD_RUST=1\ + #NVIDIA_VISIBLE_DEVICES all \ + NVIDIA_DRIVER_CAPABILITIES="compute,video,utility,graphics" + +ENV UMASK="0002" \ + TZ="Etc/UTC" + +USER root +WORKDIR /app + +RUN \ + apt-get update && apt-get install -y --no-install-recommends \ + curl \ + unzip \ + catatonit \ + build-essential \ + python3-venv \ + && python3 -m venv /opt/venv \ + && . /opt/venv/bin/activate \ + && \ + curl -fsSL -o /tmp/app.zip "https://github.com/jwohlwend/boltz/archive/refs/tags/v${VERSION}.zip" \ + && unzip -q /tmp/app.zip -d /app \ + && \ + printf "UpdateMethod=docker\nBranch=master\nPackageVersion=%s\nPackageAuthor=[RareCompute](https://github.com/RareCompute)\n" "${VERSION}" > /app/package_info \ + && pip install --no-cache-dir \ + torch \ + torchvision \ + torchaudio \ + --requirement /app/requirements.txt \ + && chown -R root:root /app && chmod -R 755 /app \ + && apt-get purge -y build-essential \ + && apt-get autoremove -y \ + && apt-get clean \ + && rm -rf /root/.cache /var/lib/apt/lists/* /tmp/* /var/tmp/* + +COPY ./apps/boltz/entrypoint.sh /entrypoint.sh + +USER nobody:nogroup +WORKDIR /cache +VOLUME ["/cache"] + +WORKDIR /predictions +VOLUME ["/predictions"] + + +ENTRYPOINT ["/usr/bin/catatonit", "--", "/entrypoint.sh"] + +LABEL org.opencontainers.image.source="https://github.com/jwohlwend/bolt" diff --git a/apps/boltz/README.md b/apps/boltz/README.md new file mode 100644 index 0000000..6783df3 --- /dev/null +++ b/apps/boltz/README.md @@ -0,0 +1,34 @@ +# Boltz + +By default the image will attempt a prediction based on the provided flags and input. If you want to create a shell pod, i.e. research environment, you can pass `SHELL: true` + +Attach a volume for data and models by referencing `/cache` and for generated predictions by referencing `/predictions` + +## Upstream repository + +From the [author repository](https://github.com/jwohlwend/boltz/tree/main): + +> Boltz-1 is the state-of-the-art open-source model that predicts the 3D structure of proteins, RNA, DNA, and small molecules; it handles modified residues, covalent ligands and glycans, as well as condition the generation on pocket residues. + +## Environment variables + +You can configure the docker image using the below environment variables + +| Environment Variable | CLI Flag | Type | Default Value | Description | +| ---------------------- | ------------------------ | -------------------- | --------------------------- | ------------------------------------------------------------------- | +| `OUTPUT_DIR` | `--out_dir` | `PATH` | `./` | The path where predictions will be saved. | +| `DOWNLOAD_DIR` | `--cache` | `PATH` | `/cache` | The directory for downloading data and models. | +| `DEVICE_TYPE` | `--accelerator` | `[gpu, cpu, tpu]` | `gpu` | The type of accelerator to use for predictions. | +| `DEVICES_COUNT` | `--devices` | `INTEGER` | `1` | The number of devices to use for predictions. | +| `RECYCLING_STEPS` | `--recycling_steps` | `INTEGER` | `3` | The number of recycling steps to use for predictions. | +| `SAMPLING_STEPS` | `--sampling_steps` | `INTEGER` | `200` | The number of sampling steps to use for predictions. | +| `DIFFUSION_SAMPLES` | `--diffusion_samples` | `INTEGER` | `1` | The number of diffusion samples to use for predictions. | +| `OUTPUT_FORMAT` | `--output_format` | `[pdb, mmcif]` | `mmcif` | The output format for predictions. | +| `WORKERS` | `--num_workers` | `INTEGER` | `2` | The number of workers for data loading. | +| `OVERRIDE` | `--override` | `FLAG` | `False` | Whether to override existing predictions. | +| `USE_MSA_SERVER` | `--use_msa_server` | `FLAG` | `False` | Whether to use the MSA server to generate MSAs. | +| `MSA_SERVER_URL` | `--msa_server_url` | `STR` | `https://api.colabfold.com` | The URL of the MSA server (used only if `--use_msa_server` is set). | +| `MSA_PAIRING_STRATEGY` | `--msa_pairing_strategy` | `[greedy, complete]` | `greedy` | The MSA pairing strategy to use (requires `--use_msa_server`). | +| `WRITE_FULL_PAE` | `--write_full_pae` | `FLAG` | `False` | Whether to save the full PAE matrix as a file. | +| `WRITE_FULL_PDE` | `--write_full_pde` | `FLAG` | `False` | Whether to save the full PDE matrix as a file. | +| `SHELL` | N/A | N.A | N/A | Start the docker container as a shell | diff --git a/apps/boltz/ci/goss.yaml b/apps/boltz/ci/goss.yaml new file mode 100644 index 0000000..4488e21 --- /dev/null +++ b/apps/boltz/ci/goss.yaml @@ -0,0 +1,7 @@ +--- +# yaml-language-server: $schema=https://raw.githubusercontent.com/goss-org/goss/master/docs/schema.yaml +file: + /opt/venv/bin/boltz: + exists: true + /app/LICENSE: + exists: true diff --git a/apps/boltz/ci/latest.sh b/apps/boltz/ci/latest.sh new file mode 100644 index 0000000..b324345 --- /dev/null +++ b/apps/boltz/ci/latest.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env bash +version=$(curl -sX GET "https://api.github.com/repos/jwohlwend/boltz/releases/latest" | jq --raw-output '.tag_name' 2>/dev/null) +version="${version#*v}" +version="${version#*release-}" +printf "%s" "${version}" diff --git a/apps/boltz/entrypoint.sh b/apps/boltz/entrypoint.sh new file mode 100644 index 0000000..eb369a2 --- /dev/null +++ b/apps/boltz/entrypoint.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +#shellcheck disable=SC2086 + +CACHE_PATH="${CACHE_PATH:-/cache}" +OUTPUT_PATH="${OUTPUT_PATH:-/predictions}" + +EXTRA_FLAGS="" +declare -A FLAG_MAP=( + [CACHE_PATH]="--cache" + [OUTPUT_PATH]="--out_dir" + [DEVICE_TYPE]="--accelerator" + [DEVICES_COUNT]="--devices" + [RECYCLING_STEPS]="--recycling_steps" + [SAMPLING_STEPS]="--sampling_steps" + [DIFFUSION_SAMPLES]="--diffusion_samples" + [OUTPUT_FORMAT]="--output_format" + [WORKERS]="--num_workers" + [OVERRIDE]="--override" + [USE_MSA_SERVER]="--use_msa_server" + [MSA_SERVER_URL]="--msa_server_url" + [MSA_PAIRING_STRATEGY]="--msa_pairing_strategy" + [WRITE_FULL_PAE]="--write_full_pae" + [WRITE_FULL_PDE]="--write_full_pde" +) + +for VAR in "${!FLAG_MAP[@]}"; do + if [ -n "${!VAR}" ]; then + EXTRA_FLAGS+=" ${FLAG_MAP[$VAR]} ${!VAR}" + fi +done + +if [ "${SHELL}" = "true" ]; then + exec /opt/venv/boltz "$@" +else + exec /opt/venv/boltz predict "$@" $EXTRA_FLAGS +fi diff --git a/apps/boltz/metadata.yaml b/apps/boltz/metadata.yaml new file mode 100644 index 0000000..03e5deb --- /dev/null +++ b/apps/boltz/metadata.yaml @@ -0,0 +1,11 @@ +--- +#yamllint disable +app: rfdiffusion +semver: true +channels: + - name: stable + platforms: ["linux/amd64", "linux/arm64"] + stable: true + tests: + enabled: true + type: web diff --git a/boltz/12.4.1-cudnn-runtime-ubuntu22.04/Dockerfile b/apps/rfdiffusion/Dockerfile similarity index 91% rename from boltz/12.4.1-cudnn-runtime-ubuntu22.04/Dockerfile rename to apps/rfdiffusion/Dockerfile index f7f8a2a..3662ace 100644 --- a/boltz/12.4.1-cudnn-runtime-ubuntu22.04/Dockerfile +++ b/apps/rfdiffusion/Dockerfile @@ -1,8 +1,8 @@ # Base image # ---------------------- -ARG PROJECT_NAME=boltz -ARG PROJECT_URL=https://github.com/jwohlwend/boltz.git -ARG BASE_IMAGE=nvidia/cuda:12.1.0-cudnn8-runtime-ubuntu22.04 +ARG PROJECT_NAME=rfdiffusion +ARG PROJECT_URL=https://github.com/RosettaCommons/RFdiffusion.git +ARG BASE_IMAGE=nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04 FROM ${BASE_IMAGE} AS builder # Build diff --git a/boltz/11.8.0-cudnn8-runtime-ubuntu22.04/Dockerfile b/boltz/11.8.0-cudnn8-runtime-ubuntu22.04/Dockerfile deleted file mode 100644 index f030e59..0000000 --- a/boltz/11.8.0-cudnn8-runtime-ubuntu22.04/Dockerfile +++ /dev/null @@ -1,64 +0,0 @@ -# Base image -# ---------------------- -ARG PROJECT_NAME=boltz -ARG PROJECT_URL=https://github.com/jwohlwend/boltz.git -ARG BASE_IMAGE=nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu22.04 -FROM ${BASE_IMAGE} AS builder - -# Build -# ---------------------- -RUN apt-get update && apt-get install -y --no-install-recommends \ - git \ - build-essential \ - python3 \ - python3-dev \ - python3-venv \ - && python3 -m venv /opt/venv \ - && . /opt/venv/bin/activate \ - && git clone https://github.com/jwohlwend/boltz.git /app \ - && pip install --no-cache-dir --upgrade pip \ - && pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 \ - && pip install --no-cache-dir boltz \ - && apt-get purge -y git build-essential \ - && apt-get autoremove -y \ - && apt-get clean \ - && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* - -# Metadata -# ---------------------- -FROM ${BASE_IMAGE} - -LABEL base_image=${BASE_IMAGE} \ - software=${PROJECT_NAME} \ - about.home=${PROJECT_URL} \ - about.summary="Biomolecular interaction model for 3D protein structure prediction" \ - about.tags="Model" \ - about.license="MIT" \ - about.license_file="https://github.com/jwohlwend/boltz/blob/main/LICENSE" \ - version="1" - -# Configuration -# ---------------------- -ARG USERNAME=rare -ARG USER_UID=900 -ARG USER_GID=900 - -ENV DEBIAN_FRONTEND=noninteractive -ENV LANG=C.UTF-8 -ENV PATH="/opt/venv/bin:$PATH" - -# Installation -# ---------------------- -RUN groupadd --gid $USER_GID $USERNAME && \ - useradd --uid $USER_UID --gid $USER_GID --create-home --shell /bin/bash $USERNAME \ - && apt-get update && apt-get install -y --no-install-recommends python3 \ - && apt-get autoremove -y \ - && apt-get clean \ - && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* - -COPY --from=builder --chown=$USER_UID:$USER_GID /opt/venv /opt/venv -COPY --from=builder --chown=$USER_UID:$USER_GID /app /app - -WORKDIR /app -USER $USERNAME -ENTRYPOINT ["boltz"] diff --git a/boltz/12.1.0-cudnn8-runtime-ubuntu22.04/Dockerfile b/boltz/12.1.0-cudnn8-runtime-ubuntu22.04/Dockerfile deleted file mode 100644 index 0adf044..0000000 --- a/boltz/12.1.0-cudnn8-runtime-ubuntu22.04/Dockerfile +++ /dev/null @@ -1,64 +0,0 @@ -# Base image -# ---------------------- -ARG PROJECT_NAME=boltz -ARG PROJECT_URL=https://github.com/jwohlwend/boltz.git -ARG BASE_IMAGE=nvidia/cuda:12.1.0-cudnn8-runtime-ubuntu22.04 -FROM ${BASE_IMAGE} AS builder - -# Build -# ---------------------- -RUN apt-get update && apt-get install -y --no-install-recommends \ - git \ - build-essential \ - python3 \ - python3-dev \ - python3-venv \ - && python3 -m venv /opt/venv \ - && . /opt/venv/bin/activate \ - && git clone https://github.com/jwohlwend/boltz.git /app \ - && pip install --no-cache-dir --upgrade pip \ - && pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121 \ - && pip install --no-cache-dir boltz \ - && apt-get purge -y git build-essential \ - && apt-get autoremove -y \ - && apt-get clean \ - && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* - -# Metadata -# ---------------------- -FROM ${BASE_IMAGE} - -LABEL base_image=${BASE_IMAGE} \ - software=${PROJECT_NAME} \ - about.home=${PROJECT_URL} \ - about.summary="Biomolecular interaction model for 3D protein structure prediction" \ - about.tags="Model" \ - about.license="MIT" \ - about.license_file="https://github.com/jwohlwend/boltz/blob/main/LICENSE" \ - version="1" - -# Configuration -# ---------------------- -ARG USERNAME=rare -ARG USER_UID=900 -ARG USER_GID=900 - -ENV DEBIAN_FRONTEND=noninteractive -ENV LANG=C.UTF-8 -ENV PATH="/opt/venv/bin:$PATH" - -# Installation -# ---------------------- -RUN groupadd --gid $USER_GID $USERNAME && \ - useradd --uid $USER_UID --gid $USER_GID --create-home --shell /bin/bash $USERNAME \ - && apt-get update && apt-get install -y --no-install-recommends python3 \ - && apt-get autoremove -y \ - && apt-get clean \ - && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* - -COPY --from=builder --chown=$USER_UID:$USER_GID /opt/venv /opt/venv -COPY --from=builder --chown=$USER_UID:$USER_GID /app /app - -WORKDIR /app -USER $USERNAME -ENTRYPOINT ["boltz"] diff --git a/boltz/README.md b/boltz/README.md deleted file mode 100644 index 842962c..0000000 --- a/boltz/README.md +++ /dev/null @@ -1,5 +0,0 @@ -From [jwohlwend/boltz](https://github.com/jwohlwend/boltz): - -> Boltz-1 is an open-source model which predicts the 3D structure of proteins, RNA, DNA and small molecules; it handles modified residues, covalent ligands and glycans, as well as condition the generation on pocket residues. -> -> For more information about the model, see our [technical report](https://gcorso.github.io/assets/boltz1.pdf). diff --git a/boltz/python3.11-slim/Dockerfile b/boltz/python3.11-slim/Dockerfile deleted file mode 100644 index df694ff..0000000 --- a/boltz/python3.11-slim/Dockerfile +++ /dev/null @@ -1,58 +0,0 @@ -# Base image -# ---------------------- -ARG PROJECT_NAME=boltz -ARG PROJECT_URL=https://github.com/jwohlwend/boltz.git -ARG BASE_IMAGE=python:3.11-slim -FROM ${BASE_IMAGE} AS builder - -# Build -# ---------------------- -RUN apt-get update && apt-get install -y --no-install-recommends \ - git \ - build-essential \ - python3-venv \ - && python3 -m venv /opt/venv \ - && . /opt/venv/bin/activate \ - && git clone https://github.com/jwohlwend/boltz.git /app \ - && pip install --no-cache-dir --upgrade pip \ - && pip install --no-cache-dir torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu \ - && pip install --no-cache-dir boltz \ - && apt-get purge -y git build-essential \ - && apt-get autoremove -y \ - && apt-get clean \ - && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* - -# Metadata -# ---------------------- -FROM ${BASE_IMAGE} - -LABEL base_image=${BASE_IMAGE} \ - software=${PROJECT_NAME} \ - about.home=${PROJECT_URL} \ - about.summary="Biomolecular interaction model for 3D protein structure prediction" \ - about.tags="Model" \ - about.license="MIT" \ - about.license_file="https://github.com/jwohlwend/boltz/blob/main/LICENSE" \ - version="1" - -# Configuration -# ---------------------- -ARG USERNAME=rare -ARG USER_UID=900 -ARG USER_GID=900 - -ENV DEBIAN_FRONTEND=noninteractive -ENV LANG=C.UTF-8 -ENV PATH="/opt/venv/bin:$PATH" - -# Installation -# ---------------------- -RUN groupadd --gid $USER_GID $USERNAME && \ - useradd --uid $USER_UID --gid $USER_GID --create-home --shell /bin/bash $USERNAME - -COPY --from=builder --chown=$USER_UID:$USER_GID /opt/venv /opt/venv -COPY --from=builder --chown=$USER_UID:$USER_GID /app /app - -WORKDIR /app -USER $USERNAME -ENTRYPOINT ["boltz"] diff --git a/metadata.rules.cue b/metadata.rules.cue new file mode 100644 index 0000000..d8b53ae --- /dev/null +++ b/metadata.rules.cue @@ -0,0 +1,19 @@ +#Spec: { + app: #AppName + semver?: bool + channels: [...#Channels] +} + +#Channels: { + name: #ChannelName + platforms: [...#Platforms] + stable: bool + tests: { + enabled: bool + type?: =~"^(cli|web)$" + } +} + +#AppName: string & !="" & =~"^[a-zA-Z0-9_-]+$" +#ChannelName: string & !="" & =~"^[a-zA-Z0-9._-]+$" +#Platforms: "linux/amd64" | "linux/arm64" diff --git a/qlora/11.8.0-cudnn8-runtime-ubuntu22.04/Dockerfile b/qlora/11.8.0-cudnn8-runtime-ubuntu22.04/Dockerfile deleted file mode 100644 index 9ecc2a2..0000000 --- a/qlora/11.8.0-cudnn8-runtime-ubuntu22.04/Dockerfile +++ /dev/null @@ -1,63 +0,0 @@ -# Base image -# ---------------------- -ARG PROJECT_NAME=qlora -ARG PROJECT_URL=https://github.com/artidoro/qlora -ARG BASE_IMAGE=nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu22.04 -FROM ${BASE_IMAGE} AS builder - -# Build -# ---------------------- -RUN apt-get update && apt-get install -y --no-install-recommends \ - git \ - build-essential \ - python3 \ - python3-venv \ - && python3 -m venv /opt/venv \ - && . /opt/venv/bin/activate \ - && git clone https://github.com/artidoro/qlora /app \ - && pip install --no-cache-dir --upgrade pip \ - && pip install --no-cache-dir torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 \ - && pip install --no-cache-dir -r /app/requirements.txt \ - && apt-get purge -y git build-essential \ - && apt-get autoremove -y \ - && apt-get clean \ - && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* - -# Metadata -# ---------------------- -FROM ${BASE_IMAGE} - -LABEL base_image=${BASE_IMAGE} \ - software=${PROJECT_NAME} \ - about.home=${PROJECT_URL} \ - about.summary="QLoRA: Efficient Finetuning of Quantized LLMs" \ - about.tags="Model" \ - about.license="MIT" \ - about.license_file="https://github.com/artidoro/qlora" \ - version="1" - -# Configuration -# ---------------------- -ARG USERNAME=rare -ARG USER_UID=900 -ARG USER_GID=900 - -ENV DEBIAN_FRONTEND=noninteractive -ENV LANG=C.UTF-8 -ENV PATH="/opt/venv/bin:$PATH" - -# Installation -# ---------------------- -RUN groupadd --gid $USER_GID $USERNAME && \ - useradd --uid $USER_UID --gid $USER_GID --create-home --shell /bin/bash $USERNAME \ - && apt-get update && apt-get install -y --no-install-recommends python3 \ - && apt-get autoremove -y \ - && apt-get clean \ - && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* - -COPY --from=builder --chown=$USER_UID:$USER_GID /opt/venv /opt/venv -COPY --from=builder --chown=$USER_UID:$USER_GID /app /app - -WORKDIR /app -USER $USERNAME -ENTRYPOINT ["python /app/qlora.py"] diff --git a/qlora/12.1.0-cudnn8-runtime-ubuntu22.04/Dockerfile b/qlora/12.1.0-cudnn8-runtime-ubuntu22.04/Dockerfile deleted file mode 100644 index 1bbdbfd..0000000 --- a/qlora/12.1.0-cudnn8-runtime-ubuntu22.04/Dockerfile +++ /dev/null @@ -1,63 +0,0 @@ -# Base image -# ---------------------- -ARG PROJECT_NAME=qlora -ARG PROJECT_URL=https://github.com/artidoro/qlora -ARG BASE_IMAGE=nvidia/cuda:12.1.0-cudnn8-runtime-ubuntu22.04 -FROM ${BASE_IMAGE} AS builder - -# Build -# ---------------------- -RUN apt-get update && apt-get install -y --no-install-recommends \ - git \ - build-essential \ - python3 \ - python3-venv \ - && python3 -m venv /opt/venv \ - && . /opt/venv/bin/activate \ - && git clone https://github.com/artidoro/qlora /app \ - && pip install --no-cache-dir --upgrade pip \ - && pip install --no-cache-dir torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121 \ - && pip install --no-cache-dir -r /app/requirements.txt \ - && apt-get purge -y git build-essential \ - && apt-get autoremove -y \ - && apt-get clean \ - && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* - -# Metadata -# ---------------------- -FROM ${BASE_IMAGE} - -LABEL base_image=${BASE_IMAGE} \ - software=${PROJECT_NAME} \ - about.home=${PROJECT_URL} \ - about.summary="QLoRA: Efficient Finetuning of Quantized LLMs" \ - about.tags="Model" \ - about.license="MIT" \ - about.license_file="https://github.com/artidoro/qlora" \ - version="1" - -# Configuration -# ---------------------- -ARG USERNAME=rare -ARG USER_UID=900 -ARG USER_GID=900 - -ENV DEBIAN_FRONTEND=noninteractive -ENV LANG=C.UTF-8 -ENV PATH="/opt/venv/bin:$PATH" - -# Installation -# ---------------------- -RUN groupadd --gid $USER_GID $USERNAME && \ - useradd --uid $USER_UID --gid $USER_GID --create-home --shell /bin/bash $USERNAME \ - && apt-get update && apt-get install -y --no-install-recommends python3 \ - && apt-get autoremove -y \ - && apt-get clean \ - && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* - -COPY --from=builder --chown=$USER_UID:$USER_GID /opt/venv /opt/venv -COPY --from=builder --chown=$USER_UID:$USER_GID /app /app - -WORKDIR /app -USER $USERNAME -ENTRYPOINT ["python /app/qlora.py"] diff --git a/qlora/12.4.1-cudnn-runtime-ubuntu22.04/Dockerfile b/qlora/12.4.1-cudnn-runtime-ubuntu22.04/Dockerfile deleted file mode 100644 index 43f663d..0000000 --- a/qlora/12.4.1-cudnn-runtime-ubuntu22.04/Dockerfile +++ /dev/null @@ -1,63 +0,0 @@ -# Base image -# ---------------------- -ARG PROJECT_NAME=qlora -ARG PROJECT_URL=https://github.com/artidoro/qlora -ARG BASE_IMAGE=nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04 -FROM ${BASE_IMAGE} AS builder - -# Build -# ---------------------- -RUN apt-get update && apt-get install -y --no-install-recommends \ - git \ - build-essential \ - python3 \ - python3-venv \ - && python3 -m venv /opt/venv \ - && . /opt/venv/bin/activate \ - && git clone https://github.com/artidoro/qlora /app \ - && pip install --no-cache-dir --upgrade pip \ - && pip install --no-cache-dir torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124 \ - && pip install --no-cache-dir -r /app/requirements.txt \ - && apt-get purge -y git build-essential \ - && apt-get autoremove -y \ - && apt-get clean \ - && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* - -# Metadata -# ---------------------- -FROM ${BASE_IMAGE} - -LABEL base_image=${BASE_IMAGE} \ - software=${PROJECT_NAME} \ - about.home=${PROJECT_URL} \ - about.summary="QLoRA: Efficient Finetuning of Quantized LLMs" \ - about.tags="Model" \ - about.license="MIT" \ - about.license_file="https://github.com/artidoro/qlora" \ - version="1" - -# Configuration -# ---------------------- -ARG USERNAME=rare -ARG USER_UID=900 -ARG USER_GID=900 - -ENV DEBIAN_FRONTEND=noninteractive -ENV LANG=C.UTF-8 -ENV PATH="/opt/venv/bin:$PATH" - -# Installation -# ---------------------- -RUN groupadd --gid $USER_GID $USERNAME && \ - useradd --uid $USER_UID --gid $USER_GID --create-home --shell /bin/bash $USERNAME \ - && apt-get update && apt-get install -y --no-install-recommends python3 \ - && apt-get autoremove -y \ - && apt-get clean \ - && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* - -COPY --from=builder --chown=$USER_UID:$USER_GID /opt/venv /opt/venv -COPY --from=builder --chown=$USER_UID:$USER_GID /app /app - -WORKDIR /app -USER $USERNAME -ENTRYPOINT ["python /app/qlora.py"] diff --git a/qlora/README.md b/qlora/README.md deleted file mode 100644 index e7bfedf..0000000 --- a/qlora/README.md +++ /dev/null @@ -1,3 +0,0 @@ -From [artidoro/qlora](https://github.com/artidoro/qlora): - -> We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. QLoRA backpropagates gradients through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters (LoRA). Our best model family, which we name Guanaco, outperforms all previous openly released models on the Vicuna benchmark, reaching 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU. QLoRA introduces a number of innovations to save memory without sacrificing performance: (a) 4-bit NormalFloat (NF4), a new data type that is information theoretically optimal for normally distributed weights (b) Double Quantization to reduce the average memory footprint by quantizing the quantization constants, and (c) Paged Optimizers to manage memory spikes. We use QLoRA to finetune more than 1,000 models, providing a detailed analysis of instruction following and chatbot performance across 8 instruction datasets, multiple model types (LLaMA, T5), and model scales that would be infeasible to run with regular finetuning (e.g. 33B and 65B parameter models). Our results show that QLoRA finetuning on a small high-quality dataset leads to state-of-the-art results, even when using smaller models than the previous SoTA. We provide a detailed analysis of chatbot performance based on both human and GPT-4 evaluations showing that GPT-4 evaluations are a cheap and reasonable alternative to human evaluation. Furthermore, we find that current chatbot benchmarks are not trustworthy to accurately evaluate the performance levels of chatbots. We release all of our models and code, including CUDA kernels for 4-bit training. diff --git a/qlora/python3.11-slim/Dockerfile b/qlora/python3.11-slim/Dockerfile deleted file mode 100644 index 6d05448..0000000 --- a/qlora/python3.11-slim/Dockerfile +++ /dev/null @@ -1,58 +0,0 @@ -# Base image -# ---------------------- -ARG PROJECT_NAME=qlora -ARG PROJECT_URL=https://github.com/artidoro/qlora -ARG BASE_IMAGE=python:3.11-slim -FROM ${BASE_IMAGE} AS builder - -# Build -# ---------------------- -RUN apt-get update && apt-get install -y --no-install-recommends \ - git \ - build-essential \ - python3-venv \ - && python3 -m venv /opt/venv \ - && . /opt/venv/bin/activate \ - && git clone https://github.com/artidoro/qlora /app \ - && pip install --no-cache-dir --upgrade pip \ - && pip install --no-cache-dir torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu \ - && pip install --no-cache-dir -r /app/requirements.txt \ - && apt-get purge -y git build-essential \ - && apt-get autoremove -y \ - && apt-get clean \ - && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* - -# Metadata -# ---------------------- -FROM ${BASE_IMAGE} - -LABEL base_image=${BASE_IMAGE} \ - software=${PROJECT_NAME} \ - about.home=${PROJECT_URL} \ - about.summary="QLoRA: Efficient Finetuning of Quantized LLMs" \ - about.tags="Model" \ - about.license="MIT" \ - about.license_file="https://github.com/artidoro/qlora" \ - version="1" - -# Configuration -# ---------------------- -ARG USERNAME=rare -ARG USER_UID=900 -ARG USER_GID=900 - -ENV DEBIAN_FRONTEND=noninteractive -ENV LANG=C.UTF-8 -ENV PATH="/opt/venv/bin:$PATH" - -# Installation -# ---------------------- -RUN groupadd --gid $USER_GID $USERNAME && \ - useradd --uid $USER_UID --gid $USER_GID --create-home --shell /bin/bash $USERNAME - -COPY --from=builder --chown=$USER_UID:$USER_GID /opt/venv /opt/venv -COPY --from=builder --chown=$USER_UID:$USER_GID /app /app - -WORKDIR /app -USER $USERNAME -ENTRYPOINT ["python /app/qlora.py"] diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..661361b --- /dev/null +++ b/requirements.txt @@ -0,0 +1,4 @@ +requests==2.32.3 +pyyaml==6.0.2 +packaging==24.2 +jinja2==3.1.4 diff --git a/scripts/actions/prepare-matrices.py b/scripts/actions/prepare-matrices.py new file mode 100644 index 0000000..c07f747 --- /dev/null +++ b/scripts/actions/prepare-matrices.py @@ -0,0 +1,221 @@ +#!/usr/bin/env python3 +import importlib.util +import json +import os +import sys +from os.path import isfile +from subprocess import check_output + +import requests +import yaml + +repo_owner = os.environ.get("REPO_OWNER", os.environ.get("GITHUB_REPOSITORY_OWNER")) + +TESTABLE_PLATFORMS = ["linux/amd64"] + + +def load_metadata_file_yaml(file_path): + with open(file_path, "r") as f: + return yaml.safe_load(f) + + +def load_metadata_file_json(file_path): + with open(file_path, "r") as f: + return json.load(f) + + +def get_latest_version_py(latest_py_path, channel_name): + spec = importlib.util.spec_from_file_location("latest", latest_py_path) + latest = importlib.util.module_from_spec(spec) + sys.modules["latest"] = latest + spec.loader.exec_module(latest) + return latest.get_latest(channel_name) + + +def get_latest_version_sh(latest_sh_path, channel_name): + out = check_output([latest_sh_path, channel_name]) + return out.decode("utf-8").strip() + + +def get_latest_version(subdir, channel_name): + ci_dir = os.path.join(subdir, "ci") + if os.path.isfile(os.path.join(ci_dir, "latest.py")): + return get_latest_version_py(os.path.join(ci_dir, "latest.py"), channel_name) + elif os.path.isfile(os.path.join(ci_dir, "latest.sh")): + return get_latest_version_sh(os.path.join(ci_dir, "latest.sh"), channel_name) + elif os.path.isfile(os.path.join(subdir, channel_name, "latest.py")): + return get_latest_version_py( + os.path.join(subdir, channel_name, "latest.py"), channel_name + ) + elif os.path.isfile(os.path.join(subdir, channel_name, "latest.sh")): + return get_latest_version_sh( + os.path.join(subdir, channel_name, "latest.sh"), channel_name + ) + return None + + +def get_published_version(image_name): + r = requests.get( + f"https://api.github.com/users/{repo_owner}/packages/container/{image_name}/versions", + headers={ + "Accept": "application/vnd.github.v3+json", + "Authorization": "token " + os.environ["TOKEN"], + }, + ) + + if r.status_code != 200: + return None + + data = json.loads(r.text) + for image in data: + tags = image["metadata"]["container"]["tags"] + if "rolling" in tags: + tags.remove("rolling") + # Assume the longest string is the complete version number + return max(tags, key=len) + + +def get_image_metadata(subdir, meta, forRelease=False, force=False, channels=None): + imagesToBuild = {"images": [], "imagePlatforms": []} + + if channels is None: + channels = meta["channels"] + else: + channels = [ + channel for channel in meta["channels"] if channel["name"] in channels + ] + + for channel in channels: + version = get_latest_version(subdir, channel["name"]) + if version is None: + continue + + # Image Name + toBuild = {} + if channel.get("stable", False): + toBuild["name"] = meta["app"] + else: + toBuild["name"] = "-".join([meta["app"], channel["name"]]) + + # Skip if latest version already published + if not force: + published = get_published_version(toBuild["name"]) + if published is not None and published == version: + continue + toBuild["published_version"] = published + + toBuild["version"] = version + + # Image Tags + toBuild["tags"] = ["rolling", version] + if meta.get("semver", False): + parts = version.split(".")[:-1] + while len(parts) > 0: + toBuild["tags"].append(".".join(parts)) + parts = parts[:-1] + + # Platform Metadata + for platform in channel["platforms"]: + + if platform not in TESTABLE_PLATFORMS and not forRelease: + continue + + toBuild.setdefault("platforms", []).append(platform) + + target_os = platform.split("/")[0] + target_arch = platform.split("/")[1] + + platformToBuild = {} + platformToBuild["name"] = toBuild["name"] + platformToBuild["platform"] = platform + platformToBuild["target_os"] = target_os + platformToBuild["target_arch"] = target_arch + platformToBuild["version"] = version + platformToBuild["channel"] = channel["name"] + platformToBuild["label_type"] = "org.opencontainers.image" + + if isfile(os.path.join(subdir, channel["name"], "Dockerfile")): + platformToBuild["dockerfile"] = os.path.join( + subdir, channel["name"], "Dockerfile" + ) + platformToBuild["context"] = os.path.join(subdir, channel["name"]) + platformToBuild["goss_config"] = os.path.join( + subdir, channel["name"], "goss.yaml" + ) + else: + platformToBuild["dockerfile"] = os.path.join(subdir, "Dockerfile") + platformToBuild["context"] = subdir + platformToBuild["goss_config"] = os.path.join(subdir, "ci", "goss.yaml") + + platformToBuild["goss_args"] = ( + "tail -f /dev/null" + if channel["tests"].get("type", "web") == "cli" + else "" + ) + + platformToBuild["tests_enabled"] = ( + channel["tests"]["enabled"] and platform in TESTABLE_PLATFORMS + ) + + imagesToBuild["imagePlatforms"].append(platformToBuild) + imagesToBuild["images"].append(toBuild) + return imagesToBuild + + +if __name__ == "__main__": + apps = sys.argv[1] + forRelease = sys.argv[2] == "true" + force = sys.argv[3] == "true" + imagesToBuild = {"images": [], "imagePlatforms": []} + + if apps != "all": + channels = None + apps = apps.split(",") + if len(sys.argv) == 5: + channels = sys.argv[4].split(",") + + for app in apps: + if not os.path.exists(os.path.join("./apps", app)): + print(f'App "{app}" not found') + exit(1) + + meta = None + if os.path.isfile(os.path.join("./apps", app, "metadata.yaml")): + meta = load_metadata_file_yaml( + os.path.join("./apps", app, "metadata.yaml") + ) + elif os.path.isfile(os.path.join("./apps", app, "metadata.json")): + meta = load_metadata_file_json( + os.path.join("./apps", app, "metadata.json") + ) + + imageToBuild = get_image_metadata( + os.path.join("./apps", app), + meta, + forRelease, + force=force, + channels=channels, + ) + if imageToBuild is not None: + imagesToBuild["images"].extend(imageToBuild["images"]) + imagesToBuild["imagePlatforms"].extend(imageToBuild["imagePlatforms"]) + else: + for subdir, dirs, files in os.walk("./apps"): + for file in files: + meta = None + if file == "metadata.yaml": + meta = load_metadata_file_yaml(os.path.join(subdir, file)) + elif file == "metadata.json": + meta = load_metadata_file_json(os.path.join(subdir, file)) + else: + continue + if meta is not None: + imageToBuild = get_image_metadata( + subdir, meta, forRelease, force=force + ) + if imageToBuild is not None: + imagesToBuild["images"].extend(imageToBuild["images"]) + imagesToBuild["imagePlatforms"].extend( + imageToBuild["imagePlatforms"] + ) + print(json.dumps(imagesToBuild)) diff --git a/scripts/actions/render-readme.py b/scripts/actions/render-readme.py new file mode 100644 index 0000000..b961c7a --- /dev/null +++ b/scripts/actions/render-readme.py @@ -0,0 +1,66 @@ +#!/usr/bin/env python3 +import logging +import os + +import yaml +from jinja2 import Environment, PackageLoader, select_autoescape + +logging.basicConfig(level=logging.INFO) + +repo_owner = os.getenv("REPO_OWNER") or os.getenv( + "GITHUB_REPOSITORY_OWNER", "default_owner" +) +repo_name = os.getenv("REPO_NAME") or os.getenv("GITHUB_REPOSITORY", "default_repo") + +env = Environment(loader=PackageLoader("render-readme"), autoescape=select_autoescape()) + + +def load_metadata(file_path): + try: + with open(file_path, "r") as f: + return yaml.safe_load(f) + except yaml.YAMLError as e: + logging.error(f"Error loading YAML file {file_path}: {e}") + except FileNotFoundError: + logging.error(f"File {file_path} not found.") + return None + + +def process_metadata(apps_dir): + app_images = [] + for subdir, _, files in os.walk(apps_dir): + if "metadata.yaml" not in files: + continue # Skip if metadata file not found + + meta = load_metadata(os.path.join(subdir, "metadata.yaml")) + if not meta: + continue # Skip if metadata couldn't be loaded + + # Iterate through the channels and build image metadata + for channel in meta.get("channels", []): + name = ( + meta["app"] + if channel.get("stable", False) + else f"{meta['app']}-{channel['name']}" + ) + image = { + "name": name, + "channel": channel["name"], + "html_url": f"https://github.com/{repo_owner}/pkgs/container/{name}", + "owner": repo_owner, + } + app_images.append(image) + logging.info(f"Added image {name} from channel {channel['name']}") + return app_images + + +if __name__ == "__main__": + apps_dir = "./apps" + app_images = process_metadata(apps_dir) + try: + template = env.get_template("README.md.j2") + with open("./README.md", "w") as f: + f.write(template.render(app_images=app_images)) + logging.info("README.md successfully generated.") + except Exception as e: + logging.error(f"Error rendering template: {e}") diff --git a/scripts/templates/README.md.j2 b/scripts/templates/README.md.j2 new file mode 100644 index 0000000..2c6610a --- /dev/null +++ b/scripts/templates/README.md.j2 @@ -0,0 +1,119 @@ + +
+ + +## Containers + +Collection of bioinformatics containers + +
+ +
+ +![GitHub Repo stars](https://img.shields.io/github/stars/RareCompute/containers?style=for-the-badge) +![GitHub forks](https://img.shields.io/github/forks/RareCompute/containers?style=for-the-badge) +![GitHub Workflow Status (with event)](https://img.shields.io/github/actions/workflow/status/RareCompute/containers/release-scheduled.yaml?style=for-the-badge&label=Scheduled%20Release) + +
+ +## Credits + +The CI/CD and format of this repository was adapted from onedr0p's [containers](https://github.com/onedr0p/containers). From onedr0p: A lot of inspiration and ideas are thanks to the hard work of [hotio.dev](https://hotio.dev/) and [linuxserver.io](https://www.linuxserver.io/) contributors. + +## Tag immutability + +The containers built here do not use immutable tags, as least not in the more common way you have seen from [linuxserver.io](https://fleet.linuxserver.io/) or [Bitnami](https://bitnami.com/stacks/containers). + +We do take a similar approach but instead of appending a `-ls69` or `-r420` prefix to the tag we instead insist on pinning to the sha256 digest of the image, while this is not as pretty it is just as functional in making the images immutable. + +| Container | Immutable | +|----------------------------------------------------|-----------| +| `ghcr.io/RareCompute/boltz:rolling` | ❌ | +| `ghcr.io/RareCompute/boltz:1.0.0` | ❌ | +| `ghcr.io/RareCompute/boltz:rolling@sha256:...` | ✅ | +| `ghcr.io/RareCompute/boltz:1.0.0@sha256:...` | ✅ | + +_If pinning an image to the sha256 digest, tools like [Renovate](https://github.com/renovatebot/renovate) support updating the container on a digest or application version change._ + +## Rootless + +To run these containers as non-root make sure you update your configuration to the user and group you want. + +### Docker compose + +```yaml +networks: + boltz: + name: boltz + external: true +services: + boltz: + image: ghcr.io/RareCompute/boltz:1.0.0 + container_name: boltz + user: 65534:65534 + # ... +``` + +### Kubernetes + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: boltz +# ... +spec: + # ... + template: + # ... + spec: + # ... + securityContext: + runAsUser: 65534 + runAsGroup: 65534 + fsGroup: 65534 + fsGroupChangePolicy: OnRootMismatch +# ... +``` + +## Passing arguments to an application + +Some applications do not support defining configuration via environment variables and instead only allow certain config to be set in the command line arguments for the app. To circumvent this, for applications that have an `entrypoint.sh` read below. + +1. First read the Kubernetes docs on [defining command and arguments for a Container](https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/). +2. Look up the documentation for the application and find a argument you would like to set. +3. Set the extra arguments in the `args` section like below. + + ```yaml + args: + - --port + - "8080" + ``` + +## Configuration volume + +For applications that need to have persistent configuration data the config volume is hardcoded to `/config` inside the container. This is not able to be changed in most cases. + +## Available Images + +Each Image will be built with a `rolling` tag, along with tags specific to it's version. Available Images Below + +Container | Channel | Image +--- | --- | --- +{% for image in app_images | sort(attribute="name") -%} +[{{ image.name }}]({{ image.html_url }}) | {{ image.channel }} | ghcr.io/{{ image.owner }}/{{ image.name }} +{% endfor %} + +## Deprecations + +Containers here can be **deprecated** at any point, this could be for any reason described below. + +1. The upstream application is **no longer actively developed** +2. The upstream application has an **official upstream container** that follows closely to the mission statement described here +3. The upstream application has been **replaced with a better alternative** +4. The **maintenance burden** of keeping the container here **is too bothersome** + +**Note**: Deprecated containers will remained published to this repo for 6 months after which they will be pruned. diff --git a/scripts/wip.sh b/scripts/wip.sh new file mode 100644 index 0000000..1dc001e --- /dev/null +++ b/scripts/wip.sh @@ -0,0 +1,43 @@ +#!/bin/bash + +# Modified version of brettinternet's wip.sh +# https://github.com/brettinternet/homeops/blob/main/scripts/wip.sh +# +# Usage: +# create WIP commit +# push to current branch +# remove previous WIP commit +# + +# Source: https://github.com/ohmyzsh/ohmyzsh/blob/1546e1226a7b739776bda43f264b221739ba0397/lib/git.zsh#L68-L81 +function git_current_branch { + local ref + ref=$(command git symbolic-ref --quiet HEAD 2> /dev/null) + local ret=$? + if [[ $ret != 0 ]]; then + [[ $ret == 128 ]] && return # no git repo. + ref=$(command git rev-parse --short HEAD 2> /dev/null) || return + fi + echo "${ref#refs/heads/}" +} + +function gwip { + git add -A + git rm $(git ls-files --deleted) 2> /dev/null + git commit --no-gpg-sign -m "--wip-- [skipci]" +} + +function gunwip { + git log -n 1 | grep -q -c "\-\-wip\-\-" && git reset HEAD~1 +} + +function gpcf { + git push --force-with-lease origin "$(git_current_branch)" +} + +gwip +gpcf +echo "" +echo "Git Log:" +gunwip +echo ""