Weekly Auto-PR via Dependabot for updating dependancies

[alextreme]

Thema / Theme

Other

Omschrijving / Description

Discussed with @sergei-maertens . Most of our Maykin / Team Bron repositories make use of the dependabot alerts, however we disable the default PR-creating feature of dependabot because you get 1 PR per security update which drives everyone insane.

I'd like to try out configuring Dependabot in a smarter way, namely to auto-create on a weekly basis a single PR to update all pypi dependancies in one go. This seems possible via the groups mechanism:

https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#groups

Example 3 shows how this can be done on a weekly basis for npm:

https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#example-3

This should reduce the amount of noise and help us out into autopatching of dependancies with security advisories. Maybe updates should be limited to minor+patch versions to avoid automatically taking major version upgrades into this PR.

The same could also be done for npm dependancies if applicable. Please experiment with configuring dependabot via the yml file

Toegevoegde waarde / Added value

No response

Aanvullende opmerkingen / Additional context

No response

[sam-bertin]

Dependabot Configuration Report

Configuration Overview

The current Dependabot configuration is designed to manage dependency updates for both pip and npm ecosystems. The configuration is structured as follows:

version: 2
updates:
  - package-ecosystem: "pip"
    directory: "/requirements"
    versioning-strategy: "auto"
    open-pull-requests-limit: 20
    schedule:
      interval: "weekly"
    groups:
      pypi-dependencies:
        patterns:
          - "*"
        update-types:
          - "minor"
          - "patch"

  - package-ecosystem: "npm"
    directory: "/"
    versioning-strategy: "auto"
    open-pull-requests-limit: 20
    schedule:
      interval: "weekly"
    groups:
      npm-dependencies:
        patterns:
          - "*"
        update-types:
          - "minor"
          - "patch"

Observed Behavior

Despite the configuration, Dependabot is not grouping all updates into a single pull request (PR) for each ecosystem. Instead, it is creating multiple PRs, including individual PRs for some dependencies. For example, for the npm ecosystem, the following PRs were observed:

• Bump the npm-dependencies group with 19 updates
  

• Bump @ckeditor/ckeditor5-table from 33.0.0 to 43.0.0
  

• Bump microscope-sass from 1.1.0 to 2.0.0
  

• Bump @ckeditor/ckeditor5-autoformat from 33.0.0 to 43.0.0
  

• ... and more (total of 20 PRs regarding npm, which corresponds to the limit set in the configuration)
  

Analysis

The observed behavior can be attributed to several factors:

• Grouping Limitations:
  Dependabot may not always be able to group all updates into a single PR due to version incompatibilities or other constraints.
  

• Individual Updates:
  Certain updates may be critical or have specific dependencies that require them to be handled individually to minimize conflicts or compatibility issues.
  

Python requirements

The same way Python requirements should have been parsed and grouped in a single PR but the current requirements couldn't be parsed with dependabot. https://github.com/sam-bertin/test-dependabot/network/updates/869830707

[sam-bertin]

Experimentation

version: 2
updates:
  - package-ecosystem: "npm"
    directory: "/"
    versioning-strategy: "auto"
    open-pull-requests-limit: 60
    schedule:
      interval: "weekly"
    groups:
      npm-dependencies:
        patterns:
          - "*"
        update-types:
          - "minor"
          - "patch"

Changes

• Bumped the limit of pull requests to 60
• Limited to npm only to clarify the results

Results

There were 38 npm security alerts with the following severity levels

It resulted in 8 pull requests :

Analysis

• A single action was launched to handle the security alerts. (https://github.com/sam-bertin/test-dependabot/actions/runs/10403571493/job/28810321903)
• Augmenting the open-pull-requests-limit does not affect the limit of dependencies in the group.
• 19 + 7 = 26 dependencies addressed out of the 38.
• No idea what happened with the latest 12.
• I also don't understand why I had much more PR for npm updates on the previous run.

---

After merging the pull requests it suggested there was 17 npm security alerts.

When relaunching the pipeline with :

version: 2
updates:
  - package-ecosystem: "npm"
    directory: "/"
    versioning-strategy: "auto"
    open-pull-requests-limit: 1
    schedule:
      interval: "weekly"
    groups:
      npm-dependencies:
        patterns:
          - "*"
        update-types:
          - "minor"
          - "patch"

It did not address any of the 17 security issues that were left and finished the job without errors.
No pull request was created from the process.
There are definite limitations.

[sam-bertin]

Experimentation

- package-ecosystem: "pip"
    directory: "/requirements"
    versioning-strategy: "auto"
    open-pull-requests-limit: 10
    schedule:
      interval: "weekly"
    groups:
      pypi-dev:
        patterns:
          - "dev.txt"
        update-types:
          - "minor"
          - "patch"
      pypi-staging:
        patterns:
          - "staging.txt"
        update-types:
          - "minor"
          - "patch"
      pypi-prod:
        patterns:
          - "production.txt"
        update-types:
          - "minor"
          - "patch"
      pypi-test:
        patterns:
          - "test.txt"
        update-types:
          - "minor"
          - "patch"
      pypi-ci:
        patterns:
          - "ci.txt"
        update-types:
          - "minor"
          - "patch"
      pypi-base:
        patterns:
          - "base.txt"
        update-types:
          - "minor"
          - "patch"

I decoupled the dependency files for python to figure out whether or not one file was causing the parsing problem.

This did not help, job still fails without much detail : https://github.com/sam-bertin/test-dependabot/actions/runs/10404169142/job/28812078285
https://github.com/sam-bertin/test-dependabot/network/updates/869836917

Here the example where I tried to only parse python deps from ONLY the requirements/base.txt:

version: 2
updates:
 - package-ecosystem: "npm"
   directory: "/"
   versioning-strategy: "auto"
   open-pull-requests-limit: 1
   schedule:
     interval: "weekly"
   groups:
     npm-dependencies:
       patterns:
         - "*"
       update-types:
         - "minor"
         - "patch"
 - package-ecosystem: "pip"
   directory: "/requirements"
   versioning-strategy: "auto"
   open-pull-requests-limit: 1
   schedule:
     interval: "weekly"
   groups:
     pypi-dependencies:
       patterns:
         - "base.txt"
       update-types:
         - "minor"
         - "patch"

This still sends us error and cannot parse the dependencies.

Dependabot failed to update your dependencies because there was an error evaluating your Python dependency files.

Dependabot encountered the following error:

InstallationError("Could not open requirements file: [Errno 2] No such file or directory: 'dependabot_tmp_dir/ci.txt'")

It seems weird to me that the ci.txt file is mentioned when I specified base.txt in the pattern argument

---

I have tried to strip up the case on maximum with only one .txt dependency file within a single folder.

  - package-ecosystem: "pip"
    directory: "/requirements/dependabot"
    versioning-strategy: "auto"
    open-pull-requests-limit: 1
    schedule:
      interval: "weekly"
    groups:
      pypi-dependencies:
        patterns:
          - "requirements.txt"
        update-types:
          - "minor"
          - "patch"

So it's not totally broken.

[sam-bertin]

Possibly linked issue

dependabot/dependabot-core#8380

[sam-bertin]

@alextreme this config is giving me the best results so far :

version: 2
updates:
  - package-ecosystem: "npm"
    directory: "/"
    versioning-strategy: "auto"
    open-pull-requests-limit: 1
    schedule:
      interval: "weekly"
    groups:
      npm-dependencies:
        patterns:
          - "*"
        update-types:
          - "minor"
          - "patch"
  - package-ecosystem: "pip"
    directory: "/"
    versioning-strategy: "auto"
    open-pull-requests-limit: 1
    schedule:
      interval: "weekly"
    groups:
      pypi-dependencies:
        patterns:
          - "*"
        exclude-patterns:
          - "ci.txt"  
        update-types:
          - "minor"
          - "patch"

---

Results observed

• 2 actions launched
  
• 2 Pull requests open
  

---

I have prepared this documentation to set it up :

Dependabot Configuration Documentation

Introduction

Dependabot is a tool that helps keep your dependencies up-to-date by automatically creating pull requests for updates. This documentation explains our Dependabot configuration, how it works, and how to set it up.

Configuration Overview

Our repository is set up with the following Dependabot configuration:

version: 2
updates:
  - package-ecosystem: "npm"
    directory: "/"
    versioning-strategy: "auto"
    open-pull-requests-limit: 1
    schedule:
      interval: "weekly"
    groups:
      npm-dependencies:
        patterns:
          - "*"
        update-types:
          - "minor"
          - "patch"
  - package-ecosystem: "pip"
    directory: "/"
    versioning-strategy: "auto"
    open-pull-requests-limit: 1
    schedule:
      interval: "weekly"
    groups:
      pypi-dependencies:
        patterns:
          - "*"
        exclude-patterns:
          - "ci.txt"
        update-types:
          - "minor"
          - "patch"

What It Does for You

• Automatic Updates: Dependabot will automatically create pull requests to update your dependencies.
• Weekly Schedule: Updates are checked and pull requests are created on a weekly basis.
• Grouped Updates: Dependencies are grouped to minimize the number of pull requests.
• Versioning Strategy: The versioning strategy is set to "auto," meaning Dependabot will use the most appropriate versioning strategy for each dependency.
• Open Pull Requests Limit: Only one pull request will be open at a time for each ecosystem to avoid overwhelming the review process.

How to Set It Up

• Create a .github Directory: If it doesn't already exist, create a .github directory in the root of your repository.
• Create a dependabot.yml File: Inside the .github directory, create a file named dependabot.yml.
• Add the Configuration: Copy the configuration provided above into the dependabot.yml file.
• Commit and Push: Commit the changes and push them to the repository.
• Activate Dependabot Version Updates (inside your repository settings):
  
• Avoid to activate the security updates or group updates before setting up your dependabot.yml file

Important Notes

• Excluding ci.txt: We have excluded ci.txt from the pip dependencies because there is no matching ci.in file. Ensure that both .txt and .in are present if you want them to be parsed (only works for version updates - security updated still flaky).

• Dependabot Documentation: For more detailed information, refer to the official Dependabot documentation.

• Use the groups option with the applies-to: security-updates key to create sets of dependencies (per package manager), so that Dependabot opens a single pull request to update multiple dependencies at the same time. You can define groups by package name (the patterns and exclude-patterns keys), dependency type (dependency-type key), and SemVer (the update-types key).
  Dependabot creates groups in the order they appear in your dependabot.yml file. If a dependency update could belong to more than one group, it is only assigned to the first group it matches with.

• If you only require security updates and want to exclude version updates, you can set open-pull-requests-limit to 0 in order to prevent version updates for a given package-ecosystem.

• Don't forget to enable grouped security updates if you also want them (but that's more PRs).

If you have any questions or need further assistance, please reach out.

[alextreme]

Great, thanks Samuel! We'll discuss it with Team Bron on tuesday

[sam-bertin]

It's probably a hot take but I feel like the way github handles security updates is bit of unpredictable and therefore painful to maintain.
The way version updates work is clearly more stable.
The last section (Important Notes) talks about how the version update config can be applied to the security-updates but there seem to be issues with it (pip deps did not parse for security while it did for version updates PRs).
I know that updating versions won't cover all security concerns but that will probably cover part of it so I would consider it an improvement.

[sergei-maertens]

Great, thanks Samuel! We'll discuss it with Team Bron on tuesday

To bring into this discussion: my main concert is that dependabot does not seem to respect our custom uv pip compile configuration - we do everything through a wrapper script and it seems not wise to me if dependabot starts messing directly in *.txt files or runs its own flavour/variant of pip-compile. If merging a dependabot PR and then running ./bin/compile_dependencies.sh produces a diff, it's going to cause weird diffs at unexpected times 🤔

[alextreme]

Samuel and I discussed exactly this, we may have to go for a bin/update_dependancies.sh script and do this ourselves instead of doing this via dependabot. We'll discuss the various approaches and try this out on one or two repositories to see what works

[sergei-maertens]

in case you didn't know: ./bin/compile_dependencies -P <package1> -P <package2> can already be used to bump only those packages. Note that you sometimes also need to bump some other package depending on it since the dependency resolution can limit the available versions. It's a hard problem to solve, might be worthwhile to look at the uv pip compile command line options too.

edit: and now I will unsubscribe from this issue :-)

[joeribekker]

Refinement: We still need to see how this behaviour with our pip-compile tooling (see sergeis comment above) but lets try it out for Objects API.

[joeribekker]

Lets evaluate its behaviour in the next refinmenent

[joeribekker]

Refinement: We're checking if #44 covers our needs first before picking up this ticket.