Treewide: Develop a way to tag auxiliary dependencies (such as checkdeps for tests) for autobuild resolver purposes

[ermo]

Now that @GZGavinZhao is putting the final touches on his autobuild dependency (and reverse dependency) resolver tool, he has begun doing some analysis runs on the package recipes in the monorepo.

As it turns out, there are quite a few dependencies that don't actually need to be included as builddeps in the forward and reverse dependency graph for packages, but should instead be moved to a separate kind of dependency and/or be put in a denylist so they don't get included in the buildgraph unnecessarily.

Here's a bit more context from a development discussion I had with Gavin on this:

ermo
Gavin: What is the overarching goal here for you re. checkdeps (and its ilk)?
(I'm asking in the hope that we might arrive on a better, more encompassing name than just checkdeps)
Gavin Zhao
checkdeps, at least in the way I'm currently using it, has sort of becoming an overarching term that encompass basically any dependency that the package doesn't strictly rely on and therefore can be tossed out when considering build order
so this includes not only dependencies for tests, but also for documentation generation
doxygen is a common one here
because it seems like the entire world depends on LLVM right now 😅
ermo
secondary deps?
no, that doesn't explain the issue
auxiliary deps?
Gavin Zhao
auxdeps?
ermo
yeah, let's go with that.
(if you wrote that before you saw my suggestion, then ^5)
Gavin Zhao

ermo
(if you wrote that before you saw my suggestion, then ^5)

I did write it before you lol, but maybe someone has an idea for a better name?
ermo
"The term 'Auxiliary Dependencies' (auxdeps for short) covers deps such as checkdeps and other auxiliary deps (such as doxygen) that aren't directly relevant to the dependency graph when figuring out what to build and what to rebuild" <- maybe something like that?
I think auxdeps is good. It's short and it covers everything that isn't a builddep or a rundep
Gavin Zhao

ermo
"The term 'Auxiliary Dependencies' (auxdeps for short) covers deps such as checkdeps and other auxiliary deps (such as doxygen) that aren't directly relevant to the dependency graph when figuring out what to build and what to rebuild" <- maybe something like that?

That sounds good!
ermo
OK. Let's run with that for now in the issue + related PR?
Gavin Zhao
Yep!
(...)
ermo
Gavin: Should we have a docdeps field? Meaning that auxdeps would be a term that covers both actual recipe docdeps and checkdeps keys?
ermo
Ikey: ^ Possibly relevant for serpent as well.
ermo
in terms of minimising build graphs.
i.e. legit "tooling that makes life easier for packagers and maintainers"-driven addition to the recipe format.
Gavin Zhao
That was my initial thought, the concern was maybe in the future we will discover additional types of dependencies that fit into neither checkdeps nor docdeps?

As can be seen above, for now (and to have a way to begin to refer to this concept), Gavin and I have tentatively settled on using the term 'Auxiliary Dependencies' (or auxdeps for short) to refer to these kinds of deps, which constitute e.g. checkdeps and other auxiliary deps (such as doxygen) that aren't directly relevant to the dependency graph when autobuild is resolving and planning forward and reverse build orders based on its dependency traversal code.

This term is mostly a "working term for now", so if we end deciding to adopt a better term for the concept, that's fine too.

This issue tracks the PRs and issues we will begin filing in relation to solving this problem on a treewide basis.

Proposed implementation:

Add the ability for ypkg (and boulder) to interpret "Auxiliary Dependencies" that have an annotation tacked on and can be put in existing dependency keys like so:

builddeps:
    - asciidoc: { docdep: true }
    - doxygen: { docdep: true }
    - git: { auxdep: true} # some build systems use this to embed the git ref next to the version displayed
    - python-pytest { checkdep: true }
    - python-sphinx: { docdep: true }

Here, the implication is that, though doxygen is used to generate API documentation etc. at build-time, it is not considered a part of the actual forward and reverse dependency chain as it pertains to planning build and revdep-rebuild orders

Issues:

• moss-format: Support Auxiliary Dependency annotations serpent-os/tools#206 (since @EbonJaeger asked "can't we just do what serpent does...?)

PRs:

• samba: Add autobuild.yml files for dep resolution #2279
• [NFC] many: migrate some python packages to use checkdeps #2174
• autobuild: initial manual cycle breaking #637

[EbonJaeger]

Soo.... we're no longer having checkdeps?

When you say "we decided" I think it means "two people decided", and not "the project as a whole decided". I would like more of a discussion on this with more people that do packaging and work on packaging tools about all of this. I, for one, am not totally convinced about having a new prefix for existing deps or what it should be called, let alone a whole new key in the packaging spec. Prefixing dependencies with aux is not very intuitive; I'm finding it a bit confusing, so I worry that it will be even worse for packagers with even less experience, or new packagers.

[ermo]

@EbonJaeger
When you say "we decided" I think it means "two people decided", and not "the project as a whole decided".

Gavin and I decided to start referring to it as Auxiliary Deps or auxdeps for short to have a way to express and refer to the concept, based on Gavin's experience up until now with autobuild.

I would like more of a discussion on this with more people that do packaging and work on packaging tools about all of this.

This is what this issue is for.

I, for one, am not totally convinced about having a new prefix for existing deps or what it should be called, let alone a whole new key in the packaging spec. Prefixing dependencies with aux is not very intuitive; I'm finding it a bit confusing, so I worry that it will be even worse for packagers with even less experience, or new packagers.

What is your alternative proposal?

Soo.... we're no longer having checkdeps?

Checkdeps were introduced after Gavin pointed out some initial results of his autobuild research. Another way of solving the problem would be to use a prefix (similar to the suggested aux one), so maybe test(somedep) or check(somedep)?

I would like to note that Serpent uses prefixes extensively already as a well-established concept, so this would fit well with the Serpent tooling model that Solus has agreed to move towards...?

[ermo]

@EbonJaeger I have updated the initial text to better reflect the idea behind the auxdeps naming and the scope it covers.

We have to start somewhere. 🤔 😅