feat(pkgs/dmd): Automate definition of package versions #9
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PetarKirov
force-pushed
the
refactor/gen-pkg-versions
branch
2 times, most recently
from
8057991 to
7733c3d
Compare
dukc
previously requested changes
Oct 7, 2023
Closed
PetarKirov
force-pushed
the
refactor/gen-pkg-versions
branch
from
7733c3d to
b90d46d
Compare
PetarKirov
force-pushed
the
refactor/gen-pkg-versions
branch
6 times, most recently
from
0f09d56 to
fd109c3
Compare
dukc
reviewed
Jan 25, 2024
PetarKirov
force-pushed
the
refactor/gen-pkg-versions
branch
2 times, most recently
from
9745022 to
998f679
Compare
PetarKirov
force-pushed
the
refactor/gen-pkg-versions
branch
5 times, most recently
from
c61978d to
74a50c5
Compare
It seems to add more overhead then it actually saves time.
* style: re-order imports * fix: handling of `--help` and invalid cmd-line args * improve: pass-through stderr of child processes * improve: separate logging vs result output by using stderr and stdout as appropriate
Done with this command: ``` ./scripts/fetch_binary.d \ --compiler=dmd \ --versions=2.090.1,2.098.0 \ --dry-run=false \ | jq '.' > \ ./pkgs/dmd/supported-binary-versions.json ```
This commits adds the following dmd-binary versions: * 2.079.1 * 2.080.1 * 2.081.2 * 2.082.1 * 2.083.1 * 2.084.1 * 2.085.1 * 2.086.1 * 2.087.1 * 2.088.1 * 2.089.1 This was done with the following command: ``` ./scripts/fetch_binary.d \ --compiler=dmd \ --versions=2.079.1,2.080.1,2.081.2,2.082.1,2.083.1,2.084.1,2.085.1,2.086.1,2.087.1,2.088.1,2.089.1,2.090.1,2.098.0 \ --dry-run=false \ | jq '.' > \ ./pkgs/dmd/supported-binary-versions.json ``` The versions 2.079.1 - 2.083.1 (including) require linking in libstdc++ which is available from `stdenv.cc.cc.lib`.
Why is this change is necessary?
=============================
Currently, only a small set of versions are available - e.g. three for dmd.
Before this change, adding one new version was doable, albeit the process was
completely manual and so not reproducible. In order to add support for as many
versions as possible, we need to introduce some infrastructure to automate the
process.
What does this change do?
=========================
This change defines "version-catalog" Nix API based on machine generated and
readable JSON files listing the supported package versions along with the hashes
for each of their components.
More concretely, for each packackage:
* add (up to two) new JSON files:
* `pkgs/${package}/supported-source-versions.json`
* `pkgs/${package}/supported-binary-versions.json`
* add a `pkgs/${package}/version-catalog.nix` file which exposes the following
API:
* `supportedVersions` an attrset containing:
* `source` - a version -> component src hashes attrset, like this:
```json
{
"2.098.1" = {
dmd = "sha256-61hViU/IBUKHwXO+mt63lo3kBGL4Gtve83Uxi32gwnU=";
druntime = "sha256-HN/bxmGJHqubiUESZ7Y/4gdUj8ja9WhyLq24IDdZoe0=";
phobos = "sha256-GMa3k9QWfDiLSx0HH+n6eqgY6Pq0qkmXhbAslTsCH9A=";
tools = "sha256-+pH7X9YRh0yG0iZoDkC6X50uscb4WKXmk2V+74cMSW8=";
};
}
```
* `binary` - a [version -> release tarball hashes] attrset, like this:
```json
{
"2.090.1" = {
freebsd-64 = "sha256-A5uqLKtl3D+2LNUcZjF9bSviKhrMO/GV3THxuXco1j8=";
linux = "sha256-ByCrIA4Nt7i9YT0L19VXIL1IqIp+iObcZux407amZu4=";
osx = "sha256-9HwGVO/8jfZ6aTiDIUi8w4C4Ukry0uUS8ACP3Ig8dmU=";
windows = "sha256-ZDOJjN6Cml8GCQY2jrdVjDdJKmaRMp7vMhXD03SrUnU=";
};
```
* add a `getSourceVersion` function that when given a version returns a
derivation with the version and component hashes (from
`supportedVersions.source.${version}`) for that version passed in -
essentially a pkgs.callPackage-style callable package.
* add a `getBinaryVersion` function - equivalent for the binary packages.
…er nix identifiers
Why this change is needed?
==========================
The test matrix { package x version x system } contains various elements with
problems and the sheer size of the matrix makes it difficult to manage (e.g. to
know which CI jobs to mark as allowed to fail, to know on which elements the
checkPhase can be run or which tests need to be removed.
Investigation of the problems shows that many of the failures are caused by a
handful of tests that fail on a specific version and system. Instead of marking
the whole job as failed, it is better to omit the failing tests, as the rest of
the suite (more than 99%!) will still pass.
What does this change do?
=========================
For each package:
* Add a `pkgs/${packageName}/build-status.json` file containing
* each version known to have problems
* each system on which this version is known to have problems
* `build` - does it build succesfully
* if `false` the CI will be marked as allowed to fail
* `check` - does the check phase complete
* `skippedTests` - any tests that are necessary to be skipped to allow the
checkPhase to succeed
* Add to its derivation `passthru.buildStatus` field surfacing this information
on the Nix eval level
* Set `doCheck` to `buildStatus.check`
* Add to the `checkPhase`
* for each `test_file` in buildStatus.skippedTests
* a line `rm -v ${test_file}` that will remove the test on the corresponding
tripple (package, version, system)
PetarKirov
force-pushed
the
refactor/gen-pkg-versions
branch
from
d884879 to
d5da1a5
Compare
• Added input 'flake-compat':
'github:edolstra/flake-compat/0f9255e01c2351cc7d116c072cb317785dd33b33' (2023-10-04)
This change allows Nix users using to use this project without enabling Nix
flakes via the `default.nix` file. All flake outputs are directly accessible
through it.
$ nix repl
Welcome to Nix 2.16.2. Type :? for help.
nix-repl> :l .
Added 22 variables.
nix-repl> packages.x86_64-linux.dub
«derivation /nix/store/cyjprbl3s9c8b5qvdkkhqzfzzxxikj6q-dub-1.30.0.drv»
PetarKirov
force-pushed
the
refactor/gen-pkg-versions
branch
from
608ec89 to
3502a9f
Compare
PetarKirov
dismissed
dukc’s stale review
I've addressed all comments (please correct me if I'm wrong)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
dtoolsdepsupported-binary-versions.jsonfor 2.090.1 and 2.098.0