Build failure: nix-format-unstable build can fail on GH checks

[ghost]

Steps To Reproduce

commit d94495d added check-nix-format.yml

https://github.com/NixOS/nixpkgs/actions/runs/8540287690/job/23396972473

the new check-format GH job needs to rebuild lots of stuff when run on staging and its dependencies seem to be flaky, causing the test to fail.

Build log

https://github.com/NixOS/nixpkgs/actions/runs/8540287690/job/23396972473

=================================== FAILURES ===================================
_________________ test_too_many_requests_retry_after_int_delay _________________
[gw1] linux -- Python 3.11.8 /nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/bin/python3.11

app = <SphinxTestApp buildername='linkcheck'>
capsys = <_pytest.capture.CaptureFixture object at 0x7fffef16de90>
status = <_io.StringIO object at 0x7fffedb06c20>

    @pytest.mark.sphinx('linkcheck', testroot='linkcheck-localserver', freshenv=True)
    def test_too_many_requests_retry_after_int_delay(app, capsys, status):
        with http_server(make_retry_after_handler([(429, "0"), (200, None)])), \
             mock.patch("sphinx.builders.linkcheck.DEFAULT_DELAY", 0), \
             mock.patch("sphinx.builders.linkcheck.QUEUE_POLL_SECS", 0.01):
            app.build()
        content = (app.outdir / 'output.json').read_text(encoding='utf8')
>       assert json.loads(content) == {
            "filename": "index.rst",
            "lineno": 1,
            "status": "working",
            "code": 0,
            "uri": "http://localhost:7777/",
            "info": "",
        }
E       AssertionError

tests/test_build_linkcheck.py:758: AssertionError
--------------------------- Captured stdout teardown ---------------------------
# testroot: root
# builder: linkcheck
# srcdir: /build/pytest-of-nixbld/pytest-0/popen-gw1/linkcheck-localserver
# outdir: /build/pytest-of-nixbld/pytest-0/popen-gw1/linkcheck-localserver/_build/linkcheck
# status: 
Running Sphinx v7.2.6
building [mo]: targets for 0 po files that are out of date
writing output... 
building [linkcheck]: targets for 1 source files that are out of date
updating environment: [new config] 1 added, 0 changed, 0 removed
reading sources... [100%] index
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
copying assets... done
writing output... [100%] index

(           index: line    1) broken    http://localhost:7777/ - HTTPConnectionPool(host='localhost', port=7777): Read timed out. (read timeout=0.05)
build finished with problems.

# warning: 

=============================== warnings summary ===============================
tests/test_build_html.py::test_validate_html_extra_path
  /build/source/sphinx/builders/html/__init__.py:1262: RemovedInSphinx80Warning: Sphinx 8 will drop support for representing paths as strings. Use "pathlib.Path" or "os.fspath" instead.
    config.html_extra_path.remove(entry)

tests/test_build_html.py::test_validate_html_extra_path
  /build/source/sphinx/builders/html/__init__.py:1258: RemovedInSphinx80Warning: Sphinx 8 will drop support for representing paths as strings. Use "pathlib.Path" or "os.fspath" instead.
    config.html_extra_path.remove(entry)

tests/test_build_html.py::test_validate_html_static_path
  /build/source/sphinx/builders/html/__init__.py:1275: RemovedInSphinx80Warning: Sphinx 8 will drop support for representing paths as strings. Use "pathlib.Path" or "os.fspath" instead.
    config.html_static_path.remove(entry)

tests/test_build_html.py::test_validate_html_static_path
  /build/source/sphinx/builders/html/__init__.py:1271: RemovedInSphinx80Warning: Sphinx 8 will drop support for representing paths as strings. Use "pathlib.Path" or "os.fspath" instead.
    config.html_static_path.remove(entry)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/test_build_linkcheck.py::test_too_many_requests_retry_after_int_delay - AssertionError
=========== 1 failed, 2017 passed, 31 skipped, 4 warnings in 46.84s ============
/nix/store/v5lsd029lz5lfhamivbgqyp3zdv94ah2-stdenv-linux/setup: line 1578: pop_var_context: head of shell_variables not a function context
error: builder for '/nix/store/j9xwyflla0nvw81kdikkmyp3xnfvq6g2-python3.11-sphinx-7.2.6.drv' failed with exit code 1;
       last 25 log lines:
       >
       > # warning:
       >
       > =============================== warnings summary ===============================
       > tests/test_build_html.py::test_validate_html_extra_path
       >   /build/source/sphinx/builders/html/__init__.py:1262: RemovedInSphinx80Warning: Sphinx 8 will drop support for representing paths as strings. Use "pathlib.Path" or "os.fspath" instead.
       >     config.html_extra_path.remove(entry)
       >
       > tests/test_build_html.py::test_validate_html_extra_path
       >   /build/source/sphinx/builders/html/__init__.py:1258: RemovedInSphinx80Warning: Sphinx 8 will drop support for representing paths as strings. Use "pathlib.Path" or "os.fspath" instead.
       >     config.html_extra_path.remove(entry)
       >
       > tests/test_build_html.py::test_validate_html_static_path
       >   /build/source/sphinx/builders/html/__init__.py:1275: RemovedInSphinx80Warning: Sphinx 8 will drop support for representing paths as strings. Use "pathlib.Path" or "os.fspath" instead.
       >     config.html_static_path.remove(entry)
       >
       > tests/test_build_html.py::test_validate_html_static_path
       >   /build/source/sphinx/builders/html/__init__.py:1271: RemovedInSphinx80Warning: Sphinx 8 will drop support for representing paths as strings. Use "pathlib.Path" or "os.fspath" instead.
       >     config.html_static_path.remove(entry)
       >
       > -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
       > =========================== short test summary info ============================
       > FAILED tests/test_build_linkcheck.py::test_too_many_requests_retry_after_int_delay - AssertionError
       > =========== 1 failed, 2017 passed, 31 skipped, 4 warnings in 46.84s ============
       > /nix/store/v5lsd029lz5lfhamivbgqyp3zdv94ah2-stdenv-linux/setup: line 1578: pop_var_context: head of shell_variables not a function context
       For full logs, run 'nix log /nix/store/j9xwyflla0nvw81kdikkmyp3xnfvq6g2-python3.11-sphinx-7.2.6.drv'.
error: 1 dependencies of derivation '/nix/store/z5khlkqncy92756kdk51djagk3szk2fb-ghc-9.6.4.drv' failed to build
error: 1 dependencies of derivation '/nix/store/6gj21qm1596akb9hbpy0dpgqq7nl4rl4-nixfmt-unstable-2024-03-01.drv' failed to build
Error: Process completed with exit code 1.

log here if short otherwise a link to a gist

Additional context

Add any other context about the problem here.

Notify maintainers

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
output here

@ConnorBaker

---

Add a 👍 reaction to issues you find important.

[ConnorBaker]

Thank you for the report @a-n-n-a-l-e-e!

@infinisil my thoughts on way(s) forward are:

1. Pin the version of nixfmt and enable some sort of caching to avoid rebuilds
2. Guard the workflow so it only runs on PRs which touch CUDA-related files (treewide: CUDA reformat with nixfmt-rfc-style #299578 (comment))

Is there precedent for a way to pin or cache tools used as part of CI (like Cachix)?

[ConnorBaker]

CC'ing a few more people who followed #299578. You might find this interesting or have ideas. I suspect adding more tooling or checks to Nixpkgs via workflows will run into similar issues!

@SomeoneSerge @samuela @piegamesde @philiptaron @gabyx @dasJ

[philiptaron]

Pin the version of nixfmt and enable some sort of caching to avoid rebuilds

We did that with nixpkgs-check-by-name.

1. Made a release that's a gzip-compressed Nix Archive of the build closure.
2. Use that archive in the workflow check.

Solves this specific problem.

Here's the scripts/release.sh that makes the release process relatively pain free.

[SomeoneSerge]

Let's start by pinning some commit from nixos-unstable? This suggests some further optimizations, like getting nixfmt into the github runners' own hot cache?

[infinisil]

Yeah pinning it to a Nixpkgs commit sounds like a good idea. We're not ready for an official stable release, but once we are we could consider doing the release artifact thing instead.

[ghost]

how about disabling the test in the meantime? i have a PR to disable the flaky test causing the build to fail, but to build on stating is taking a long time (113 minutes) to just run the format checks on files changed that aren't covered by the format checker.

https://github.com/NixOS/nixpkgs/actions/runs/8542206020/job/23403346673?pr=301285

[SomeoneSerge]

how about disabling the test in the meantime?

Takes the same number of LOC to set NIX_PATH 😆

[ghost]

how about disabling the test in the meantime?

Takes the same number of LOC to set NIX_PATH 😆

revert takes 2 clicks.

[SomeoneSerge]

@a-n-n-a-l-e-e #301310 should hopefully suffice