Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rebuild for numpy 2.0 #247

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

regro-cf-autotick-bot
Copy link
Contributor

This PR has been triggered in an effort to update numpy2.

Notes and instructions for merging this PR:

  1. Please merge the PR only after the tests have passed.
  2. Feel free to push to the bot's branch to update this PR if needed.

Please note that if you close this PR we presume that the feedstock has been rebuilt, so if you are going to perform the rebuild yourself don't close this PR until the your rebuild has been merged.


Here are some more details about this specific migrator:

TL;DR: The way we build against numpy has changed as of numpy 2.0. This bot
PR has updated the recipe to account for the changes (see below for details).

The biggest change is that we no longer need to use the oldest available numpy
version at build time in order to support old numpy version at runtime - numpy
will by default use a compatible ABI for the oldest still-supported numpy versions.

Additionally, we no longer need to use {{ pin_compatible("numpy") }} as a
run requirement - this has been handled for more than two years now by a
run-export on the numpy package itself. The migrator will therefore remove
any occurrences of this.

However, you will still need to add the lower bound for the numpy version,
in line with what the upstream package requires. The default lower bound from
the run-export is >=1.19; if your package needs a newer version than that,
please add numpy >=x.y under run:.

Finally, by default, building against numpy 2.0 will assume that the package
is compatible with numpy 2.0, which is not necessarily the case. You should
check that the upstream package explicitly supports numpy 2.0, otherwise you
need to add a - numpy <2.0dev0 run requirement until that happens (check numpy
issue 26191 for an overview of the most important packages).

To-Dos:

  • Match run-requirements for numpy (i.e. check upstream pyproject.toml or however the project specifies numpy compatibility)
    • If upstream is not yet compatible with numpy 2.0, add numpy <2.0dev0 upper bound under run:.
    • If upstream is already compatible with numpy 2.0, double-check their supported numpy versions.
    • If upstream requires a minimum numpy version newer than 1.19, you need to add numpy >=x.y under run:.
  • Remove any remaining occurrences of {{ pin_compatible("numpy") }} that the bot may have missed.

PS. If the build does not compile anymore, this is almost certainly a sign that
the upstream project is not yet ready for numpy 2.0; do not close this PR until
a version compatible with numpy 2.0 has been released upstream and on this
feedstock (in the meantime, you can keep the bot from reopening this PR in
case of git conflicts by marking it as a draft).


If this PR was opened in error or needs to be updated please add the bot-rerun label to this PR. The bot will close this PR and schedule another one. If you do not have permissions to add this label, you can use the phrase @conda-forge-admin, please rerun bot in a PR comment to have the conda-forge-admin add it for you.

This PR was created by the regro-cf-autotick-bot. The regro-cf-autotick-bot is a service to automatically track the dependency graph, migrate packages, and propose package version updates for conda-forge. Feel free to drop us a line if there are any issues! This PR was generated by https://github.com/regro/cf-scripts/actions/runs/12244016446 - please use this URL for debugging.

TL;DR: The way we build against numpy has changed as of numpy 2.0. This bot
PR has updated the recipe to account for the changes (see below for details).

The biggest change is that we no longer need to use the oldest available numpy
version at build time in order to support old numpy version at runtime - numpy
will by default use a compatible ABI for the oldest still-supported numpy versions.

Additionally, we no longer need to use `{{ pin_compatible("numpy") }}` as a
run requirement - this has been handled for more than two years now by a
run-export on the numpy package itself. The migrator will therefore remove
any occurrences of this.

However, you will still need to add the lower bound for the numpy version,
in line with what the upstream package requires. The default lower bound from
the run-export is `>=1.19`; if your package needs a newer version than that,
please add `numpy >=x.y` under `run:`.

Finally, by default, building against numpy 2.0 will assume that the package
is compatible with numpy 2.0, which is not necessarily the case. You should
check that the upstream package explicitly supports numpy 2.0, otherwise you
need to add a `- numpy <2.0dev0` run requirement until that happens (check numpy
issue 26191 for an overview of the most important packages).

### To-Dos:
  * [ ] Match run-requirements for numpy (i.e. check upstream `pyproject.toml` or however the project specifies numpy compatibility)
    * If upstream is not yet compatible with numpy 2.0, add `numpy <2.0dev0` upper bound under `run:`.
    * If upstream is already compatible with numpy 2.0, double-check their supported numpy versions.
    * If upstream requires a minimum numpy version newer than 1.19, you need to add `numpy >=x.y` under `run:`.
  * [ ] Remove any remaining occurrences of `{{ pin_compatible("numpy") }}` that the bot may have missed.

PS. If the build does not compile anymore, this is almost certainly a sign that
the upstream project is not yet ready for numpy 2.0; do not close this PR until
a version compatible with numpy 2.0 has been released upstream and on this
feedstock (in the meantime, you can keep the bot from reopening this PR in
case of git conflicts by marking it as a draft).
@conda-forge-admin
Copy link
Contributor

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.

@sblauth
Copy link
Contributor

sblauth commented Dec 10, 2024

It also seems to me that the patch for numpy 2.0 was not sufficient. Is it okay if I modify the upstream PR to include the necessary changes and modify the patch here? Then, the other PRs can be re-run and should not fail anymore.

@minrk
Copy link
Member

minrk commented Dec 10, 2024

Yes, that should be fine. Since this is for stable, the upstream patch may need to be based on 2019.1.0.post0 instead of main.

@sblauth
Copy link
Contributor

sblauth commented Dec 10, 2024

Is there anything we have to keep in mind when merging this PR? I mean in the sense of the things the bot mentioned when opening the PR.
Also: To me it seems that now only numpy >= 2.0 is built here - do we also want to support older numpy versions?

@sblauth
Copy link
Contributor

sblauth commented Dec 11, 2024

Hmm, here we have a strange bug in the build: ls: cannot access 'jitfailure*': No such file or directory and qemu: uncaught target signal 11 (Segmentation fault) - core dumped

It only seems to affect the cross-compilation builds, but it also happens for the python 3.13 PRs - however, there the python 3.11 version, which fails here, is not affected.

@minrk
Copy link
Member

minrk commented Dec 11, 2024

building with numpy 2 does not exclude running with numpy 1. numpy 2 is abi-compatible back to numpy 1.21 (that's why the pin_compatible is removed, it places an unnecessarily high lower bound).

The ls jitfailure error just means that there weren't any jit failures to report (failure was likely earlier and more serious). A segfault is a bigger deal, I don't know what would cause it. It is interesting that it is only arm Python 3.11 that seems affected.

The 3.11 env is probably different between the two PRs, since in the 3.13 PR, only 3.13 is building with numpy 2, whereas here all Pythons are built against numpy 2.

@sblauth
Copy link
Contributor

sblauth commented Dec 12, 2024

The environments look identical for the failing and passing tests. The segfault seems to arise in libucs.so, see here

2024-12-10T15:57:33.1829612Z  0  /home/conda/feedstock_root/build_artifacts/fenics-pkgs_1733844805080/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.11/site-packages/dolfin/../../.././libucs.so.0(ucs_handle_error+0x38c) [0x7f6de852b7a0]
2024-12-10T15:57:33.1836144Z  1  /home/conda/feedstock_root/build_artifacts/fenics-pkgs_1733844805080/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.11/site-packages/dolfin/../../.././libucs.so.0(+0x339a4) [0x7f6de852b9a4]
2024-12-10T15:57:33.1837038Z  2  /home/conda/feedstock_root/build_artifacts/fenics-pkgs_1733844805080/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.11/site-packages/dolfin/../../.././libucs.so.0(+0x33d8c) [0x7f6de852bd8c]

However, ucx variants are identical between failing and passing builds - and it doesn't cause an issue under "normal" linux.

@sblauth
Copy link
Contributor

sblauth commented Dec 17, 2024

As I currently don't have another idea, do you think it would be worthwhile to try if the aarch64 builds work without cross-compiling @minrk ?

@minrk
Copy link
Member

minrk commented Dec 17, 2024

Worth a try

@sblauth
Copy link
Contributor

sblauth commented Dec 17, 2024

@conda-forge-admin please rerender

@minrk
Copy link
Member

minrk commented Dec 17, 2024

after three hours, arm builds are 50% through the C++ compilation. Not sure if it's going to finish in the six hour time limit.

@sblauth
Copy link
Contributor

sblauth commented Dec 17, 2024

... and even the case that did finish compiling faster now throws a segfault, this time for python 3.10, not 3.11 as before...

@sblauth
Copy link
Contributor

sblauth commented Dec 18, 2024

@conda-forge-admin please rerender

@sblauth
Copy link
Contributor

sblauth commented Dec 18, 2024

With cross-compilation enabled again for ARM, the tests again fail with a segfault for the python 3.11 builds on ARM, like before. To be honest I have no idea where this is coming from.

@sblauth
Copy link
Contributor

sblauth commented Jan 7, 2025

@minrk How much are the arm and ppc builds actually needed? I am asking because it might be an idea to drop the support for them here - for legacy fenics. This would also make sense in the light of the discussion at conda-forge/conda-forge-pinning-feedstock#6900

But I don't really have any experience with these builds and do not require them - so this might not be such a good idea after all...

@minrk
Copy link
Member

minrk commented Jan 7, 2025

I'm totally fine dropping ppc. ARM less so, because it seems most new developer laptops are ARM these days.

But dealing with these things is part of why I'm not putting time into maintaining this legacy recipe anymore. I would be okay dropping the linux cross compiles, if that's what makes this maintainable. I am also okay stopping dependency upgrades altogether on this feedstock.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants