Add support for Numpy 2 #2048
Assignees
Labels
enhancement
New feature or request
Comments
G-D-Petrov
added a commit
that referenced
this issue
Jan 14, 2025
#### Reference Issues/PRs #2048 #### What does this implement or fix? There quite a few distinct changes as most of the work was in the CI and testing. I will try to separate the different changes here: - Numpy 2 Support: - [support for new c++ interface](https://github.com/man-group/ArcticDB/pull/2050/files#diff-d23d2e3a46aa783e6c27de2ff06123f6149cda2727b09eb48e0755bc29a3ede2) - [small change for how == and is are evaluated](https://github.com/man-group/ArcticDB/pull/2050/files#diff-8d6759b6a2a11f86f70ee321e943d6c8e7898d4419ed281ad5a7358426a49a9a) - [added some more checks during normalization](https://github.com/man-group/ArcticDB/pull/2050/files#diff-fa52ad937e8dc0daf708927a99994e7264fd9cb62557326919c023696808859eL121) - changes to tests to support new Numpy conventions (all of the changes to the pytests) - Pybind11 2.13 Support (Needed for numpy2): - removed python 3.6 and added 3.12 (relevant changes [here](https://github.com/man-group/ArcticDB/pull/2050/files#diff-5c3fa597431eda03ac3339ae6bf7f05e1a50d6fc7333679ec38e21b337cb6721L30), [here ](https://github.com/man-group/ArcticDB/pull/2050/files#diff-ea07d77332dfd2427efe1d2148bec3d99c7dd3c06160f7975d5ad53069b9083f), and [here](https://github.com/man-group/ArcticDB/pull/2050/files#diff-ea07d77332dfd2427efe1d2148bec3d99c7dd3c06160f7975d5ad53069b9083f) - Added new version of pybind11 and made changes to support the new interface: - [changed PythonUtils](https://github.com/man-group/ArcticDB/pull/2050/files#diff-57acdaae55064408141a79c555b1fe7c4f2d4a6ce7f7b291e814b36abe4bdbce) to add more info about the Python executable/libs neded for the newer version of pybind11 - Changes to support new pybind11 interfaces (relevant changes [here](https://github.com/man-group/ArcticDB/pull/2050/files#diff-9357e7e6906b72682dbae796b1a310682b3f7b334e3bba0cfee71c08f2e8cf57), [here](https://github.com/man-group/ArcticDB/pull/2050/files#diff-dc9806fb1c292ea747ea6f0d333206c084159dcf6fc94919b98bfce3fd90d8c8), [here](https://github.com/man-group/ArcticDB/pull/2050/files#diff-02770184046bcf181939ea70f927d1f835687e57fbcc843a71a500684674b8e9), [here](https://github.com/man-group/ArcticDB/pull/2050/files#diff-87b188a9b8a3a1ab16c10f975faa03e7f33e42347eb3019505b240432a4502e6), [here](https://github.com/man-group/ArcticDB/pull/2050/files#diff-6caec3c44be32d3a881cccbff1f03bbf8aedcdbf65e6e0c2c963e34bc675a246), [here](https://github.com/man-group/ArcticDB/pull/2050/files#diff-6c2d76c22148e822d651baf5e8c3c8e23528d7360afe98d3ba085104e46237db), [here](https://github.com/man-group/ArcticDB/pull/2050/files#diff-c7f1753eda666a2b8fd8c840f0ed2be21b809132b58ea6adbb14f49afb34039f) and [here](https://github.com/man-group/ArcticDB/pull/2050/files#diff-ed781e679ed39eaca3b7b13549967958654ab47eb42f920bfa5535933a1c097b) - Changes for testing: (also added tickets for these): - [Changed the persistent tests to use http instead of https, because the sts is broken on windows ](https://github.com/man-group/ArcticDB/pull/2050/files#diff-9c13e1c7a3ca0bbad7e24e48944e815de7bfe7ed57584ce31b0549eb7be6c94dL3) - #2080 - [change for persistent tests to use only encoding version 1, so as to not write the encoding_version to the cfg, because older versions cannot read it](https://github.com/man-group/ArcticDB/pull/2050/files#diff-11d84f04de731d90230614eaaeb91eb376a0b00061ce90dacbef195db6125005) - #2077 - [skip for storage lock test because it is flaky](https://github.com/man-group/ArcticDB/pull/2050/files#diff-f0c82624218a7a1ad65345a6e6ae054e689288ded7b0670999c5eabb9a1c89d9) - #2079 - [change finalize stage data to not be run against real s3 because it is too slow and it times out](https://github.com/man-group/ArcticDB/pull/2050/files#diff-67a5a1e0b68a1442a89e31f917eeaf651954e0b389d15c0ce0a6e7fd80706574) - #2078 I've also tested in against the full suite of tests incl the persistent ones (run [here](https://github.com/man-group/ArcticDB/actions/runs/12388677078/job/34581688721)) **N.B.:** We need to continue using the submodule on pybind11 instead of getting it from vcpkg, because the latest version of pybind11 are not available on its vcpkg distribution. Specifically, the versions that introduce the Numpy 2 Support. #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> --------- Co-authored-by: phoebusm <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
ArcticDBcurrently supportNumpy<2Describe the solution you'd like
ArcticDBshould supportNumpy>=2and the pin to it should be removed.N.B. this involves and upgrade to
pybind to >=2.12as wellThe text was updated successfully, but these errors were encountered: