Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ci: explicitly list Conda compiler packages #2406

Closed
lidavidm opened this issue Dec 31, 2024 · 8 comments · Fixed by #2428
Closed

ci: explicitly list Conda compiler packages #2406

lidavidm opened this issue Dec 31, 2024 · 8 comments · Fixed by #2428
Labels
Type: enhancement New feature or request
Milestone
ADBC Libraries 16

Comments

@lidavidm
Copy link
Member

What feature or improvement would you like to see?

#2396 (comment)

compilers metapackage is a convenience that appears to not always be up to date. It would be better to list the packages we need. For instance it appears that Ruby and other packages were built with clang 18. But if we try to install compilers it still pulls in clang 17.

Since the conda requirements files don't support platform-specific requirements, I may migrate us to use something like pixi to manage environments instead.
@lidavidm lidavidm added the Type: enhancement New feature or request label Dec 31, 2024
@lidavidm lidavidm mentioned this issue Dec 31, 2024
Merged
@lidavidm
Copy link
Member Author

Example: if I try to install compilers right now I get clang 17.

lidavidm@lidavidm-nix ~/C/a/j/build (gh-2027)> CONDA_SUBDIR=osx-arm64 mamba create -n foo --dry-run compilers                                                                          (base) 

Looking for: ['compilers']

warning  libmamba osx version not found (virtual package skipped)
https://proxy.artifacts.voltrondata.run/artifact..  Checked  0.2s
rapidsai/osx-arm64 (check zst)                      Checked  0.2s
rapidsai/noarch                                             Using cache
conda-forge/noarch                                          Using cache
nvidia/osx-arm64 (check zst)                        Checked  0.0s
nvidia/noarch                                               Using cache
rapidsai/osx-arm64                                  91.0 B @   2.5kB/s  0.0s
nvidia/osx-arm64                                    91.0 B @ 998.0 B/s  0.1s
conda-forge/osx-arm64                               13.7MB @   2.8MB/s  4.8s
Transaction

  Prefix: /home/lidavidm/miniforge3/envs/foo

  Updating specs:

   - compilers


  Package                           Version  Build                 Channel          Size
──────────────────────────────────────────────────────────────────────────────────────────
  Install:
──────────────────────────────────────────────────────────────────────────────────────────

  + libgfortran-devel_osx-arm64      13.2.0  h5d7a38c_3            conda-forge       2MB
  + libiconv                           1.17  h0d3ecfb_2            conda-forge     676kB
  + libzlib                           1.3.1  h0d3ecfb_0            conda-forge      47kB
  + libcxx                           16.0.6  h4653b0c_0            conda-forge       1MB
  + llvm-openmp                      18.1.3  hcd81f8e_0            conda-forge     276kB
  + ca-certificates              2024.12.14  hf0a4a13_0            conda-forge     157kB
  + xz                                5.2.6  h57fd34a_0            conda-forge     236kB
  + icu                                73.2  hc8870d7_0            conda-forge      12MB
  + zstd                              1.5.5  h4f39d0f_0            conda-forge     401kB
  + zlib                              1.3.1  h0d3ecfb_0            conda-forge      85kB
  + tapi                          1100.0.11  he4954df_0            conda-forge     191kB
  + isl                                0.26  imath32_h347afa1_101  conda-forge     820kB
  + gmp                               6.3.0  hebf3989_1            conda-forge     449kB
  + libgfortran5                     13.2.0  hf226fd6_3            conda-forge     997kB
  + openssl                           3.3.0  h0d3ecfb_0            conda-forge       3MB
  + libxml2                          2.12.6  h35eab27_2            conda-forge     588kB
  + mpfr                              4.2.1  h41d338b_1            conda-forge     347kB
  + libgfortran                       5.0.0  13_2_0_hd922786_3     conda-forge     110kB
  + sigtool                           0.1.3  h44b9a77_0            conda-forge     210kB
  + libllvm17                        17.0.6  haab561b_1            conda-forge      25MB
  + mpc                               1.3.1  h91ba8db_0            conda-forge     104kB
  + ld64_osx-arm64                      711  h6a44367_0            conda-forge       1MB
  + llvm-tools                       17.0.6  haab561b_1            conda-forge      22MB
  + libclang-cpp17                   17.0.6  default_he012953_4    conda-forge      12MB
  + gfortran_impl_osx-arm64          13.2.0  h252ada1_3            conda-forge      18MB
  + ld64                                711  h39a299f_0            conda-forge      19kB
  + cctools_osx-arm64                   986  h998149b_0            conda-forge       1MB
  + clang-17                         17.0.6  default_he012953_4    conda-forge     718kB
  + cctools                             986  hf67d63f_0            conda-forge      22kB
  + clang                            17.0.6  default_h4cf2255_4    conda-forge      23kB
  + clangxx                          17.0.6  default_h4cf2255_4    conda-forge      23kB
  + compiler-rt_osx-arm64            17.0.6  h832e737_2            conda-forge      10MB
  + compiler-rt                      17.0.6  h3808999_1            conda-forge      95kB
  + clang_impl_osx-arm64             17.0.6  he47c785_13           conda-forge      18kB
  + clang_osx-arm64                  17.0.6  h54d7cd3_13           conda-forge      21kB
  + clangxx_impl_osx-arm64           17.0.6  h50f59cd_13           conda-forge      18kB
  + gfortran_osx-arm64               13.2.0  h57527a5_1            conda-forge      35kB
  + c-compiler                        1.8.0  hf48404e_1            conda-forge       6kB
  + clangxx_osx-arm64                17.0.6  h54d7cd3_13           conda-forge      19kB
  + gfortran                         13.2.0  h1ca8e4b_1            conda-forge      32kB
  + cxx-compiler                      1.8.0  h18dbf2f_1            conda-forge       6kB
  + fortran-compiler                  1.8.0  hc3477c4_1            conda-forge       6kB
  + compilers                         1.8.0  hce30654_1            conda-forge       7kB

  Summary:

  Install: 43 packages

  Total download: 115MB

──────────────────────────────────────────────────────────────────────────────────────────


Dry run. Exiting.

DryRunExit: Dry run. Exiting.

@kou
Copy link
Member

kou commented Dec 31, 2024

I tried the following but it didn't work:

diff --git a/ci/conda_env_glib.txt b/ci/conda_env_glib.txt
index fbe46652f..f961e27dc 100644
--- a/ci/conda_env_glib.txt
+++ b/ci/conda_env_glib.txt
@@ -16,12 +16,13 @@
 # under the License.
 
 arrow-c-glib
+clang >=18.*
 glib
 gobject-introspection
 meson
 postgresql
-# It appears 3.4.0 adds a dependency on stdckdint.h which isn't actually usable
-ruby <3.4.0
+ruby
 # TODO(https://github.com/apache/arrow-adbc/issues/2176): pin for now because
 # gobject-introspection uses a deprecated/removed API
 setuptools <74
+zlib

https://github.com/kou/arrow-adbc/actions/runs/12555264022/job/35005157611#step:10:78

FAILED: adbc-arrow-glib/ADBCArrow-1.0.gir 
env PKG_CONFIG_PATH=/Users/runner/local/lib/pkgconfig:/Users/runner/miniconda3/envs/test/lib/pkgconfig:/Users/runner/work/arrow-adbc/arrow-adbc/build/glib/meson-uninstalled PKG_CONFIG=/Users/runner/miniconda3/envs/test/bin/pkg-config CC=x86_64-apple-darwin13.4.0-clang /Users/runner/miniconda3/envs/test/bin/g-ir-scanner --quiet --no-libtool --namespace=ADBCArrow --nsversion=1.0 --warn-all --output adbc-arrow-glib/ADBCArrow-1.0.gir --c-include=adbc-arrow-glib/adbc-arrow-glib.h --warn-all -I/Users/runner/work/arrow-adbc/arrow-adbc/glib/adbc-arrow-glib -I/Users/runner/work/arrow-adbc/arrow-adbc/build/glib/adbc-arrow-glib -I/Users/runner/work/arrow-adbc/arrow-adbc/glib/. -I/Users/runner/work/arrow-adbc/arrow-adbc/build/glib/. -I/Users/runner/work/arrow-adbc/arrow-adbc/glib/../c/include -I/Users/runner/work/arrow-adbc/arrow-adbc/build/glib/../c/include --filelist=/Users/runner/work/arrow-adbc/arrow-adbc/build/glib/adbc-arrow-glib/libadbc-arrow-glib.1.dylib.p/ADBCArrow_1.0_gir_filelist --include=ADBC-1.0 --include=Arrow-1.0 --symbol-prefix=gadbc_arrow --identifier-prefix=GADBCArrow --pkg-export=adbc-arrow-glib --cflags-begin -I/Users/runner/work/arrow-adbc/arrow-adbc/glib/. -I/Users/runner/work/arrow-adbc/arrow-adbc/build/glib/. -I/Users/runner/work/arrow-adbc/arrow-adbc/glib/../c/include -I/Users/runner/work/arrow-adbc/arrow-adbc/build/glib/../c/include -I/Users/runner/local/include -I/Users/runner/miniconda3/envs/test/include -I/Users/runner/miniconda3/envs/test/include/glib-2.0 -I/Users/runner/miniconda3/envs/test/lib/glib-2.0/include -I/Users/runner/miniconda3/envs/test/include/gobject-introspection-1.0 --cflags-end --add-include-path=/Users/runner/miniconda3/envs/test/share/gir-1.0 --add-include-path=/Users/runner/work/arrow-adbc/arrow-adbc/build/glib/adbc-glib -L/Users/runner/work/arrow-adbc/arrow-adbc/build/glib/adbc-arrow-glib --library adbc-arrow-glib -L/Users/runner/work/arrow-adbc/arrow-adbc/build/glib/adbc-glib -L/Users/runner/local/lib -L/Users/runner/miniconda3/envs/test/lib -L/Users/runner/local/lib --extra-library=adbc_driver_manager -L/Users/runner/miniconda3/envs/test/lib --extra-library=gobject-2.0 --extra-library=glib-2.0 --extra-library=intl --extra-library=arrow-glib --extra-library=arrow --extra-library=girepository-1.0 --sources-top-dirs /Users/runner/work/arrow-adbc/arrow-adbc/glib/ --sources-top-dirs /Users/runner/work/arrow-adbc/arrow-adbc/build/glib/ --warn-error
dyld[8111]: Symbol not found: __ZN4absl12lts_2024072212log_internal10LogMessagelsIiLi0EEERS2_RKT_
  Referenced from: <ABD8BED3-9FF3-30AE-9F2C-FB6F306CF934> /Users/runner/miniconda3/envs/test/lib/libprotobuf.28.2.0.dylib
  Expected in:     <E99B267F-7144-31DD-A5D4-F5B0B82894DA> /Users/runner/miniconda3/envs/test/lib/libabsl_log_internal_message.2407.0.0.dylib
Command '['/Users/runner/work/arrow-adbc/arrow-adbc/build/glib/tmp-introspectnprlnsvi/ADBCArrow-1.0', '--introspect-dump=/Users/runner/work/arrow-adbc/arrow-adbc/build/glib/tmp-introspectnprlnsvi/functions.txt,/Users/runner/work/arrow-adbc/arrow-adbc/build/glib/tmp-introspectnprlnsvi/dump.xml']' died with <Signals.SIGABRT: 6>.

It seems that we need other approach like you mentioned or keep pinning Ruby for now.

@lidavidm
Copy link
Member Author

That error above is because of conda-forge/abseil-cpp-feedstock#87

@lidavidm
Copy link
Member Author

It's already affecting our CI, I think we need to wait for the Protobuf migration to finish

@kou
Copy link
Member

kou commented Dec 31, 2024

Oh! I missed the issue!

@xhochy
Copy link
Member

xhochy commented Jan 8, 2025

+clang >=18.*

This is not the compiler but the packages you want contain the target platform, e.g. clang_osx-arm64 18.x.

The actual fix is to update the compilers package: conda-forge/compilers-feedstock#71

@lidavidm
Copy link
Member Author

lidavidm commented Jan 8, 2025

Ah, well, it would be much easier to just wait for that to be updated!

@xhochy
Copy link
Member

xhochy commented Jan 9, 2025

This should be fixed now as the compilers package has been updated.

lidavidm added a commit to lidavidm/arrow-adbc that referenced this issue Jan 10, 2025
@lidavidm
chore: unpin some GLib/Ruby dependencies
79ebbca 
Fixes apache#2176.
Fixes apache#2406.
@lidavidm lidavidm mentioned this issue Jan 10, 2025
Merged
@github-actions github-actions bot added this to the ADBC Libraries 17 milestone Jan 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type: enhancement New feature or request
Projects
None yet
Milestone
ADBC Libraries 16
Development

Successfully merging a pull request may close this issue.

chore: unpin some GLib/Ruby dependencies lidavidm/arrow-adbc
3 participants
@kou @xhochy @lidavidm