Problems switching from numpy.distutils to Meson and wrapping Fortran code into Python

[rstoneback]

Describe the bug
I can't sort out how to use Meson to build a Python package incorporating Fortran that is working under numpy.distutils. I've found the documentation over at numpy (https://numpy.org/doc/stable/f2py/buildtools/meson.html) but following that hasn't worked for me so far.

I also found this discussion, #10536, for help moving the Python package apexpy to Meson. I've used apexpy's meson.build file as a guide but unfortunately it isn't working correctly for me.

The installation appears to run just fine but I can't import any of the compiled Fortran modules. I get an ImportError. This is using GitHub Actions computers.

I would greatly appreciate some assistance or direction.

As a secondary issue on my local Mac I get "ld: library not found for -lcrt1.o
collect2: error: ld returned 1 exit status"
when trying to install my OMMBV package on a fresh virtual environment of Python 3.12 using conda and "pip install ." I can install the package on this machine using numpy.distutils.

To Reproduce
The public repo I'm having issues with is over here: CosmicStudioSoftware/OMMBV#63

Full meson.build file is below.

project('OMMBV', 'c',
  version : '1.0.2',
  license: 'BSD-3',
  meson_version: '>=0.64.0',
  default_options : [
    'warning_level=2',
    'c_args=-Wno-unused-parameter -Wno-cast-function-type -Wno-missing-field-initializers',
    'fortran_args=-Wno-line-truncation -Wno-conversion -Wno-unused-variable -Wno-maybe-uninitialized -Wno-unused-dummy-argument -Wno-compare-reals',
    'fortran_std=legacy'],
)

add_languages('fortran', native: false)

py_mod = import('python')
py = py_mod.find_installation(pure: false)
py_dep = py.dependency()

incdir_numpy = run_command(py,
  ['-c', 'import os; os.chdir(".."); import numpy; print(numpy.get_include())'],
  check : true
).stdout().strip()

incdir_f2py = run_command(py,
    ['-c', 'import os; os.chdir(".."); import numpy.f2py; print(numpy.f2py.get_include())'],
    check : true
).stdout().strip()

inc_np = include_directories(incdir_numpy, incdir_f2py)

# Unlike distutils, meson doesn't yet include some of the f2py stuff
fortranobject_c = incdir_f2py / 'fortranobject.c'

fortranobject_lib = static_library('_fortranobject',
  fortranobject_c,
  dependencies: py_dep,
  include_directories: [incdir_numpy, incdir_f2py])

fortranobject_dep = declare_dependency(
  link_with: fortranobject_lib,
  include_directories: [incdir_numpy, incdir_f2py])


igrf_source = custom_target('igrfmodule.c',
  input : ['OMMBV/igrf13.f'],  # .f so no F90 wrappers
  output : ['igrfmodule.c', 'igrf-f2pywrappers.f'],
  command : [py, '-m', 'numpy.f2py', '@INPUT@', '-m', 'igrf', '--lower']
)
py.extension_module('igrf',
  [
    'OMMBV/igrf13.f',
    igrf_source,
    fortranobject_c
  ],
  include_directories: inc_np,
  link_with: fortranobject_lib,
  dependencies : [py_dep, fortranobject_dep],
  subdir: 'OMMBV',
  install : true
)


sources_source = custom_target('sourcesmodule.c',
  input : ['OMMBV/sources.f', 'OMMBV/igrf13.f'],  # .f so no F90 wrappers
  output : ['sourcesmodule.c', 'sources-f2pywrappers.f'],
  command : [py, '-m', 'numpy.f2py', '@INPUT@', '-m', 'sources', '--lower']
)
py.extension_module('sources',
  [
   'OMMBV/sources.f',
   'OMMBV/igrf13.f',
   sources_source,
   fortranobject_c
 ],
  include_directories: inc_np,
  link_with: fortranobject_lib,
  dependencies : [py_dep, fortranobject_dep],
  subdir: 'OMMBV',
  install : true
)


fcoords_source = custom_target('fortran_coordsmodule.c',
  input : ['OMMBV/_coords.f'],  # .f so no F90 wrappers
  output : ['fortran_coordsmodule.c', 'fortran_coords-f2pywrappers.f'],
  command : [py, '-m', 'numpy.f2py', '@INPUT@', '-m', 'fortran_coords', '--lower']
)
py.extension_module('fortran_coords',
  [
    'OMMBV/_coords.f',
    fcoords_source,
    fortranobject_c
  ],
  include_directories: inc_np,
  link_with: fortranobject_lib,
  dependencies : [py_dep, fortranobject_dep],
  subdir: 'OMMBV',
  install : true
)

py.install_sources(
  'OMMBV/_core.py',
  'OMMBV/heritage.py',
  'OMMBV/satellite.py',
  'OMMBV/trace.py',
  'OMMBV/trans.py',
  'OMMBV/utils.py',
  'OMMBV/vector.py',
  'OMMBV/__init__.py',
  pure: false,
  subdir: 'OMMBV'
)

Expected behavior
My expectation was that swapping out numpy.distutils for Meson using numpy docs would produce the same functional package

system parameters

• Is this a cross build or just a plain native build (for the same computer)? Native
• what operating system (e.g. MacOS Catalina, Windows 10, CentOS 8.0, Ubuntu 18.04, etc.) Ubuntu
• what Python version are you using e.g. 3.8.0 Multiple, 3.9, 3.11, 3.12 using GitHub Actions
• what meson --version
• what ninja --version if it's a Ninja build

[eli-schwartz]

The installation appears to run just fine but I can't import any of the compiled Fortran modules. I get an ImportError. This is using GitHub Actions computers.

It looks like it is getting it from the wrong directory, truthfully:

E   ImportError: cannot import name 'igrf' from partially initialized module 'OMMBV' (most likely due to a circular import) (/home/runner/work/OMMBV/OMMBV/OMMBV/__init__.py)

meson doesn't have a build_ext --inplace equivalent. meson-python supports editable installs, which remap imports to the correct location and "should" work (I think?) for similar scenarios at least. But in general, you'll want to be able to import and run your tests independently of the installed modules.

I noticed that your module is in OMMBV/ and your tests are in OMMBV/tests/ but the tests aren't actually installed. My personal packaging suggestion is that it only leads to confusion if you do this mixture (it might be different if you were installing them).

Moving the tests out would allow you to import OMMBV from an installed environment and the tests from the source tree. SciPy's dev.py has some clever handling demonstrating how to test a project where the tests are installed as part of the wheel.

[eli-schwartz]

@rgommers this is an interesting one :)

[rstoneback]

I noticed that your module is in OMMBV/ and your tests are in OMMBV/tests/ but the tests aren't actually installed. My personal packaging suggestion is that it only leads to confusion if you do this mixture (it might be different if you were installing them).

I must've missed adding them to the install this round. I believe under distutils they got installed.

Moving the tests out would allow you to import OMMBV from an installed environment and the tests from the source tree. SciPy's dev.py has some clever handling demonstrating how to test a project where the tests are installed as part of the wheel.

Thanks for the tip! I'll check that out tomorrow.

My plan was to release a version of OMMBV where the only change was the switch to Meson. After that I feel ok making more changes to the install, like removing the tests from distributions, where it isn't strictly needed.

@rgommers this is an interesting one :)

Glad to hear the problem is of interest. Given how many issues y'all must've seen I'll take it as a compliment :)

[dnicolodi]

Uh! This is a funny one, and meson or meson-python have nothing to do with the issue.

The error you are getting is:

ImportError: cannot import name 'igrf' from partially initialized module 'OMMBV' (most likely due to a circular import) (/home/runner/work/OMMBV/OMMBV/OMMBV/__init__.py)

from executing import OMMBV.

Indeed in OMMBV/__init__.py there is a from OMMBV import igrf statement. This should usually not be a problem because OMMBV.igrf does not import OMMBV and the Python interpreter can resolve all imports. However, note the path from where OMMBV/__init__.py is loaded: it is loaded from /home/runner/work/OMMBV/OMMBV/ which is the source path and not the path where the compiled igrf module is installed. At that location there is no OMMBV.igrf module to be loaded.

What I think happens at this point is that Python tries to find it at another location is sys.path and then it gets confused because it finds another OMMBV module that is different from the first, thus the initialization of the first OMMBV module never completes and this results in the circular import error.

All this happens because . is implicitly ins sys.path. If you change the current working dir to something else before executing import OMMBV, the error goes away.

Please note that this is independent of the build system you use to install the package. I believe that the reason why you did not encounter the error with setuptools is that yuo were doing an "old school" editable installation where the extension modules are copied into the source tree.

[rstoneback]

If you change the current working dir to something else before executing import OMMBV, the error goes away.

Thanks for the tip. You are correct, after changing my directory I can import OMMBV and use the fortran compiled functions.

I have run into an issue with the import though. On my local system I can do the following:

In [1]: import OMMBV

In [2]: OMMBV.sources.colat_long_r_to_ecef(10., 10., 6350.)
Out[2]: array([ 2898.60117106,  1879.33945374, -5328.10420964])

Works great.

This same command does not work on the Meson built version on GitHub. I have to import the fortran modules specifically.

        python -c "import OMMBV; from OMMBV import sources; print(sources.colat_long_r_to_ecef(10., 10., 6350.))"
        python -c "import OMMBV; import OMMBV.sources; print(OMMBV.sources.colat_long_r_to_ecef(10., 10., 6350.))"
        python -c "import OMMBV; print(OMMBV.sources.colat_long_r_to_ecef(10., 10., 6350.))"

yields

[ 2898.60117106  1879.33945374 -5328.10420964]

[ 2898.60117106  1879.33945374 -5328.10420964]

AttributeError: module 'OMMBV' has no attribute 'sources'

I'm not sure this is Meson specific but since the init files are the same in both cases I'm not sure what the problem could be. Clearly the fortran built sources submodule is there, but why doesn't it show up at 'OMMBV.sources' after the simple 'import OMMBV'? I've probably being doing "old school" editable installs too long but it doesn't seem like that would be an issue here since the system clearly can find a working .sources.

I believe that the reason why you did not encounter the error with setuptools is that yuo were doing an "old school" editable installation where the extension modules are copied into the source tree.

Yes, I use the "old school" approach on my local machine for convenience. OMMBV did work outside of my machine but I'm not sure anyone tried to run the tests.

I haven't got the tests working yet but I'll be back on that next week. Thanks again for the assistance.

[eli-schwartz]

This same command does not work on the Meson built version on GitHub. I have to import the fortran modules specifically.

It should work fine, it does for me locally using the Meson built version.

The issue which traditionally causes this is that __init__.py must import the module so that it is in the sys.modules cache, and then it will automatically be a property of the top-level module. Your __init__.py does so, at least on my machine with a meson-built copy.

[eli-schwartz]

.... aha. I git pulled the new code, and that changes. Now __init__.py is empty.

$ git diff @{1}..@{0}

diff --git a/meson.build b/meson.build
index 772f5d1..eeb597d 100644
--- a/meson.build
+++ b/meson.build
@@ -106,6 +106,9 @@ py.install_sources(
   'OMMBV/utils.py',
   'OMMBV/vector.py',
   'OMMBV/__init__.py',
+  'OMMBV/tests/__init__.py',
+  'OMMBV/tests/test_apex.py',
+  'OMMBV/tests/test_core.py',
   pure: false,
   subdir: 'OMMBV'
 )
\ No newline at end of file

This is installing tests/__init__.py as __init__.py. What you should do is use preserve_path: true (instead of subdir: 'OMMBV') so that e.g.

• OMMBV/utils.py gets installed as site-packages/OMMBV/utils.py
• OMMBV/tests/__init__.py gets installed as site-packages/OMMBV/tests/__init__.py

Alternatively, the files installed to subdir: 'OMMBV/tests' need to go in their own py.install_sources() invocation.